With the advent of modern educational technologies, detecting student engagement has emerged as a crucial tool for enhancing educational effectiveness and classroom management. To improve the accuracy and efficiency of existing engagement detection systems, this study presents an enhanced model utilizing the YOLOv8 architecture. The proposed framework incorporates the cascading of two YOLOv8 models to refine the search for faces in images, employs a modified ResNet50 for feature extraction at different engagement levels, and finally uses a Support Vector Machine (SVM) classifier to accurately detect student engagement levels based on the features extracted by the modified ResNet50. The integration of additional convolutional layers and fully connected layers within the YOLOv8 architecture optimizes detection capabilities and classification accuracy. For training, we used a comprehensive dataset encompassing various scenarios of student engagement to ensure the robustness of the detection system. Experimental results highlight the superior performance of the model, with a precision of 94.5%, a recall of 92.3%, an mAP0.5 of 92.7%, and an mAP0.5:0.95 of 66.3%, outperforming conventional models such as YOLOv5, ViT, RT-DETR, Faster-RCNN, and ResNet. This framework not only demonstrates significant potential for real-time applications but also represents a considerable advancement in the field of educational monitoring through artificial intelligence. Future research will explore the integration of Explainable Artificial Intelligence (XAI) techniques to further enhance the transparency and interpretability of the model’s decisions, thereby increasing the confidence and acceptance of these technologies in educational settings.