Engagement plays an essential role in the learning process. Recognition of learning engagement in the classroom helps us understand the student’s learning state and optimize the teaching and study processes. Traditional recognition methods such as self-report and teacher observation are time-consuming and obtrusive to satisfy the needs of large-scale classrooms. With the development of big data analysis and artificial intelligence, applying intelligent methods such as deep learning to recognize learning engagement has become the research hotspot in education. In this paper, based on non-invasive classroom videos, first, a multi-cues classroom learning engagement database was constructed. Then, we introduced the power IoU loss function to You Only Look Once version 5 (YOLOv5) to detect the students and obtained a precision of 95.4%. Finally, we designed a bimodal learning engagement recognition method based on ResNet50 and CoAtNet. Our proposed bimodal learning engagement method obtained an accuracy of 93.94% using the KNN classifier. The experimental results confirmed that the proposed method outperforms most state-of-the-art techniques.
Due to the impact of the COVID-19 pandemic, many students are unable to attend face-to-face courses, Therefore, in this case, distance education should be promoted to replace face-to-face education. However, because of the imbalance of education in different regions, such as the imbalance of education resources between rural and urban areas, the quality of distance education may not be guaranteed. Therefore, in China and some regions, there have been efforts made to carry out blended synchronous classroom attempts. In hybrid synchronous classroom situations, teachers’ workloads have increased, and it is difficult to fully understand students’ learning efficiency and class participation. We use deep learning to identify the behaviors of teachers and students in a blended synchronous classroom-based situation, aiming to automate the analysis of classroom videos, which can help teachers in classroom reflection and summary in a blended synchronous classroom or face-to-face classroom. In the behavior recognition of students and teachers, we combine the head, hand, and body posture information of teachers and students and add the feature pyramid (FPN) and convolutional block attention module (CBAM) for comparative experiments. Finally, S–T (student–teacher) analysis and engagement analysis were carried out on the identification results.
Emotion recognition is critical to understanding students’ emotional states. However, problems such as crowded classroom environments, changing light, and occlusion often affect the accuracy of recognition. This study proposes an emotion recognition algorithm specifically for classroom environments. Firstly, the study adds the self-made MCC module and the Wise-IoU loss function to make object detection in the YOLOv8 model more accurate and efficient. Compared with the native YOL0v8x, it reduces the parameters by 16% and accelerates the inference speed by 20%. Secondly, in order to address the intricacies of the classroom setting and the specific requirements of the emotion recognition task, a multi-channel emotion recognition network (MultiEmoNet) has been developed. This network fuses skeletal, environmental, and facial information, and introduces a central loss function and an attention module AAM to enhance the feature extraction capability. The experimental results show that MultiEmoNet achieves a classification accuracy of 91.4% on a homemade classroom student emotion dataset, which is a 10% improvement over the single-channel classification algorithm. In addition, this study also demonstrates the dynamic changes in students’ emotions in the classroom through visual analysis, which helps teachers grasp students’ emotional states in real time. This paper validates the potential of multi-channel information-fusion deep learning techniques for classroom teaching analysis and provides new ideas and tools for future improvements to emotion recognition techniques.