With the rapid development of artificial intelligence and big data technology, intelligent education systems have become a key research focus in the field of modern educational technology. This study aims to enhance the intelligence level of educational systems by accurately detecting student behavior in the classroom using deep learning techniques. We propose a method for detecting student classroom behavior based on an improved RT DETR (Real-Time Detection Transformer) object detection algorithm. By combining actual classroom observation data with AI-generated data, we create a comprehensive and diverse student behavior dataset (FSCB-dataset). This dataset not only more realistically simulates the classroom environment but also effectively addresses the scarcity of datasets and reduces the cost of dataset construction. The study introduces MobileNetV3 as a lightweight backbone network, reducing the model parameters to one-tenth of the original while maintaining nearly the same accuracy. Additionally, by incorporating learnable position encoding and dynamic upsampling techniques, the model significantly improves its ability to recognize small objects and complex scenes. Test results on the FSCB-dataset show that the improved model achieves significant improvements in real-time performance and computational efficiency. The lightweight network is also easy to deploy on mobile devices, demonstrating its practicality in resource-constrained environments.