With advances in digital technology, including deep learning and big data analytics, new methods have been developed for autism diagnosis and intervention. Emotion recognition and the detection of autism in children are prominent subjects in autism research. Typically using single-modal data to analyze the emotional states of children with autism, previous research has found that the accuracy of recognition algorithms must be improved. Our study creates datasets on the facial and speech emotions of children with autism in their natural states. A convolutional vision transformer-based emotion recognition model is constructed for the two distinct datasets. The findings indicate that the model achieves accuracies of 79.12% and 83.47% for facial expression recognition and Mel spectrogram recognition, respectively. Consequently, we propose a multimodal data fusion strategy for emotion recognition and construct a feature fusion model based on an attention mechanism, which attains a recognition accuracy of 90.73%. Ultimately, by using gradient-weighted class activation mapping, a prediction heat map is produced to visualize facial expressions and speech features under four emotional states. This study offers a technical direction for the use of intelligent perception technology in the realm of special education and enriches the theory of emotional intelligence perception of children with autism.
Restricted and Repetitive Behaviors (RRBs) are hallmark features of children with autism spectrum disorder (ASD) and are also one of the diagnostic criteria for the condition. Traditional methods of RRBs assessment through manual observation are limited by low diagnostic efficiency and uncertainty in outcomes. As a result, AI-assisted screening for autism has emerged as a promising research direction. In this study, we explore the synergy of visual foundation models and multimodal large language models (MLLMs), proposing a Multi-Model Synergistic Restricted and Repetitive Behavior Recognition method (MS-RRBR). Based on this method, we developed an interpretable multi-model autonomous question-answering system. To evaluate the effectiveness of our approach, we collected and annotated the Autism Restricted and Repetitive Behavior Dataset (ARRBD), which includes 10 ASD-related behaviors easily observable from various visual perspectives. Experimental results on the ARRBD dataset demonstrate that our multi-model collaboration outperforms single-model approaches, achieving the highest recognition accuracy of 94.94%. The MS-RRBR leverages the extensive linguistic knowledge of GPT-4o to enhance the zero-shot visual recognition capabilities of the MLLM, while also providing clear explanations for system decisions. This approach holds promise for providing timely, reliable, and accurate technical support for clinical diagnosis and educational rehabilitation in ASD.
With advances in digital technology, including deep learning and big data analytics, new methods have been developed for autism diagnosis and intervention. Emotion recognition and the detection of autism in children are prominent subjects in autism research. Typically using single-modal data to analyze the emotional states of children with autism, previous research has found that the accuracy of recognition algorithms must be improved. Our study creates datasets on the facial and speech emotions of children with autism in their natural states. A convolutional vision transformer-based emotion recognition model is constructed for the two distinct datasets. The findings indicate that the model achieves accuracies of 79.12% and 83.47% for facial expression recognition and Mel spectrogram recognition, respectively. Consequently, we propose a multimodal data fusion strategy for emotion recognition and construct a feature fusion model based on an attention mechanism, which attains a recognition accuracy of 90.73%. Ultimately, by using gradient-weighted class activation mapping, a prediction heat map is produced to visualize facial expressions and speech features under four emotional states. This study offers technical direction for the use of intelligent perception technology in the realm of special education and enriches the theory of emotional intelligence perception of children with autism.