30 May 2022 in JMIR Biomedical Engineering
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Autism spectrum disorder (ASD) is a prevalent neurodevelopmental disorder encountered by 1 in 44 children in the United States. Patients with autism face difficulty effectively communicating with peers, articulating feelings and emotions, and controlling behaviors. Rapid and early diagnosis leads to improved treatment outcomes; however, current medical techniques often take years. Difficult, time-consuming diagnoses accompanied by the recent advances in computer vision have led to a surge of interest from researchers in developing models to streamline the autism diagnosis process. Objective: We aimed to assess the viability of at-home autism diagnosis by leveraging video data collected from a mobile game app to train a computer vision model using insights extracted from the change in emotion-related expression features over time. Methods: With our GuessWhat game-based mobile app, we collected a video data set of 74 children with ASD and neurotypical diagnoses actively playing in a natural home environment. To investigate and quantify the significance of facial emotion features for ASD detection, we developed a deep learning–based autism classifier with the following two components: a convolutional neural network (CNN) attached to a long short-term memory (LSTM). We pretrained our CNN backbone in the following two ways: (1) on ImageNet or (2) on a compilation of facial expression recognition data sets to analyze autism detection performance improvements using emotion features. The output of each CNN is fed into an LSTM model to diagnose ASD from video data. Results: Our top-performing architecture employing a CNN backbone trained on the facial expression recognition data sets and a unidirectional LSTM achieved a top accuracy of 91.2%, precision of 91.2%, recall of 89.8%, and an F1-score of 90.6%. Conclusions: We discovered the change in an individual’s facial expression features over time as a relevant marker for autism detection. Extracting emotional features from each frame resulted in a 27.8% F1-score improvement when compared to ImageNet weights. Our study demonstrates the capability of mobile apps to collect a natural, diverse data set for an improved autism diagnosis. These results demonstrate that deep learning and computer vision–based methods are instrumental for automated autism diagnosis from at-home recorded videos using unspecialized equipment.
30 May 2022 in JMIR Preprints
BACKGROUND Autism Spectrum Disorder (ASD) is a prevalent neurodevelopmental disorder encountered by 1 in 44 children in the United States of America. Autism patients face difficulty effectively communicating with peers, articulating feelings and emotions, and controlling behaviors. Rapid and early diagnosis leads to improved treatment outcomes, however current medical techniques often take years. Difficult, time-consuming diagnoses accompanied by the recent advances in computer vision have led to a surge of interest from researchers in developing models to streamline the autism diagnosis process. OBJECTIVE We aim to assess the viability of at-home autism diagnosis by leveraging video data collected from a mobile game app to train a computer vision model utilizing insights extracted from the change in emotion expression features over time. METHODS With our GuessWhat game-based mobile app, we collect a video dataset of 74 ASD and NT children actively playing in a natural home environment. To investigate and quantify the significance of facial emotion features for ASD detection, we develop a deep learning-based autism classifier with two components: a Convolutional Neural Network (CNN) attached to a Long Short Term Memory (LSTM). We pre-train our CNN backbone in two fashions: (1) on ImageNet or (2) on a compilation of Facial Expression Recognition (FER) datasets to analyze autism detection performance improvements utilizing emotion features. The output of each CNN is fed into an LSTM model to diagnose ASD from video data. RESULTS Our top-performing architecture utilizing a CNN backbone pre-trained on ImageNet obtained an accuracy of 45.8% and an F1 score of 62.8% while our corresponding top architecture employing a CNN backbone trained on the FER datasets achieved top accuracy of 91.2% and an F1 score of 90.6%. CONCLUSIONS We discovered the change in an individual's facial expression features over time as a relevant marker for autism detection. Extracting emotional features from each frame resulted in a 27.8% F1 score improvement when compared to non-emotion weights. Our study demonstrates the capability of mobile applications to collect a natural, diverse dataset for an improved autism diagnosis. These results demonstrate Deep Learning and Computer Vision based methods are instrumental for automated autism diagnosis from at-home recorded videos using unspecialized equipment.