Early detection of autism, a neurodevelopmental disorder marked by social communication challenges, is crucial for timely intervention. Recent advancements have utilized naturalistic home videos captured via the mobile application GuessWhat. Through interactive games played between children and their guardians, GuessWhat has amassed over 3,000 structured videos from 382 children, both diagnosed with and without Autism Spectrum Disorder (ASD). This collection provides a robust dataset for training computer vision models to detect ASD-related phenotypic markers, including variations in emotional expression, eye contact, and head movements. We have developed a protocol to curate high-quality videos from this dataset, forming a comprehensive training set. Utilizing this set, we trained individual LSTM-based models using eye gaze, head positions, and facial landmarks as input features, achieving test AUCs of 86%, 67%, and 78%, respectively. To boost diagnostic accuracy, we applied late fusion techniques to create ensemble models, improving the overall AUC to 90%. This approach also yielded more equitable results across different genders and age groups. Our methodology offers a significant step forward in the early detection of ASD by potentially reducing the reliance on subjective assessments and making early identification more accessibly and equitable.
Early detection of Autism Spectrum Disorder (ASD), a neurodevelopmental condition characterized by social communication challenges, is essential for timely intervention. Naturalistic home videos collected via mobile applications offer scalable opportunities for digital diagnostics. We leveraged GuessWhat, a mobile game designed to engage parents and children, which has generated over 3000 structured videos from 382 children. From this collection, we curated a final analytic sample of 688 feature-rich videos centered on a single dyad, enabling more consistent modeling. We developed a two-step pipeline: (1) filtering to isolate high-quality videos, and (2) feature engineering to extract interpretable behavioral signals. Unimodal LSTM-based models trained on eye gaze, head position, and facial expression achieved test AUCs of 86% (95% CI: 0.79–0.92), 78% (95% CI: 0.69–0.86), and 67% (95% CI: 0.55–0.78), respectively. Late-stage fusion of unimodal outputs significantly improved predictive performance, yielding a test AUC of 90% (95% CI: 0.84–0.95). Our findings demonstrate the complementary value of distinct behavioral channels and support the feasibility of using mobile-captured videos for detecting clinically relevant signals. While further work is needed to improve generalizability and inclusivity, this study highlights the promise of real-time, scalable autism phenotyping for early interventions.