Many children with Autism Spectrum Disorder (ASD) exhibit atypical gaze behaviors related to joint attention, a fundamental social-communication skill. Specifically, children with ASD show differences in the skills of gaze sharing and gaze following. In this work we present a novel virtual reality (VR)-based system, called InViRS, in which children with ASD play games allowing them to practice gaze sharing and gaze following. InViRS has three main design contributions: (i) a closed-loop joint attention paradigm with real-time tracking of the participant's eye gaze and game performance measures, (ii) an assistive feedback mechanism that provides guidance and hints in real time, and (iii) a controller that adaptively changes the avatar's gaze prompts according to the performance measures. Results from a pilot study to evaluate the feasibility of InViRS with 9 autistic ¹ children and 9 typically developing (TD) children offered preliminary support for the feasibility of successful gameplay as well as positive impacts on the targeted skills of gaze sharing and gaze following.

Autistic adults possess many skills sought by employers, but may be at a disadvantage in the workplace if social-communication differences negatively impact teamwork. We present a novel collaborative virtual reality (VR)-based activities simulator, called ViRCAS, that allows autistic and neurotypical adults to work together in a shared virtual space, offering the chance to practice teamwork and assess progress. ViRCAS has three main contributions: 1) a new collaborative teamwork skills practice platform; 2) a stakeholder-driven collaborative task set with embedded collaboration strategies; and 3) a framework for multimodal data analysis to assess skills. Our feasibility study with 12 participant pairs showed preliminary acceptance of ViRCAS, a positive impact of the collaborative tasks on supported teamwork skills practice for autistic and neurotypical individuals, and promising potential to quantitatively assess collaboration through multimodal data analysis. The current work paves the way for longitudinal studies that will assess whether the collaborative teamwork skill practice that ViRCAS provides also contributes towards improved task performance.

Collaborative virtual environment (CVE)-based teamwork training offers a promising avenue for inclusive teamwork training. The incorporation of a feedback mechanism within virtual training environments can enhance the training experience by scaffolding learning and promoting active collaboration. However, an effective feedback mechanism requires a robust prediction model of collaborative behaviors. This paper presents a novel approach using hidden Markov models (HMMs) to predict human behavior in collaborative interactions based on multimodal signals collected from a CVE-based teamwork training simulator. The HMM was trained using k-fold cross-validation, achieving an accuracy of 97.77%. The HMM was evaluated against expert-labeled data and compared against a rule-based prediction model, demonstrating the superior predictive capabilities of the HMM, with the HMM achieving 90.59% accuracy compared to 76.53% for the rule-based model. These results highlight the potential of HMMs to predict collaborative behaviors that could be used in a feedback mechanism to enhance teamwork training experiences despite the complexity of these behaviors. This research contributes to advancing inclusive and supportive virtual learning environments, bridging gaps in cross-neurotype collaborations.

Effective workplace collaboration is essential for productivity and creativity, yet achieving the necessary mutual understanding can be challenging, particularly involving individuals from different neurotypes. This work evaluates VIRTUES, a Virtual Reality (VR) platform designed to foster mutual understanding and collaborative behaviors between autistic and neurotypical individuals. VIRTUES integrates multimodal sensing (eye tracking, interaction logs, and transcribed speech) to objectively quantify five defined dimensions of collaboration while providing real-time, context-aware support through an embedded rule-based feedback mechanism. A user study involving 12 autistic–neurotypical pairs demonstrates that VIRTUES can assess and support collaborative efforts across different neurotypes. Through synchronized sensing data, we identified that Information Pooling serves as a critical driving factor for successful collaborative performance. These preliminary findings suggest that VIRTUES provides a foundation for exploring inclusive teamwork and may inform the design of future interventions to support neurodiverse social-technical skill acquisition.