Adaptive human–computer systems require the recognition of human behavior states to provide real-time feedback to scaffold skill learning. These systems are being researched extensively for intervention and training in individuals with autism spectrum disorder (ASD). Autistic individuals are prone to social communication and behavioral differences that contribute to their high rate of unemployment. Teamwork training, which is beneficial for all people, can be a pivotal step in securing employment for these individuals. To broaden the reach of the training, virtual reality is a good option. However, adaptive virtual reality systems require real-time detection of behavior. Manual labeling of data is time-consuming and resource-intensive, making automated data annotation essential. In this paper, we propose a semi-supervised machine learning method to supplement manual data labeling of multimodal data in a collaborative virtual environment (CVE) used to train teamwork skills. With as little as 2.5% of the data manually labeled, the proposed semi-supervised learning model predicted labels for the remaining unlabeled data with an average accuracy of 81.3%, validating the use of semi-supervised learning to predict human behavior.

This study explores the intersection of Theory of Mind (ToM) abilities and driving performance among novice drivers, with a focus on autistic individuals. The purpose is to investigate how ToM deficits may impact driving behaviors and decision-making, ultimately informing the development of tailored interventions and training programs for autistic drivers. We conducted a series of driving simulations using a custom-built driving simulator, capturing multimodal data including driving performance metrics, attention allocation, and physiological responses. Participants were categorized based on NEPSY scores, which assess ToM abilities, and self-reported autism spectrum disorder (ASD) diagnosis. Driving tasks were designed to simulate real-world scenarios, particularly focusing on intersections and merging, where ToM skills are crucial for safe navigation. Our analysis revealed differences in driving behaviors among participants with varying ToM abilities as determined through the NEPSY. Participants with lower NEPSY scores exhibited less smooth driving behaviors, increased risk-taking tendencies, and differences in attention allocation compared to those with higher scores. Alternatively, individuals with ASD displayed comparable driving patterns overall. ToM abilities influence driving behaviors and decision-making, particularly in complex social driving scenarios. Tailored interventions addressing ToM deficits and stress management could improve driving safety and accessibility for autistic individuals. This study underscores the importance of considering social cognitive factors in driving education and licensure pathways, aiming for greater inclusivity and accessibility in transportation systems.

Collaborative virtual environment (CVE)-based teamwork training offers a promising avenue for inclusive teamwork training. The incorporation of a feedback mechanism within virtual training environments can enhance the training experience by scaffolding learning and promoting active collaboration. However, an effective feedback mechanism requires a robust prediction model of collaborative behaviors. This paper presents a novel approach using hidden Markov models (HMMs) to predict human behavior in collaborative interactions based on multimodal signals collected from a CVE-based teamwork training simulator. The HMM was trained using k-fold cross-validation, achieving an accuracy of 97.77%. The HMM was evaluated against expert-labeled data and compared against a rule-based prediction model, demonstrating the superior predictive capabilities of the HMM, with the HMM achieving 90.59% accuracy compared to 76.53% for the rule-based model. These results highlight the potential of HMMs to predict collaborative behaviors that could be used in a feedback mechanism to enhance teamwork training experiences despite the complexity of these behaviors. This research contributes to advancing inclusive and supportive virtual learning environments, bridging gaps in cross-neurotype collaborations.