13 February 2020 in IEEE Transactions on Visualization and Computer Graphics
We conduct novel analyses of users' gaze behaviors in dynamic virtual scenes and, based on our analyses, we present a novel CNN-based model called DGaze for gaze prediction in HMD-based applications. We first collect 43 users' eye tracking data in 5 dynamic scenes under free-viewing conditions. Next, we perform statistical analysis of our data and observe that dynamic object positions, head rotation velocities, and salient regions are correlated with users' gaze positions. Based on our analysis, we present a CNN-based model (DGaze) that combines object position sequence, head velocity sequence, and saliency features to predict users' gaze positions. Our model can be applied to predict not only realtime gaze positions but also gaze positions in the near future and can achieve better performance than prior method. In terms of realtime prediction, DGaze achieves a 22.0% improvement over prior method in dynamic scenes and obtains an improvement of 9.5% in static scenes, based on using the angular distance as the evaluation metric. We also propose a variant of our model called DGaze ET that can be used to predict future gaze positions with higher precision by combining accurate past gaze data gathered using an eye tracker. We further analyze our CNN architecture and verify the effectiveness of each component in our model. We apply DGaze to gaze-contingent rendering and a game, and also present the evaluation results from a user study.
28 December 2021 in IEEE Transactions on Visualization and Computer Graphics
Understanding human visual attention in immersive virtual reality (VR) is crucial for many important applications, including gaze prediction, gaze guidance, and gaze-contingent rendering. However, previous works on visual attention analysis typically only explored one specific VR task and paid less attention to the differences between different tasks. Moreover, existing task recognition methods typically focused on 2D viewing conditions and only explored the effectiveness of human eye movements. We first collect eye and head movements of 30 participants performing four tasks, i.e., Free viewing , Visual search , Saliency , and Track , in 15 360-degree VR videos. Using this dataset, we analyze the patterns of human eye and head movements and reveal significant differences across different tasks in terms of fixation duration, saccade amplitude, head rotation velocity, and eye-head coordination. We then propose EHTask – a novel learning-based method that employs eye and head movements to recognize user tasks in VR. We show that our method significantly outperforms the state-of-the-art methods derived from 2D viewing conditions both on our dataset (accuracy of $84.4\%$ versus $62.8\%$ ) and on a real-world dataset ( $61.9\%$ versus $44.1\%$ ). As such, our work provides meaningful insights into human visual attention under different VR tasks and guides future work on recognizing user tasks in VR.
21 June 2022 in IEEE Transactions on Visualization and Computer Graphics
Role-playing is widely used in many areas, such as psychotherapy and behavior change. However, few studies have explored the possible effects of playing multiple roles in a single role-playing process. We propose a new role-playing paradigm, called role-exchange playing, in which a user plays two opposite roles successively in the same simulated event for better cognitive enhancement. We designed an experiment with this novel role-exchange playing strategy in the immersive virtual environments; and school bullying was chosen as a scenario in this case. A total of 234 middle/high school students were enrolled in the mixed-design experiment. From the user study, we found that through role-exchange, students developed more morally correct opinions about bullying, as well as increased empathy and willingness to engage in supportive behavior. They also showed increased commitment to stopping bullying others. Our role-exchange paradigm could achieve a better effect than traditional role-playing methods in situations where participants have no prior experience associated with the roles they play. Therefore, using role-exchange playing in the immersive virtual environments to educate minors can help prevent them from bullying others in the real world. Our study indicates a positive significance in moral education of teenagers. Our role-exchange playing may have the potential to be extended to such applications as counseling, therapy, and crime prevention.
28 June 2017 in Computer Graphics Forum
12 March 2024 in IEEE Transactions on Visualization and Computer Graphics
Children diagnosed with Autism Spectrum Disorder (ASD) often exhibit motor disorders. Dance Movement Therapy (DMT) has shown great potential for improving the motor control ability of children with ASD. However, traditional DMT methods often lack vividness and are difficult to implement effectively. To address this issue, we propose a Mixed Reality DMT approach, utilizing interactive virtual agents. This approach offers immersive training content and multi-sensory feedback. To improve the training performance of children with ASD, we introduce a novel training paradigm featuring a self-guided mode. This paradigm enables the rapid creation of a virtual twin agent of the child with ASD using a single photo to embody oneself, which can then guide oneself during training. We conducted an experiment with the participation of 24 children diagnosed with ASD (or ASD propensity), recording their training performance under various experimental conditions. Through expert rating, behavior coding of training sessions, and statistical analysis, our findings revealed that the use of the twin agent for self-guidance resulted in noticeable improvements in the training performance of children with ASD. These improvements were particularly evident in terms of enhancing movement quality and refining overall target-related responses. Our study holds clinical potential in the field of medical treatment and rehabilitation for children with ASD.
13 July 2023 in The Visual Computer