Virtual Reality (VR) provides great opportunities for police officers to train decision-making and acting (DMA) in cognitively demanding and stressful situations. This paper presents a summary of findings from a three-year project, including requirements collected from experienced police trainers and industry experts, and quantitative and qualitative results of human factor studies and field trials. Findings include advantages of VR training such as the possibility to safely train high-risk situations in controllable and reproducible training environments, include a variety of avatars that would be difficult to use in real-life training (e.g., vulnerable populations or animals) and handle dangerous equipment (e.g., explosives) but also highlight challenges such as tracking, locomotion and intelligent virtual agents. The importance of strong alignment between training didactics and technical possibilities is highlighted and potential solutions presented. Furthermore training outcomes are transferable to real-world police duties and may apply to other domains that would benefit from simulation-based training.

In this article, we present a novel method for evaluating guidance systems using an immersive virtual environment in combination with a mobile eye tracking system. Accurate measurements of position, locomotion, viewing frustum, and gaze are captured in the virtual environment. They are applied to the projection of an attention map onto the virtual 3D environment for visualizing the fixation in the environment as well as the amount of time objects were fixated. To demonstrate the method's applicability, we conducted an experiment with 24 participants evaluating a guidance system of a large public infrastructure. The results show that our method allows for the creation of attention maps as well as for the identification of objects of interest based on eye tracking.

In medical simulation training two approaches are currently rather disjunct: realistic manikins are used to teach physical skills and procedures and VR systems are used to train situation assessment and decision making. We propose a mixed reality approach, which allows trainees to use real tools and their hands when interacting with a physical manikin overlaid with a responsive virtual avatar. In close exchange with first responder organizations, we developed and evaluated an MR training scenario. In the scenario, users can talk to injured people in a car accident, assess the threat of the environment, and utilize real medical equipment. Participants experienced high levels of physical- and self-presence, increased stress levels, and reported a high technology acceptance. The proposed main requirements of first responders regarding haptic multi-sensory skill training in MR and the lessons learned from the workshop aim to guide the design of training solutions for medical training in MR.

Background: Training in social-verbal interactions is crucial for medical first responders (MFRs) to assess a patient’s condition and perform urgent treatment during emergency medical service administration. Integrating conversational agents (CAs) in virtual patients (VPs), that is, digital simulations, is a cost-effective alternative to resource-intensive human role-playing. There is moderate evidence that CAs improve communication skills more effectively when used with instructional interventions. However, more recent GPT-based artificial intelligence (AI) produces richer, more diverse, and more natural responses than previous CAs and has control of prosodic voice qualities like pitch and duration. These functionalities have the potential to better match the interaction expectations of MFRs regarding habitability. Objective: We aimed to study how the integration of GPT-based AI in a mixed reality (MR)–VP could support communication training of MFRs. Methods: We developed an MR simulation of a traffic accident with a VP. ChatGPT (OpenAI) was integrated into the VP and prompted with verified characteristics of accident victims. MFRs (N=24) were instructed on how to interact with the MR scenario. After assessing and treating the VP, the MFRs were administered the Mean Opinion Scale-Expanded, version 2, and the Subjective Assessment of Speech System Interfaces questionnaires to study their perception of the voice quality and the usability of the voice interactions, respectively. Open-ended questions were asked after completing the questionnaires. The observed and logged interactions with the VP, descriptive statistics of the questionnaires, and the output of the open-ended questions are reported. Results: The usability assessment of the VP resulted in moderate positive ratings, especially in habitability (median 4.25, IQR 4-4.81) and likeability (median 4.50, IQR 3.97-5.91). Interactions were negatively affected by the approximately 3-second latency of the responses. MFRs acknowledged the naturalness of determining the physiological states of the VP through verbal communication, for example, with questions such as “Where does it hurt?” However, the question-answer dynamic in the verbal exchange with the VP and the lack of the VP’s ability to start the verbal exchange were noticed. Noteworthy insights highlighted the potential of domain-knowledge prompt engineering to steer the actions of MFRs for effective training. Conclusions: Generative AI in VPs facilitates MFRs’ training but continues to rely on instructions for effective verbal interactions. Therefore, the capabilities of the GPT-VP and a training protocol need to be communicated to trainees. Future interactions should implement triggers based on keyword recognition, the VP pointing to the hurting area, conversational turn-taking techniques, and add the ability for the VP to start a verbal exchange. Furthermore, a local AI server, chunk processing, and lowering the audio resolution of the VP’s voice could ameliorate the delay in response and allay privacy concerns. Prompting could be used in future studies to create a virtual MFR capable of assisting trainees.

The contemporary geopolitical environment and strategic uncertainty shaped by asymmetric and hybrid threats urge the future development of hands-on training in realistic environments. Training in immersive, virtual environments is a promising approach. Immersive training can support training for contexts that are otherwise hard to access, dangerous, or have high costs. This paper discusses the challenges for virtual reality training in the CBRN (chemical, biological, radioactive, nuclear) domain. Based on initial considerations and a literature review, we conducted a survey and three workshops to gather requirements for CBRN training in virtual environments. We structured the gathered insights into four overarching themes—the future of CBRN training, ethical and safety requirements, evaluation and feedback, and tangible objects and tools. We provide insights on these four themes and discuss recommendations.

Mixed reality (MR) technology has the potential to enhance the disaster preparedness of medical first responders in mass-casualty incidents through new training methods. In this manuscript, we present an MR training solution based on requirements collected from experienced medical first responders and technical experts, regular end-user feedback received through the iterative design process used to develop a prototype and feedback from two initial field trials. We discuss key features essential for an effective MR training system, including flexible scenario design, added realism through patient simulator manikins and objective performance assessment. Current technological challenges such as the responsiveness of avatars and the complexity of smart scenario control are also addressed, along with the future potential for integrating artificial intelligence. Furthermore, an advanced analytics and statistics tool that incorporates complex data integration, machine learning for data analysis and visualization techniques for performance evaluation is presented.