Natural eye movements have primarily been studied for over-learned activities such as tea-making, sandwich-making, and hand-washing, which have a fixed sequence of associated actions. These studies demonstrate a sequential activation of low-level cognitive schemas facilitating task completion. However, whether these action schemas are activated in the same pattern when a task is novel and a sequence of actions must be planned in the moment is unclear. Here, we recorded gaze and body movements in a naturalistic task to study action-oriented gaze behavior. In a virtual environment, subjects moved objects on a life-size shelf to achieve a given order. To compel cognitive planning, we added complexity to the sorting tasks. Fixations aligned with the action onset showed gaze as tightly coupled with the action sequence, and task complexity moderately affected the proportion of fixations on the task-relevant regions. Our analysis revealed that gaze fixations were allocated to action-relevant targets just in time. Planning behavior predominantly corresponded to a greater visual search for task-relevant objects before the action onset. The results support the idea that natural behavior relies on the frugal use of working memory, and humans refrain from encoding objects in the environment to plan long-term actions. Instead, they prefer just-in-time planning by searching for action-relevant items at the moment, directing their body and hand to it, monitoring the action until it is terminated, and moving on to the following action.

With the further development of highly automated vehicles, drivers will engage in non-related tasks while being driven. Still, drivers have to take over control when requested by the car. Here, the question arises, how potentially distracted drivers get back into the control-loop quickly and safely when the car requests a takeover. To investigate effective human–machine interactions, a mobile, versatile, and cost-efficient setup is needed. Here, we describe a virtual reality toolkit for the Unity 3D game engine containing all the necessary code and assets to enable fast adaptations to various human–machine interaction experiments, including closely monitoring the subject. The presented project contains all the needed functionalities for realistic traffic behavior, cars, pedestrians, and a large, open-source, scriptable, and modular VR environment. It covers roughly 25 km², a package of 125 animated pedestrians, and numerous vehicles, including motorbikes, trucks, and cars. It also contains all the needed nature assets to make it both highly dynamic and realistic. The presented repository contains a C++ library made for LoopAR that enables force feedback for gaming steering wheels as a fully supported component. It also includes all necessary scripts for eye-tracking in the used devices. All the main functions are integrated into the graphical user interface of the Unity^® editor or are available as prefab variants to ease the use of the embedded functionalities. This project’s primary purpose is to serve as an open-access, cost-efficient toolkit that enables interested researchers to conduct realistic virtual reality research studies without costly and immobile simulators. To ensure the accessibility and usability of the mentioned toolkit, we performed a user experience report, also included in this paper.

Semi-autonomous vehicles (AVs) enable drivers to engage in non-driving tasks but require them to be ready to take control during critical situations. This “out-of-the-loop” problem demands a quick transition to active information processing, raising safety concerns and anxiety. Multimodal signals in AVs aim to deliver take-over requests and facilitate driver–vehicle cooperation. However, the effectiveness of auditory, visual, or combined signals in improving situational awareness and reaction time for safe maneuvering remains unclear. This study investigates how signal modalities affect drivers’ behavior using virtual reality (VR). We measured drivers’ reaction times from signal onset to take-over response and gaze dwell time for situational awareness across twelve critical events. Furthermore, we assessed self-reported anxiety and trust levels using the Autonomous Vehicle Acceptance Model questionnaire. The results showed that visual signals significantly reduced reaction times, whereas auditory signals did not. Additionally, any warning signal, together with seeing driving hazards, increased successful maneuvering. The analysis of gaze dwell time on driving hazards revealed that audio and visual signals improved situational awareness. Lastly, warning signals reduced anxiety and increased trust. These results highlight the distinct effectiveness of signal modalities in improving driver reaction times, situational awareness, and perceived safety, mitigating the “out-of-the-loop” problem and fostering human–vehicle cooperation.

Autonomous vehicles represent a significant development in our society, and their acceptance will largely depend on trust. This study investigates strategies to increase trust and acceptance by making the cars’ decisions. For this purpose, we created a virtual reality (VR) experiment with a self-explaining autonomous car, providing participants with verbal cues about crucial traffic decisions. First, we investigated attitudes toward self-driving cars among 7850 participants using a simplified version of the Technology Acceptance Model (TAM) questionnaire. Results revealed that female participants are less accepting than male participants, and that there is a general decline among all genders. Otherwise in general, a self-explaining car has a positive impact on trust and perceived usefulness. Surprisingly, it adversely affected the intention to use and perceived ease of use. This entails dissociation of trust from the other items of the questionnaire. Second, we analyzed behavioral of 26 750 participants to investigate the effect of self-explaining systems on head movements during the VR drive. We observed significant differences in head movements during the critical events and the baseline periods of the drive between the three driving conditions. Additionally, we demonstrated positive correlations between head movement parameters and the TAM scores, where trust showed the lowest correlation. This provides further evidence of the dissociation of trust from other TAM items. These findings illustrate the benefits of combining subjective questionnaire data with objective behavioral data. Overall, the outcomes indicate a partial dissociation of self-reported trust from intention to use and objective behavioral data.

Autonomous vehicles as cognitive agents will be an important use case of artificial intelligence in modern societies. Investigating how to increase acceptance and trust, we created a self-explaining car, informing passengers before actions in virtual reality. This study investigates the attitude towards self-driving cars with data from 7850 participants. We show how gender and age affect the attitude towards autonomous vehicles, resulting in female participants being generally less trusting of overall conditions than male participants and a general decrease of acceptance with increasing age. Surprisingly, a self-explaining car providing the passenger with crucial traffic information, although it has a positive impact on trust but influences the intention of using such a car negatively. Therefore, we argue for a highly individualizable in-car communication that meets the adversarial needs of different demographic groups to enable human-machine interactions that foster safe traffic behavior and increase trust and the willingness to use such technology.