Access to complex graphical information is essential when connecting blind and visually impaired (BVI) people with the world. Tactile graphics readers enable access to graphical data through audio-tactile user interfaces (UIs), but these have yet to mature. A challenging task for blind people is locating specific elements–areas in detailed tactile graphics. To this end, we developed three audio navigation UIs that dynamically guide the user’s hand to a specific position using audio feedback. One is based on submarine sonar sounds, another relies on the target’s coordinate plan x and y-axis, and the last uses direct voice instructions. The UIs were implemented in the Tactonom Reader device, a new tactile graphic reader that enhances swell paper graphics with pinpointed audio explanations. To evaluate the effectiveness of the three different dynamic navigation UIs, we conducted a within-subject usability test that involved 13 BVI participants. Beyond comparing the effectiveness of the different UIs, we observed and recorded the interaction of the visually impaired participants with the different navigation UI to further investigate their behavioral patterns during the interaction. We observed that user interfaces that required the user to move their hand in a straight direction were more likely to provoke frustration and were often perceived as challenging for blind and visually impaired people. The analysis revealed that the voice-based navigation UI guides the participant the fastest to the target and does not require prior training. This suggests that a voice-based navigation strategy is a promising approach for designing an accessible user interface for the blind.

Sensory augmentation provides novel opportunities to broaden our knowledge of human perception through external sensors that record and transmit information beyond natural perception. To assess whether such augmented senses affect the acquisition of spatial knowledge during navigation, we trained a group of 27 participants for six weeks with an augmented sense for cardinal directions called the feelSpace belt. Then, we recruited a control group that did not receive the augmented sense and the corresponding training. All 53 participants first explored the Westbrook virtual reality environment for two and a half hours spread over five sessions before assessing their spatial knowledge in four immersive virtual reality tasks measuring cardinal, route, and survey knowledge. We found that the belt group acquired significantly more accurate cardinal and survey knowledge, which was measured in pointing accuracy, distance, and rotation estimates. Interestingly, the augmented sense also positively affected route knowledge, although to a lesser degree. Finally, the belt group reported a significant increase in the use of spatial strategies after training, while the groups’ ratings were comparable at baseline. The results suggest that six weeks of training with the feelSpace belt led to improved survey and route knowledge acquisition. Moreover, the findings of our study could inform the development of assistive technologies for individuals with visual or navigational impairments, which may lead to enhanced navigation skills and quality of life.

Current research strives to investigate cognitive processes under natural conditions. Virtual reality and EEG are promising techniques combining naturalistic settings with close experimental control. However, many questions and technical challenges remain, e.g., are saccade onsets a suitable replacement of fixation onsets as key events in continuous gaze trajectories ( Amme et al., 2024), and consequently, can VR capture differences across different stimulus categories associated with varying saccade durations? To address both questions, we investigate the N170 face effect in humans (14 males, 19 females, zero diverse) using a free-viewing and free-movement immersive VR study that contained houses, various background stimuli, and, notably, static and moving pedestrians to study face perception under naturalistic conditions. Our results show that aligning trials to saccade onsets leads to more well-defined ERPs than fixation onsets, especially for the P100 component, demonstrating that saccade-onset ERPs are a better-suited analysis method for this type of experiment. Furthermore, we observe an evolution of category-based differences, i.e., face versus background saccade-onset ERPs, compatible with previous reports but extending in a large temporal window and including all electrode sites at different points in time. In summary, employing VR, EEG, and eye-tracking to investigate differences across fixation categories provides insights into the relevance of saccadic onsets as event triggers and enhances our understanding of cognitive processes in naturalistic settings.

Sensory augmentation offers a novel opportunity to broaden our knowledge of human perception through the use of external sensors that record information that humans cannot perceive naturally. This information is then translated in a meaningful way to be presented through an inherent sensory modality. To assess whether such augmented senses affect the acquisition of spatial knowledge during navigation, we trained a group of 26 participants for six weeks with an augmented sense for cardinal directions, called the feelSpace belt. In order to assess perceptual and behavioral changes, we compared the use of spatial strategies and the spatial navigation performance of the belt group to the performance of a control group that did not receive the augmented sense or the training. To this end, all participants explored the virtual reality environment Westbrook for 150 minutes in total. Then, the participants’ spatial knowledge of the virtual reality city was assessed subjectively with the FRS questionnaire and behaviorally in four immersive virtual reality tasks within Westbrook. The belt group reported a significant increase in the use of spatial strategies after training, while the groups’ ratings were comparable at baseline. The finding is consistent with previous studies showing cognitive and subjective changes to space perception as a consequence of wearing the feelSpace belt for an extended time period. Behaviorally, we found that the belt group acquired significantly more accurate knowledge of cardinal directions and of survey knowledge, measured in pointing accuracy, distance and rotation estimates. Finally, we found that the augmented sense also positively affects route knowledge but to a lesser degree. These results suggest that six weeks of training with the feelSpace belt lead to an improved acquisition of survey and route knowledge as reflected by an increased accuracy in survey and route knowledge tasks in a virtual city.

Current research strives to investigate cognitive processes under natural conditions. Virtual reality and EEG are promising techniques combining naturalistic settings with close experimental control. However, many questions and technical challenges remain, e.g., are fixation or saccade onsets a suitable replacement as key events in continuous gaze trajectories (Amme et al., 2024), and can VR effectively capture differences across experimental conditions (Rossion & Jacques, 2008)? To address both questions, we investigate the N170 face effect in a free-viewing immersive VR study that contained houses, various background stimuli, and, notably, static and moving pedestrians to study face perception under naturalistic conditions. Our results show that aligning trials to saccade-onset leads to more well-defined ERPs, especially for the P100 component, and support that saccade-onset ERPs are the better-suited analysis method than fixation-onset ERPs for this type of experiment. Further, we observe an evolution of condition-based differences, i.e., face vs. background fixations, compatible with previous reports but extending in a large temporal window and including all electrode sites at different points in time. In summary, experiments combining VR, EEG, and eye-tracking provide further insights into the processing of faces and the relevance of saccadic onsets as event triggers under natural conditions. Significance Statement: With the effort of investigating and understanding cognitive processes under naturalistic conditions, combining virtual reality and EEG can be fruitful to implement free-viewing studies. The current work combines these technologies to explore key challenges in the context of face perception in an immersive virtual environment. Our results show that saccade-onset ERPs yield more precise measurements when analyzing continuous eye-tracking data than fixation onsets. Furthermore, when processing face compared to background stimuli, distinct temporal patterns encompassing all electrode sites can be observed, offering new insights into face perception. Overall, this work highlights the potential of integrating VR and EEG to advance our understanding of cognitive processes in naturalistic settings.

Vision is an important sense for spatial navigation, yet little is known about the role of visual behavior during spatial knowledge acquisition and its link to spatial navigation performance. In this study, participants freely explored the immersive virtual reality city of Westbrook while eye-tracking data were collected. Subsequently, participants’ spatial knowledge was tested in an immersive pointing-to-building task. To address the freedom of movement during the spatial exploration, we transformed the eye-tracking data into gaze graphs and applied a graph-theoretical analysis. We assessed gaze graph architecture and identified gaze-graph-defined landmarks, finding that their properties were consistent with key results from our earlier work. Furthermore, we investigated which factors contribute to the observed individual differences in spatial navigation performance. We found that global gaze graph measures, specifically gaze graph diameter, could explain 40% of the observed individual differences in participants’ mean task performance. In comparison, a model based on participants’ responses to a self-report questionnaire was not predictive of such differences. Consequently, we suggest that visual behavior, as reflected in global gaze graph measures, is a strong predictor of individual performance differences. Our results not only contribute to a better understanding of landmarks and individual differences in spatial navigation but also outline the potential of vision and gaze-graph-based spatial navigation research.