Gaze tracking allows hands-free and voice-free interaction with computers, and has gained more use recently in virtual and augmented reality headsets. However, it traditionally uses dwell time for selection tasks, which suffers from the Midas Touch problem. Tongue gestures are subtle, accessible and can be sensed non-intrusively using an IMU at the back of the ear, PPG and EEG. We demonstrate a novel interaction method combining gaze tracking with tongue gestures for gaze-based selection faster than dwell time and multiple selection options. We showcase its usage as a point-and-click interface in three hands-free games and a musical instrument.
Silent speech recognition has emerged as a promising approach for enabling hands-free and discreet interaction with head-worn devices. In this paper, we present QuietSync, a multimodal system that combines inertial measurement unit (IMU) and contact electrode (ExG) signals to achieve accurate silent speech recognition using off-the-shelf devices. QuietSync utilizes an IMU attached to the lower part of the headphones near the ear and strategically places ExG electrodes on the headphones, glasses (nose and behind the ear), and face (for VR applications) to capture subtle movements and muscle activity associated with silent speech production. We conducted a user study with 9 participants and successfully recognized 12 commands with an accuracy of 94.2%. Our system leverages the complementary nature of IMU and ExG signals to enhance the robustness and reliability of silent speech recognition. The IMU captures subtle movements of the jaw and facial muscles, while the ExG electrodes detect low-amplitude surface muscle activity associated with speech production. We show that our system is not affected by the length and speech mannerisms of the commands, and can be fine-tuned for users of varied native languages with only 5 samples. Our findings demonstrate the feasibility of using off-the-shelf head-worn devices to enable silent speech recognition, opening up new possibilities for seamless and discreet interaction with devices such as VR/AR headsets and earables. To the best of our knowledge, QuietSync is the first system to enable silent speech interaction for multiple form factors.
Three-dimensional virtual environments are currently inaccessible to people who are blind, as current screen-reading solutions for 2D content are not fully extensible to achieve the needed embodied spatial presence. Forefronting perceptual agency as key to any access approach for users who are blind, we offer Scene Weaving as an interactional metaphor that allows users to choose how and when they perceive the environment and the people in it. We illustrate how this metaphor can be implemented in an example prototype system. In this interactivity, users can control how and when they perceive a virtual museum environment and people within it through a range of interaction mechanisms.
Interactions with in-vehicle electronic devices can interfere with the primary task of driving and increase crash risk. Interactions with in-vehicle interfaces draw upon visual, auditory, psychomotor, and cognitive resources. Researchers often investigate how these resources interfere with performance through the use of different measurement techniques, particularly doing so in applied settings such as automotive scenarios. The goal of this workshop is to share knowledge with the community regarding the theoretical underpinnings, collection, and filtering of eye tracking data, particularly focusing on gaze distribution (percent on fixations on road center and time/number of gazes on visual field), blink rate, height of gaze in visual field, and pupil size measures of cognitive load within the scope of automotive research. The workshop will describe and demonstrate the theory behind these measures, approaches and issues in regards to collection, and successful methods of filtering data eye tracking measures. The focus of the workshop will be on ensuring participants acquire an understanding of the theoretical reasoning behind the measures shown in the workshop as well as practical knowledge of how to collect and filter data.