As eyewear devices such as smart glasses become more common, it is important to provide input methods that can be used at all times for such situations and people. Silent speech interaction (SSI) has the potential to be useful as a hands-free input method for various situations and people, including those who have difficulty with voiced speech. However, previous methods have involved sensor devices that are difficult to use anytime and anywhere. We propose a method for SSI that involves using an eyewear device equipped with infrared distance sensors. The proposed method measures facial skin movements associated with a speech from the infrared distance sensor mounted on an eyewear device and recognizes silent speech commands by applying machine learning to the time-series sensor data. The proposed method was applied to a prototype system including a sensor device consisting of eyewear and ear-mounted microphones to measure the movements of the cheek, jaw joint, and jaw. From the evaluation results, F-measure was 0.90 with five speech commands and 0.83 with ten longer speech commands, which indicates the feasibility of the proposed method as a simple hands-free input interface. Our study provides the first wearable sensing method that can easily apply SSI functions to eyewear devices.
Simple hands-free input methods using ear accessories have been proposed to broaden the range of scenarios in which information devices can be operated without hands. Although many previous studies use canal-type earphones, few studies focused on the following two points: (1) A method applicable to ear accessories other than canal-type earphones. (2) A method enabling various ear accessories with different styles to have the same hands-free input function. To realize these two points, this study proposes a method to recognize the user’s facial gesture using an infrared distance sensor attached to the ear accessory. The proposed method detects skin movement around the ear and face, which differs for each facial expression gesture. We created a prototype system for three ear accessories for the root of the ear, earlobe, and tragus. The evaluation results for nine gestures and 10 subjects showed that the F-value of each device was 0.95 or more, and the F-value of the pattern combining multiple devices was 0.99 or more, which showed the feasibility of the proposed method. Although many ear accessories could not interact with information devices, our findings enable various ear accessories with different styles to have eye-free and hands-free input ability based on facial gestures.
Memorization is necessary for various fields, such as language learning in the field of education. While memorization learning is often tedious and demotivating due to requiring conscious effort, few support approaches improve memorization unconsciously with low conscious effort. In this study, we propose a method, Mindless Memorization Booster, which improves users’ memorization unconsciously by visual stimuli of modulating the visual interface. This method is based on previous findings that the modulation of perceptual stimuli arouses attention/concentration. When the user looks at the memorization target, the proposed method presents a change in visual interface (e.g., changes in memorization target size, background color, and visual icon movement) to cause a psychological phenomenon of affecting the user’s attention and concentration, aiming at enhancing the memorization unconsciously. A prototype system of the proposed method was implemented for an English vocabulary memorization learning application. The evaluation results showed that the user’s memorization result was affected by the proposed method, and the speed of recall (i.e., outputs of the memorization word from the brain) increased by about 1 s per one memorization word without causing a negative affection on the number of correct answers for memorization. This result indicated the feasibility of the proposed method for memorization learning support. Our findings are helpful for designing visual information interfaces that consider the phenomena affecting the user’s memorization and promote memorization learning unconsciously.
With the spread of eyewear devices, people are increasingly using information devices in various everyday situations. In these situations, it is important for eyewear devices to have eye-based interaction functions for simple hands-free input at a low cost. This paper proposes a gaze movement recognition method for simple hands-free interaction that uses eyewear equipped with an infrared distance sensor. The proposed method measures eyelid skin movement using an infrared distance sensor inside the eyewear and applies machine learning to the time-series sensor data to recognize gaze movements (e.g., up, down, left, and right). We implemented a prototype system and conducted evaluations with gaze movements including factors such as movement directions at 45-degree intervals and the movement distance difference in the same direction. The results showed the feasibility of the proposed method. The proposed method recognized 5 to 20 types of gaze movements with an F-value of 0.96 to 1.0. In addition, the proposed method was available with a limited number of sensors, such as two or three, and robust against disturbance in some usage conditions (e.g., body vibration, facial expression change). This paper provides helpful findings for the design of gaze movement recognition methods for simple hands-free interaction using eyewear devices at a low cost.
Systems presenting information that encourages competition by using rankings and scores (hereafter referred to as competition information) have become widespread to support behavioral change. However, users without high levels of motivation, such as behavior change support targets, do not necessarily benefit from competition information. In this study, we propose a method to control the psychological effects caused by competition information to support behavior change. We implemented a competition information presentation system using step counts logs to support increasing one’s daily steps. We designed two patterns of competition information considering psychological effects. One is likely to have good effects, using three mechanisms to easily obtain results corresponding to the effort, make closely matched rivals with similar abilities, and pay attention to a small number of rivals. The other is unlikely to have positive effects and may potentially even have negative ones, using a mechanism that brings about the opposite results of the former pattern. We evaluated 42 participants with low levels of motivation over six weeks. The results showed that the former information pattern increased participants’ step counts by about 1000 steps per day, and the latter information pattern did not lead to an increase. We confirmed the feasibility of the proposed method and discussed the possibility of the appropriate use and potential abuse of such techniques for manipulating motivation. Our study can be helpful in designing a competition information presentation system considering psychological effects.
As eyewear devices such as smart glasses become more common, it is important to provide input methods that can be used at all times for such situations and people. Silent speech interaction (SSI) has the potential to be useful as a hands-free input method for various situations and people, including those who have difficulty with voiced speech. However, previous methods have involved sensor devices that are difficult to use anytime and anywhere. We propose a method for SSI that involves using an eyewear device equipped with infrared distance sensors. The proposed method measures facial skin movements associated with speech from the infrared distance sensor mounted on an eyewear device and recognizes silent speech commands by applying machine learning to time series sensor data. The proposed method was applied to a prototype system including a sensor device consisting of eyewear and ear-mounted microphones to measure the movements of the cheek, jaw joint, and jaw. Evaluations 1 and 2 showed that five speech commands could be recognized with an F value of 0.90 and ten longer speech commands with an F value of 0.83. Evaluation 3 showed how the recognition accuracy changes with the combination of sensor points. Evaluation 4 examined whether the proposed method can be used for a larger number of speech commands with 21 commands by using deep learning LSTM and a combination of DTW and kNN. Evaluation 5 examined the recognition accuracy in some situations affecting recognition accuracy such as re-attaching devices and walking. These results show the feasibility of the proposed method for a simple hands-free input interface, such as with media players and voice assistants. Our study provides the first wearable sensing method that can easily apply SSI functions to eyewear devices.