This paper presents a deep data-driven inertial measurement unit (IMU) online calibration (DUET) method that can compensate for the run-time errors of the accelerometer and gyroscope to improve inertial-based odometry. We design a differential error learning strategy based on the kinematic motion model to train the sensor error compensation model. This strategy allows our method to learn IMU sensor errors, such as scale factors, axis-misalignment, and biases, solely from displacement and orientation increments given by external tracking systems. Then during the odometry computation, the trained model leverages the past inertial data to mitigate the sensor errors and thus reduces the integration errors to reflect the odometry state. The experiments conducted on two public visual-inertial datasets show an average of 20% improvement in the position estimation accuracy of visual-inertial odometry, which is comparable to existing learning-based methods with lower operational complexity.

Over time, the desks of office workers often get cluttered with documents, items, and other objects that are unrelated to the work being done. These objects become visual noise. Visual noise tends to reduce the worker’s concentration on their work. Therefore, it is desirable to have an uncluttered environment to work. We propose DecluttAR, a system that controls the visibility of clutter objects by using a video see-through head-mounted display and an RGB-D camera. Our system first generates a 3D model of the empty desk, and then automatically generates a 3D model of each object that is newly placed on the desk. Then, the user can interactively control each object’s visibility which is superimposed on the corresponding real object. Four types of object appearance adjustments are implemented: normal, grayscale, outline, and transparency. To investigate changes in the ability to concentrate in different appearances, a user study was conducted in which a simple arithmetic task was performed in four appearances. The results show that the number of answers in the arithmetic task for the transparent display increased compared to the normal display in the first half of the task. This result indicates that the participants are able to enter a state of concentration on the task more quickly by making the object invisible. In addition, the questionnaire-based subjective evaluation revealed that the transparent display had a lower workload and a higher user evaluation followed by the outline display.

Experiences of virtual reality (VR) can easily break if the method of evaluating subjective user states is intrusive. Behavioral measures are increasingly used to avoid this problem. One such measure is eye tracking, which recently became more standard in VR and is often used for content-dependent analyses. This research is an endeavor to utilize content-independent eye metrics, such as pupil size and blinks, for identifying mental load in VR users. We generated mental load independently from visuals through auditory stimuli. We also defined and measured a new eye metric, focus offset, which seeks to measure the phenomenon of “staring into the distance” without focusing on a specific surface. In the experiment, VR-experienced participants listened to two native and two foreign language stimuli inside a virtual phone booth. The results show that with increasing mental load, relative pupil size on average increased 0.512 SDs (0.118 mm), with 57% reduced variance. To a lesser extent, mental load led to fewer fixations, less voluntary gazing at distracting content, and a larger focus offset as if looking through surfaces (about 0.343 SDs, 5.10 cm). These results are in agreement with previous studies. Overall, we encourage further research on content-independent eye metrics, and we hope that hardware and algorithms will be developed in the future to further increase tracking stability.

Item-finding tasks due to memory lapse are costly activities commonly experienced by many people. However, conventional systems are not suitable for use in a collaborative environment. Therefore, we propose a multi-functional, pre-registration-free, and 3D location-based item management system. The system has two main functions: registration and search. The automatic registration is performed by image-based item movement recognition from the user’s grasping and placing motions. The registered item movement data comprises the item category, and the start and end locations. We ensure privacy protection by storing item movement data without images. Also, we provide a user interaction to refuse to share the items with other users. The search is based on the item list or item location. The location-based search is performed by specifying where the user last saw the item. To optimize and test the performance of the system, we first performed parameter optimization and then conducted a user study investigating the performance of a search task. The parameter optimization performed in the registration system led to the discovery of optimal values that are difficult to reach empirically. The search experiment showed that the proposed system’s search and guidance functions are effective as an assistance system for finding items, both in terms of search time and user experience. Overall, our system demonstrated the potential to be a useful assistance system for managing items in a shared space. We further discuss the possibility of further exploiting the limited registered information by treating item location as an identifier of the moved item.

Systems with occlusion capabilities, such as those used in vision augmentation, image processing, and optical see-through head-mounted display (OST-HMD), have gained popularity. Achieving precise (hard-edge) occlusion in these systems is challenging, often requiring complex optical designs and bulky volumes. On the other hand, utilizing a single transparent liquid crystal display (LCD) is a simple approach to create occlusion masks. However, the generated mask will appear defocused (soft-edge) resulting in insufficient blocking or occlusion leakage. In our work, we delve into the perception of soft-edge occlusion by the human visual system and present a preference-based optimal expansion method that minimizes perceived occlusion leakage. In a user study involving 20 participants, we made a noteworthy observation that the human eye perceives a sharper edge blur of the occlusion mask when individuals see through it and gaze at a far distance, in contrast to the camera system's observation. Moreover, our study revealed significant individual differences in the perception of soft-edge masks in human vision when focusing. These differences may lead to varying degrees of demand for mask size among individuals. Our evaluation demonstrates that our method successfully accounts for individual differences and achieves optimal masking effects at arbitrary distances and pupil sizes.

Vision-induced gustatory manipulation interfaces can help people with dietary restrictions feel as if they are eating what they want by modulating the appearance of the alternative foods they are eating in reality. However, it is still unclear whether vision-induced gustatory change persists beyond a single bite, how the sensation changes over time, and how it varies among individuals from different cultural backgrounds. The present paper reports on a user study conducted to answer these questions using a generative adversarial network (GAN)-based real-time image-to-image translation system. In the user study, 16 participants were presented somen noodles or steamed rice through a video see-through head mounted display (HMD) both in two conditions; without or with visual modulation (somen noodles and steamed rice were translated into ramen noodles and curry and rice, respectively), and brought food to the mouth and tasted it five times with an interval of two minutes. The results of the experiments revealed that vision-induced gustatory manipulation is persistent in many participants. Their persistent gustatory changes are divided into three groups: those in which the intensity of the gustatory change gradually increased, those in which it gradually decreased, and those in which it did not fluctuate, each with about the same number of participants. Although the generalizability is limited due to the small population, it was also found that non-Japanese and male participants tended to perceive stronger gustatory manipulation compared to Japanese and female participants. We believe that our study deepens our understanding and insight into vision-induced gustatory manipulation and encourages further investigation.