When designing extended reality (XR) applications, it is important to consider multimodal interaction techniques, which employ several human senses simultaneously. Multimodal interaction can transform how people communicate remotely, practice for tasks, entertain themselves, process information visualizations, and make decisions based on the provided information. This scoping review summarized recent advances in multimodal interaction technologies for head-mounted display-based (HMD) XR systems. Our purpose was to provide a succinct, yet clear, insightful, and structured overview of emerging, underused multimodal technologies beyond standard video and audio for XR interaction, and to find research gaps. The review aimed to help XR practitioners to apply multimodal interaction techniques and interaction researchers to direct future efforts towards relevant issues on multimodal XR. We conclude with our perspective on promising research avenues for multimodal interaction technologies.

Methods of information presentation in the automotive space have been evolving continuously in recent years. As technology pushes forward the boundaries of what is possible, automobile manufacturers are trying to keep up with the current trends. Traditionally, the often-long development and quality control cycles of the automotive sector ensured slow yet steady progress. However, the exponential advancement in the mobile and hand-held computing space seen in the last 10 years has put immense pressure on automobile manufacturers to try to catch up. For this reason, we now see manufacturers trying to explore new techniques for in-vehicle interaction (IVI), which were ignored in the past. However, recent attempts have either simply extended the interaction model already used in mobile or handheld computing devices or increased visual-only presentation-of-information with limited expansion to other modalities (i.e. audio or haptics). This is also true for system interaction which generally happens within complex driving environments, making the primary task of a driver (driving) even more challenging. Essentially, there is an inherent need to design and research IVI systems that complement and natively support a multimodal interaction approach, providing all the necessary information without increasing driver’s cognitive load or at a bare minimum his/her visual load. In this research we focus on the key elements of IVI system: touchscreen interaction by developing prototype devices that can complement the conventional visual and auditory modalities in a simple and natural manner. Instead of adding primitive touch feedback cues to increase redundancy or complexity, we approach the issue by looking at the current requirements of interaction and complementing the existing system with natural and intuitive input and output methods, which are less affected by environmental noise than traditional multimodal systems.

The current food production system is unsustainable, necessitating a shift towards plant-based diets. Nutritious options fulfill basic needs but may not satisfy hedonic ones. Our novel approach is to promote healthier eating habits without compromising on the pleasantness of eating by using extended reality technologies and multimodal interaction. We present a multisensory augmentation system integrating augmentations in olfaction, touch, and vision. We studied the experience of eating plant-based balls and meatballs. In an experiment with 40 participants, haptic and visual augmentations were found to have significant effects: augmented meatballs and plant-based balls were perceived as bigger and heavier compared to non-augmented versions. However, olfactory augmentation did not produce a similar effect: participants did not notice a stronger aroma with augmented balls compared to non-augmented balls, and the augmented plant-based version had a less appealing scent than its non-augmented counterpart. Moreover, the findings of the study indicate that our multisensory augmentation system had no significant effect on taste perception.

Little is known about how plant-based products influence satiation compared to corresponding meat-based products. As augmented reality (AR) intensifies sensory experiences, it was hypothesized to improve satiation. This study compared satiation between intake of meatballs and plant-based balls and plant-based balls intensified with AR for visual, olfactory, and haptic sensory properties. Intake order of the meatballs, plant-based balls, and augmented plant-based balls, eaten on separate days, was randomized. Satiation was measured from twenty-eight non-obese adults as ad libitum intake of the balls and extra snacks, and as subjective appetite sensations. Liking and wanting to eat the products were also investigated. There were no differences between the products in satiation. Before tasting the augmented plant-based balls were less liked than the meatballs (p = 0.002) or plant-based balls (p = 0.046), but after eating the first ball or eating the ad libitum number of balls the differences in liking disappeared. Wanting evaluations were similar for each product and decreased during eating (p < 0.001). A group of participants susceptible to AR was found (n = 11), described by decreased intake when augmentation was applied. Among the sub-group, wanting to eat the augmented balls was lower before tasting (p = 0.019) and after eating the first ball (p = 0.002) and appetite was less suppressed after eating the balls ad libitum (p = 0.01), when compared to non-susceptible participants. We conclude that meatballs and plant-based balls were equal in inducing satiation, and multisensory augmentation did not influence satiation. However, the augmentation decreased liking evaluations before tasting. Further studies are needed to explore differences between consumer groups in susceptibility to augmentation.