When designing extended reality (XR) applications, it is important to consider multimodal interaction techniques, which employ several human senses simultaneously. Multimodal interaction can transform how people communicate remotely, practice for tasks, entertain themselves, process information visualizations, and make decisions based on the provided information. This scoping review summarized recent advances in multimodal interaction technologies for head-mounted display-based (HMD) XR systems. Our purpose was to provide a succinct, yet clear, insightful, and structured overview of emerging, underused multimodal technologies beyond standard video and audio for XR interaction, and to find research gaps. The review aimed to help XR practitioners to apply multimodal interaction techniques and interaction researchers to direct future efforts towards relevant issues on multimodal XR. We conclude with our perspective on promising research avenues for multimodal interaction technologies.

Virtual reality devices are used for several application domains, such as medicine, entertainment, marketing and training. A handheld controller is the common interaction method for direct object manipulation in virtual reality environments. Using hands would be a straightforward way to directly manipulate objects in the virtual environment if hand-tracking technology were reliable enough. In recent comparison studies, hand-based systems compared unfavorably against the handheld controllers in task completion times and accuracy. In our controlled study, we compare these two interaction techniques with a new hybrid interaction technique which combines the controller tracking with hand gestures for a rigid object manipulation task. The results demonstrate that the hybrid interaction technique is the most preferred because it is intuitive, easy to use, fast, reliable and it provides haptic feedback resembling the real-world object grab. This suggests that there is a trade-off between naturalness, task accuracy and task completion time when using these direct manipulation interaction techniques, and participants prefer to use interaction techniques that provide a balance between these three factors.

In medical surgery planning, virtual reality (VR) provides a working environment, where 3D images of the operation area can be utilized. VR allows 3D imaging data to be viewed in a more realistic 3D environment, reducing perceptual problems and increasing spatial understanding. In the present experiment, we compared a mouse, hand tracking, and a combination of a VR stylus and a grab-enabled VR controller as interaction methods in VR. The purpose was to investigate the suitability of the methods in VR for object manipulation and marking tasks in medical surgery planning. The tasks required interaction with 3D objects and high accuracy in the creation of landmarks. The combination of stylus and controller was the most preferred interaction method. According to subjective results, it was considered as the most appropriate because it allows the manipulation of objects in a way that is similar to the use of bare hands. In the objective results, the mouse interaction method was the most accurate.

Many virtual reality (VR) applications use teleport for locomotion. The non-continuous locomotion of teleport is suited for VR controllers and can minimize simulator sickness, but it can also reduce spatial awareness compared to continuous locomotion. Our aim was to create continuous, controller-based locomotion techniques that would support spatial awareness. We compared the new techniques, slider and grab, with teleport in a task where participants counted small visual targets in a VR environment. Task performance was assessed by asking participants to report how many visual targets they found. The results showed that slider and grab were significantly faster to use than teleport, and they did not cause significantly more simulator sickness than teleport. Moreover, the continuous techniques provided better spatial awareness than teleport.

Smartwatches are widely available and increasingly adopted by consumers. The most common way of interacting with smartwatches is either touching a screen or pressing buttons on the sides. However, such techniques require using both hands. We propose glance awareness and active gaze interaction as alternative techniques to interact with smartwatches. We will describe an experiment conducted to understand the user preferences for visual and haptic feedback on a "glance" at the wristwatch. Following the glance, the users interacted with the watch using gaze gestures. Our results showed that user preferences differed depending on the complexity of the interaction. No clear preference emerged for complex interaction. For simple interaction, haptics was the preferred glance feedback modality.

Purpose: In dental implantology, the optimal placement of dental implants is important to meet functional and aesthetic requirements. Planning dental implants in virtual three-dimensional (3D) environment is possible using virtual reality (VR) technologies. The three-dimensional stereoscopic virtual reality environment offers advantages over three-dimensional projection on a two-dimensional display. The use of voice commands in virtual reality environment to replace button presses and other simple actions frees the user’s hands and eyes for other tasks. Methods: Six dentomaxillofacial radiologists experimented using a prototype version of a three-dimensional virtual reality implant planning tool, and used two different tool selection methods, using either only button presses or also voice commands. We collected objective measurements of the results and subjective data of the participant experience to compare the two conditions. Results: The tool was approved by the experts and they were able to do the multiple-implant planning satisfactorily. The radiologists liked the possibility to use the voice commands. Most of the radiologists were willing to use the tool as part of their daily work routines. Conclusion: The voice commands were useful, natural, and accurate for mode change, and they could be expanded to other tasks. Button presses and the voice commands should be both available and used in parallel. The input methods can be further improved based on the expert comments.