As interactive devices become pervasive, people are beginning to look for more advanced interaction with televisions in the living room. Interactive television has the potential to offer a very engaging experience. But most common user tasks are still challenging with such systems, such as menu selection or text input, and little work has been done on understanding and supporting the effective design of freehand interaction with a TV in the domestic environment. In this paper, we report two studies investigating freehand gestural interaction with a consumer level sensor that is suitable for TV use scenarios. In the first study, we investigate a range of design factors for tiled layout menu selection, including wearable feedback, push gesture depth, target size and position in motor space. The results show that tactile and audio feedback have no significant effect on user performance and preference, and these results inform potential designs for high selection performance. In the second study, we investigate using freehand gestures for the common TV user task of text input. We design and evaluate two virtual keyboard layouts and three freehand selection methods. Results show that ease of use and error tolerance can both be achieved using a text entry method utilizing a dual circle layout and an expanding target selection technique. Finally, we propose design guidelines for effective, usable and comfortable freehand gestural interaction for interactive TV based on the findings.

Traditional ball sport training methods often lack targeted feedback and personalized guidance, limiting beginners’ ability to develop the rapid responses and precise movements essential for such sports. This study investigates the effectiveness of integrating Visual-Haptic Feedback into simulated ball sport training to enhance body awareness and coordination without increasing cognitive load. By delivering tactile feedback to the wrists and ankles, we aim to improve beginners’ motor skills and overall performance. Our research reviews advancements in human-computer interaction focused on enhancing bodily awareness and discusses the application of tactile feedback in sports training. Experimental results demonstrate that Visual-Haptic Feedback significantly enhances beginners’ body awareness, enabling them to perceive and adjust their movements more accurately without imposing a cognitive burden. Furthermore, this approach effectively standardizes and coordinates motor performance, with a particularly notable improvement in lower limb movements. Additionally, Visual-Haptic Feedback reduces the error rate in executing complex tasks and improves the precision of limb movements. This study enriches the understanding of using Visual-Haptic Feedback in sports training and highlights its potential to improve skills and body coordination, offering valuable insights for future applications and developments in this field.

In complex environments, users frequently need to manage multiple tasks simultaneously, which poses significant challenges for user interface design. For instance, when driving, users must maintain continuous visual attention on the road ahead while also monitoring rearview mirrors and performing shoulder checks. These multitasking scenarios present substantial design challenges in effectively guiding users. To address these challenges, we focus on investigating and designing visual and haptic guidance systems to augment users’ performance. We initially propose the use of visual guidance, specifically employing a dynamic arrow as a guidance technique. Our evaluation shows that dynamic arrows significantly expedite both reaction and selection times. We further introduce and evaluate haptic feedback, which users perceive as more salient than visual guidance, leading to quicker responses when switching from primary to secondary tasks. This allows users to maintain visual attention on the primary task while simultaneously responding effectively to haptic cues. Our findings suggest that multimodal guidance, especially haptic guidance, can enhance both reaction time and user experience in dual-task environments, offering promising practical implications and guidelines for designing more user-friendly interfaces and systems.

Augmented Reality (AR) and Mixed Reality (MR) technology has potential for supporting mobile applications. However, nonvisual interaction modalities are undervalued and underused in AR/MR applications. Visual displays can be ineffective or inappropriate in some situations such as walking or driving. Meanwhile, nonvisual modalities are becoming increasingly important in mobile user experiences. In this paper, we report two studies investigating nonvisual interaction modalities such as audio and haptic displays with mobile AR/MR applications. In the first study, we investigate a range of design factors for haptic and audio displays, including rhythm, amplitude, and their combination in representing tourism information to users with a mobile phone. The results show a main effect for Interaction modality, with identification rates highest for information represented in a combined Haptic-Audio display. In the second study, we investigate target location tasks in 3D space using spatial audio feedback and a head-mounted display. We evaluate several design factors including audio feedback device, volume, rhythm, and the target’s horizontal and vertical position. Results show that the vertical positions are very difficult to locate, and overall our participants prefer audio cues with loud volume and fast rhythm. Finally, we propose practical audio and haptic display design guidelines for AR/MR applications.

Augmented reality (AR) technologies can blend digital and physical space and serve a variety of applications intuitively and effectively. Specifically, wearable AR enabled by optical see-through (OST) AR head-mounted displays (HMDs) might provide users with a direct view of the physical environment containing digital objects. Besides, users could directly interact with three-dimensional (3D) digital artefacts using freehand gestures captured by OST HMD sensors. However, as an emerging user interaction paradigm, freehand interaction with OST AR still requires further investigation to improve user performance and satisfaction. Thus, we conducted two studies to investigate various freehand selection design aspects in OST AR, including target placement, size, distance, position, and haptic feedback on the hand and body. The user evaluation results indicated that 40 cm might be an appropriate target distance for freehand gestural selection. A large target size might lower the selection time and error rate, and a small target size could minimise selection effort. The targets positioned in the centre are the easiest to select, while those in the corners require extra time and effort. Furthermore, we discovered that haptic feedback on the body could lead to high user preference and satisfaction. Based on the research findings, we conclude with design recommendations for effective and comfortable freehand gestural interaction in OST AR.