13 May 2024 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Modern information querying systems are progressively incorporating multimodal inputs like vision and audio. However, the integration of gaze --- a modality deeply linked to user intent and increasingly accessible via gaze-tracking wearables --- remains underexplored. This paper introduces a novel gaze-facilitated information querying paradigm, named G-VOILA, which synergizes users' gaze, visual field, and voice-based natural language queries to facilitate a more intuitive querying process. In a user-enactment study involving 21 participants in 3 daily scenarios (p = 21, scene = 3), we revealed the ambiguity in users' query language and a gaze-voice coordination pattern in users' natural query behaviors with G-VOILA. Based on the quantitative and qualitative findings, we developed a design framework for the G-VOILA paradigm, which effectively integrates the gaze data with the in-situ querying context. Then we implemented a G-VOILA proof-of-concept using cutting-edge deep learning techniques. A follow-up user study (p = 16, scene = 2) demonstrates its effectiveness by achieving both higher objective score and subjective score, compared to a baseline without gaze data. We further conducted interviews and provided insights for future gaze-facilitated information querying systems.
3 September 2025 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
A smart ring is a wearable electronic device in the form of a ring that incorporates diverse sensors and computing technologies to perform a variety of functions. Designed for use with fingers, smart rings are capable of sensing more subtle and abundant hand movements, thus making them a good platform for interaction. Meanwhile, fingers are abundant with blood vessels and nerve endings and accustomed to wearing rings, providing an ideal site for continuous health monitoring through smart rings, which combine comfort with the ability to capture vital biometric data, making them suitable for all-day wear. We collected in total of 206 smart ring-related publications and conducted a systematic literature review. We provide a taxonomy regarding the sensing and feedback modalities, applications, and phenomena. We review and categorize these literatures into four main areas: (1) interaction - input, (2) interaction - output, (3) passive sensing - in body feature, (4) passive sensing - out body activity. This comprehensive review highlights the current advancements within the field of smart ring and identifies potential areas for future research.
30 September 2026 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Collecting richly labeled wearable activity data in everyday settings remains difficult because retrospective annotation is costly and often imprecise. Prior data collection apps rely on a labor-intensive self-reporting strategy and primarily treat participants as crowd labelers. We present Pebbl, a feasibility-stage system that incentivizes in-situ labeling through opportunistic crowdsensing. Pebbl lets users author trigger-action recipes on a smartphone and receive just-in-time reminders for beneficial actions when a trigger is detected. In the prototype, triggers are a limited set with four common audio cues, while actions are described in open-vocabulary natural language. Each confirmed execution yields a short sensor window with explicit start/end boundaries and a user-authored action label. We evaluate Pebbl through an expert workshop with wearable Human Activity Recognition (HAR) researchers ( N = 6), a within-subject in-lab study ( N = 21), and a pilot deployment ( N = 8). Experts viewed the approach as lower burden and more ecologically valid than common labeling workflows. In the lab, Pebbl produced reliable execution logs under controlled conditions (recall = 97.30%, precision = 97.15%) and was preferred over comparison workflows on perceived burden and confidence. The pilot deployment shows that the interaction and sensing pipeline can function in free-living use, while surfacing practical constraints such as false triggers and context dependence. Overall, Pebbl represents a step toward a low-burden, distributable collection approach of user-contributed wearable activity data.