21 November 2024 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Accurate and responsive 3D tracking enables interactive and context-aware workspaces, including mixed reality 3D interfaces and collaborative tangible interactions. However, limitations of current tracking mechanisms - line-of-sight occlusion, drifting errors, small working volumes, or instrumentation that requires maintenance - ultimately restrict their adoption. This paper introduces MagDesk, an interactive tabletop workspace capable of real-time 3D tracking of passive magnets embedded in objects. Using a sensing array of 112 low-cost magnetometers underneath a table, our custom-designed signal processing and localization engine enables simultaneous tracking of multiple magnets in 5 degrees of freedom with millimeter accuracy. MagDesk can continuously and robustly track magnets at a maximum height of 600 mm over a 1750 mm (L) × 950 mm (W) table, while achieving an average positional and orientational error of 2.49 mm and 0.72 ° near the table surface and 14.40 mm and 2.25° across the entire sensing range. To demonstrate MagDesk's object-tracking capabilities, this work presents a series of magnetic widgets for tangible interactions and explores two applications - a 3D drawing interface and augmented-reality tabletop games. By instrumenting a regular table surface, MagDesk presents a low-cost and accurate approach to 3D tracking passive objects for home and office environments.
3 March 2025 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Enabling computing systems to detect the objects that people hold and interact with provides valuable contextual information that has the potential to support a wide variety of mobile applications. However, existing approaches either directly instrument users' hands, which can reduce tactile sensation, or are limited in the types of objects and interactions they can detect. This work introduces HandSAW, a wireless wrist-worn device incorporating a Surface Acoustic Wave (SAW) sensor with enhanced bandwidth and signal-to-noise ratio while rejecting through-air sounds. The device features a sealed mass-spring diaphragm positioned on top of the sound port of a MEMS microphone, enabling it to capture SAWs generated by objects and through touch interaction events. This custom-designed wearable platform, paired with a real-time ML pipeline, can distinguish 20 passive object events with >99% per-user accuracy and a 91.6% unseen-user accuracy, as validated through a 16-participant user study. For devices that do not emit SAWs, our active tags enable HandSAW to detect those objects and transmit encoded data using ultrasonic signals. Ultimately, HandSAW provides an easy-to-implement, robust, and cost-effective means for enabling user-object interaction and activity detection.
14 November 2022 in Proceedings of the ACM on Human-Computer Interaction
Pervasive and interactive displays promise to present our digital content seamlessly throughout our environment. However, traditional display technologies do not scale to room-wide applications due to high per-unit-area costs and the need for constant wired power and data infrastructure. This research proposes the use of photochromic paint as a display medium. Applying the paint to any surface or object creates ultra-low-cost displays, which can change color when exposed to specific wavelengths of light. We develop new paint formulations that enable wide area application of photochromic material. Along with a specially modified wide-area laser projector and depth camera that can draw custom images and create on-demand, room-wide user interfaces on photochromic enabled surfaces. System parameters such as light intensity, material activation time, and user readability are examined to optimize the display. Results show that images and user interfaces can last up to 16 minutes and can be updated indefinitely. Finally, usage scenarios such as displaying static and dynamic images, ephemeral notifications, and the creation of on-demand interfaces, such as light switches and music controllers, are demonstrated and explored. Ultimately, the UbiChromics system demonstrates the possibility of extending digital content to all painted surfaces.