The convenient text input system is a pain point for devices such as AR glasses, and it is difficult for existing solutions to balance portability and efficiency. This paper introduces HandPad, the system that turns the hand into an on-the-go touchscreen, which realizes interaction on the hand via human capacitance. HandPad achieves keystroke and handwriting inputs for letters, numbers, and Chinese characters, reducing the dependency on capacitive or pressure sensor arrays. Specifically, the system verifies the feasibility of touch point localization on the hand using the human capacitance model and proposes a handwriting recognition system based on Bi-LSTM and ResNet. The transfer learning-based system only needs a small amount of training data to build a handwriting recognition model for the target user. Experiments in real environments verify the feasibility of HandPad for keystroke (accuracy of 100%) and handwriting recognition for letters (accuracy of 99.1%), numbers (accuracy of 97.6%) and Chinese characters (accuracy of 97.9%).
This paper introduces a novel multi-directional speaker, named MuDiS, which utilizes a parametric array to generate highly focused sound beams in multiple directions. The system capitalizes on air nonlinearity to reproduce sound from ultrasounds, successfully overcoming challenges inherent in traditional parametric arrays, such as transducer size and wavefront shape. It supports three important features simultaneously: independent beams, wide-angle digital steering, and unintended leakage suppression. To address these challenges, we designed a specialized cell structure that connects ultrasonic transducers, redirecting an approximately omnidirectional wavefront with optimal interspacing. An optimization-based algorithm is developed to minimize unintended leakages, and a nonlinear distortion reduction scheme is proposed to enhance sound quality. The paper showcases a prototype demonstrating the system's capabilities as a multidirectional speaker with a wide sound projection angle. Experimental results validate the effectiveness of our approach. The proposed multi-beam projection system rivals the performance of commercially available single-beam projection directional speakers, and improved steering angle and sound fidelity compared to multi-beamforming performance using traditional parametric arrays.
Hand-tracking technology is a pivotal input method in augmented and virtual reality environments, providing enhanced interaction accuracy through micro-gesture recognition. This allows users to control devices with minimal knuckle movements, ensuring privacy and accessibility for individuals with mobility impairments. Building on the foundation of human capacitance, this paper introduces a novel approach termed human capacitance-based micro gesture (HCMG) recognition. This system employs capacitive sensors integrated within the inner lining of a wrist guard, capable of detecting subtle changes in skin-to-electrode contact caused by finger joint movements. Our approach leverages the inherent properties of human capacitance to facilitate accurate and efficient micro-gesture recognition. HCMG achieves recognition of five common micro gestures with an accuracy of 95.0%, providing a promising solution to address the limitations of existing techniques.
4 February 2022 in IEEE Transactions on Mobile Computing
The efficiency of human-computer interaction is greatly hindered by the small size of the touch screens on mobile devices, such as smart phones and watches. This has prompted widespread interest in handwriting recognition systems, which can be divided into active and passive systems. Active systems require additional hardware devices to perceive movements of handwriting or the tracking accuracy is not adequate for handwriting recognition. Passive methods use the acoustic signal of pen rubbing and are susceptible to environmental noise (above 60 $dB$ ). This paper presents a novel handwriting recognition system based on vibration signals detected by the built-in accelerometer of smartphones. The proposed scheme is implemented in three stages: signal segmentation, signal recognition, and word suggestion. VibWriter is highly resistant to interferences since the normal environmental noise (below 70 $dB$ ) will not cause the vibration of the accelerometer. Extensive experiments demonstrated the efficacy of the system in terms of accuracy in letter recognition (75.3%), word recognition (86.4%) and number recognition (79%) in a variety of writing positions under a variety of environmental conditions.
11 December 2024 in IEEE/ACM Transactions on Networking
Acoustic tracking technology, leveraging the ubiquitous presence of speakers and microphones in commercial off-the-shelf (COTS) mobile devices, has become a versatile tool across various applications. However, current phase-based acoustic tracking methods encounter significant limitations in tracking fast movements, thereby restricting their practical utility. This paper identifies three practical challenges to enable fast hand motion tracking using acoustic signals: 1) high mobility, 2) low signal-to-noise ratio (SNR), and 3) variations in hardware frequency response. The high mobility introduces Doppler shift and phase ambiguity which is the primary cause of failure in fast movement tracking, while the latter two factors can further impair the tracking performance in practical scenarios involving high mobility. To address the high mobility issue, we effectively compensate the Doppler shift in the Channel Impulse Response (CIR) for better selection of channel taps and then propose a novel phase derivative approach to mitigate the phase ambiguity. To enhance the real-world robustness, we integrate multiple algorithms including an SNR enhancement algorithm inspired by time-domain beamforming and a hardware frequency response compensation approach that addresses both amplitude and phase distortions. Additionally, an LSTM-based distance reconstruction algorithm is further implemented to correct residual phase noise. Implemented on Android platforms under the name SwiftTrack+, our system demonstrates superior performance in tracking fast movements. Through extensive evaluations, SwiftTrack+proves its efficacy across diverse scenarios, significantly broadening the scope and reliability of acoustic tracking applications.