27 September 2023 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Sign language is a natural language widely used by Deaf and hard of hearing (DHH) individuals. Advanced wearables are developed to recognize sign language automatically. However, they are limited by the lack of labeled data, which leads to a small vocabulary and unsatisfactory performance even though laborious efforts are put into data collection. Here we propose SignRing, an IMU-based system that breaks through the traditional data augmentation method, makes use of online videos to generate the virtual IMU (v-IMU) data, and pushes the boundary of wearable-based systems by reaching the vocabulary size of 934 with sentences up to 16 glosses. The v-IMU data is generated by reconstructing 3D hand movements from two-view videos and calculating 3-axis acceleration data, by which we are able to achieve a word error rate (WER) of 6.3% with a mix of half v-IMU and half IMU training data (2339 samples for each), and a WER of 14.7% with 100% v-IMU training data (6048 samples), compared with the baseline performance of the 8.3% WER (trained with 2339 samples of IMU data). We have conducted comparisons between v-IMU and IMU data to demonstrate the reliability and generalizability of the v-IMU data. This interdisciplinary work covers various areas such as wearable sensor development, computer vision techniques, deep learning, and linguistics, which can provide valuable insights to researchers with similar research objectives.
12 June 2023 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Sign language builds up an important bridge between the d/Deaf and hard-of-hearing (DHH) and hearing people. Regrettably, most hearing people face challenges in comprehending sign language, necessitating sign language translation. However, state-of-the-art wearable-based techniques mainly concentrate on recognizing manual markers (e.g., hand gestures), while frequently overlooking non-manual markers, such as negative head shaking, question markers, and mouthing. This oversight results in the loss of substantial grammatical and semantic information in sign language. To address this limitation, we introduce SmartASL, a novel proof-of-concept system that can 1) recognize both manual and non-manual markers simultaneously using a combination of earbuds and a wrist-worn IMU, and 2) translate the recognized American Sign Language (ASL) glosses into spoken language. Our experiments demonstrate the SmartASL system's significant potential to accurately recognize the manual and non-manual markers in ASL, effectively bridging the communication gaps between ASL signers and hearing people using commercially available devices.
21 November 2024 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
The accurate assessment of human body posture through wearable technology has significant implications for sports science, clinical diagnostics, rehabilitation, and VR interaction. Traditional methods often require complex setups or are limited by the environment's constraints. In response to these challenges, this paper presents an innovative approach to human posture estimation under complex motion scenarios through the development of an advanced shoe insole embedded with pressure sensors and an Inertial Measurement Unit (IMU). Coupled with a single wrist-mounted IMU, our system facilitates a comprehensive analysis of human biomechanics by integrating physical kinematics modeling based on pressure data with a multi-region human posture estimation network. To enhance the robustness of our system model, we employed large language models to generate virtual human motion sequences. These sequences were utilized to create synthetic IMU data for data augmentation purposes, addressing the challenge of limited real-world data availability and variability. Our approach uniquely combines physical modeling with data-driven techniques to improve the accuracy and reliability of posture estimation. Experimental results demonstrate that our integrated system significantly advances wearable technology for motion analysis. The Mean Per Joint Position Error (MPJPE) was reduced to 7.75 cm, highlighting the effectiveness of our multi-modal modeling and virtual data augmentation in refining posture estimation.
27 September 2023 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Technical advances in the smart device market have fixated smartphones at the heart of our lives, warranting an ever more secure means of authentication. Although most smartphones have adopted biometrics-based authentication, after a couple of failed attempts, most users are given the option to quickly bypass the system with passcodes. To add a layer of security, two-factor authentication (2FA) has been implemented but has proven to be vulnerable to various attacks. In this paper, we introduce VibPath, a simultaneous 2FA scheme that can understand the user's hand neuromuscular system through touch behavior. VibPath captures the individual's vibration path responses between the hand and the wrist with the attention-based encoder-decoder network, authenticating the genuine users from the imposters unobtrusively. In a user study with 30 participants, VibPath achieved an average performance of 0.98 accuracy, 0.99 precision, 0.98 recall, 0.98 f1-score for user verification, and 94.3% accuracy for user identification across five passcodes. Furthermore, we also conducted several extensive studies, including in-the-wile, permanence, vulnerability, usability, and system overhead studies, to assess the practicability and viability of the VibPath from multiple aspects.