6 March 2024 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Facial expression recognition (FER) is a crucial task for human-computer interaction and a multitude of multimedia applications that typically call for friendly, unobtrusive, ubiquitous, and even long-term monitoring. Achieving such a FER system meeting these multi-requirements faces critical challenges, mainly including the tiny irregular non-periodic deformation of emotion movements, high variability in facial positions and severe self-interference caused by users' own other behavior. In this work, we present UFace, a long-term, unobtrusive and reliable FER system for daily life using acoustic signals generated by a portable smartphone. We design an innovative network model with dual-stream input based on the attention mechanism, which can leverage distance-time profile features from various viewpoints to extract fine-grained emotion-related signal changes, thus enabling accurate identification of many kinds of expressions. Meanwhile, we propose effective mechanisms to deal with a series of interference issues during actual use. We implement UFace prototype with a daily-used smartphone and conduct extensive experiments in various real-world environments. The results demonstrate that UFace can successfully recognize 7 typical facial expressions with an average accuracy of 87.8% across 20 participants. Besides, the evaluation of different distances, angles, and interferences proves the great potential of the proposed system to be employed in practical scenarios.
26 June 2024 in IEEE Transactions on Mobile Computing
Silent Speech Interfaces (SSI) have been developed to convert silent articulatory gestures into speech, aiding communication in public spaces and assisting individuals with aphasia. Previous SSIs, which rely on wearable devices or cameras, often pose issues like prolonged contact or privacy risks. Recent advancements in acoustic sensing present new opportunities for gesture sensing, but they typically focus on content classification rather than reconstructing audible speech. This results in the loss of crucial speech characteristics such as rate, intonation, and emotion.In this paper, we propose UltraSR, a novel sensing system designed for accurate audible speech reconstruction by analyzing the disturbance of tiny articulatory gestures on reflected ultrasound signals. UltraSR employs a multi-scale feature extraction scheme to aggregate information from multiple views and introduces a new model that maps ultrasound to speech signals, enabling the reconstruction of audible speech from silent gestures.Instead of the laborious collection of massive training data, UltraSR constructs an inverse task to generate virtual gestures from widely available audio (e.g., phone calls) for efficient model training. Additionally, it incorporates a finetuning mechanism using unlabeled data for user adaptation.We implemented UltraSR on a portable smartphone and evaluated it in various environments. Results show that UltraSR can achieve a Character Error Rate (CER) as low as 5.22% and reduce the CER from 80.13% to 6.31% for new users with only 1 hour of ultrasound data, outperforming state-of-the-art acoustic-based approaches while preserving rich speech information.
11 November 2024 in IEEE Transactions on Mobile Computing
Various characteristics of mobile applications (apps) and associated in-app services can reveal potentially-sensitive user information; however, privacy concerns have prompted third-party apps to restrict access to data related to mobile app usage. This paper outlines a novel approach to extracting detailed app usage information by analyzing electromagnetic (EM) signals emitted from mobile devices during app-related tasks. The proposed system, MagSpy, recovers user privacy information from magnetometer readings that do not require access permissions. This EM leakage becomes complex when multiple apps are used simultaneously and is subject to interference from geomagnetic signals generated by device movement. To address these challenges, MagSpy employs multiple techniques to extract and identify signals related to app usage. Specifically, the geomagnetic offset signal is canceled using accelerometer and gyroscope sensor data, and a Cascade-LSTM algorithm is used to classify apps and in-app services. MagSpy also uses CWT-based peak detection and a Random Forest classifier to detect PIN inputs. A prototype system was evaluated on over 50 popular mobile apps with 30 devices. Extensive evaluation results demonstrate the efficacy of MagSpy in identifying in-app services (96% accuracy), apps (93.5% accuracy), and extracting PIN input information (96% top-3 accuracy).
13 May 2024 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
With the growing popularity of smart speakers, numerous novel acoustic sensing applications have been proposed for low-frequency human speech and high-frequency inaudible sounds. Spatial information plays a crucial role in these acoustic applications, enabling various location-based services. However, typically commercial microphone arrays face limitations in spatial perception of inaudible sounds due to their sparse array geometries optimized for low-frequency speech. In this paper, we introduce MetaAng, a system designed to augment microphone arrays by enabling wideband spatial perception across both speech signals and inaudible sounds by leveraging the spatial encoding capabilities of acoustic metasurfaces. Our design is grounded in the fact that, while sensitive to high-frequency signals, acoustic metasurfaces are almost non-responsive to low-frequency speech due to significant wavelength discrepancy. This observation allows us to integrate acoustic metasurfaces with sparse array geometry, simultaneously enhancing the spatial perception of high-frequency and low-frequency acoustic signals. To achieve this, we first utilize acoustic metasurfaces and a configuration optimization algorithm to encode the unique features for each incident angle. Then, we propose an unrolling soft thresholding network that employs neural-enhanced priors and compressive sensing for high-accuracy, high-resolution multi-source angle estimation. We implement a prototype, and experimental results demonstrate that MetaAng maintains robustness across various scenarios, facilitating multiple applications, including localization and tracking.