22 August 2024 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
The smart glasses market has witnessed significant growth in recent years. The interaction of commercial smart glasses mostly relies on the hand, which is unsuitable for scenarios where both hands are occupied. In this paper, we propose EyeGesener, an eye gesture listener for smart glasses interaction using acoustic sensing. To mitigate the Midas touch problem, we meticulously design eye gestures for interaction as two intentional consecutive saccades in a specific direction without visual dwell. The proposed system is a glass-mounted acoustic sensing system with two pairs of commercial speakers and microphones to sense eye gestures. To capture the subtle movements of the eyelid and surrounding skin induced by eye gestures, we design an Orthogonal Frequency Division Multiplexing (OFDM)-based channel impulse response (CIR) estimation schema that allows two speakers to transmit at the same time and in the same frequency band without collision. We implement eye gesture filtering and adversarial-based eye gesture recognition to identify eye gestures for interaction, filtering out daily eye movements. To address the differences in eye size and facial structure among different users, we employ adversarial training to achieve user-independent eye gesture recognition. We evaluate the performance of our system through experiments on data collected from 16 subjects. The experimental result shows that our system can recognize eight eye gestures with an average F1-score of 0.93, and the false alarm rate of our system is 0.03. We develop an interactive real-time audio-video player based on EyeGesener and then conduct a user study. The result demonstrates the high usability of the proposed system.
4 September 2020 in IEEE Internet of Things Journal
Recent works have shown that acoustic signals can be leveraged to perform respiration monitoring with high accuracy and low energy consumption. Since smartphones, smart speakers, and many other IoT devices are already equipped with microphones and speakers, it is convenient to implement the acoustic sensing solutions on those devices. However, the existing technologies require the speaker to transmit certain ultrasonic signals to detect respiration. Although these signals are inaudible to adults, they are audible to children and pets and they may even have negative impacts on plants 1 2 3. In this paper, instead of using ultrasonic signals, we are trying to leverage audible signals in daily lives, e.g., music or broadcasting audios, to detect human respiration. We design a respiration detection system which derives the respiration rate by continuously estimates the channel impulse response (CIR) using music and broadcast signals. We study the inter-symbol-interference (ISI) brought by the randomness of music and broadcast signal and give our strategy to minimize the interference. We also propose several techniques to resolve some practical issues such as the multi-path effect and sampling frequency offset between the speaker and the microphone. Extensive experiments are conducted to demonstrate the feasibility of our system. The result shows that our system can achieve high respiration detection accuracy with the mean error of less than 0.5 BPM when different audio signals are used.
22 August 2024 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Practicing breathing exercises is crucial for patients with chronic obstructive pulmonary disease (COPD) to enhance lung function. Breathing mode (chest or belly breathing) and lung volume are two important metrics for supervising breathing exercises. Previous works propose that these metrics can be sensed separately in a contactless way, but they are impractical with unrealistic assumptions such as distinguishable chest and belly breathing patterns, the requirement of calibration, and the absence of body motions. In response, this research proposes DeepBreath, a novel depth camera-based breathing exercise assessment system, to overcome the limitations of the existing methods. DeepBreath, for the first time, considers breathing mode and lung volume as two correlated measurements and estimates them cooperatively with a multitask learning framework. This design boosts the performance of breathing mode classification. To achieve calibration-free lung volume measurement, DeepBreath uses a data-driven approach with a novel UNet-based deep-learning model to achieve one-model-fit-all lung volume estimation, and it is designed with a lightweight silhouette segmentation model with knowledge transferred from a state-of-the-art large segmentation model that enhances the estimation performance. In addition, DeepBreath is designed to be resilient to involuntary motion artifacts with a temporal-aware body motion compensation algorithm. We collaborate with a clinical center and conduct experiments with 22 healthy subjects and 14 COPD patients to evaluate DeepBreath. The experimental result shows that DeepBreath can achieve high breathing metrics estimation accuracy but with a much more realistic setup compared with previous works.
21 December 2022 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Spirometry is the gold standard for evaluating lung functions. Recent research has proposed that mobile devices can measure lung function indices cost-efficiently. However, these designs fall short in two aspects. First, they cannot provide the flow-volume (F-V) curve, which is more informative than lung function indices. Secondly, these solutions lack inspiratory measurement, which is sensitive to lung diseases such as variable extrathoracic obstruction. In this paper, we present EarSpiro, an earphone-based solution that interprets the recorded airflow sound during a spirometry test into an F-V curve, including both the expiratory and inspiratory measurements. EarSpiro leverages a convolutional neural network (CNN) and a recurrent neural network (RNN) to capture the complex correlation between airflow sound and airflow speed. Meanwhile, EarSpiro adopts a clustering-based segmentation algorithm to track the weak inspiratory signals from the raw audio recording to enable inspiratory measurement. We also enable EarSpiro with daily mouthpiece-like objects such as a funnel using transfer learning and a decoder network with the help of only a few true lung function indices from the user. Extensive experiments with 60 subjects show that EarSpiro achieves mean errors of 0.20L/s and 0.42L/s for expiratory and inspiratory flow rate estimation, and 0.61L/s and 0.83L/s for expiratory and inspiratory F-V curve estimation. The mean correlation coefficient between the estimated F-V curve and the true one is 0.94. The mean estimation error for four common lung function indices is 7.3%.
3 March 2025 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Human Activity Recognition (HAR) faces significant challenges when deployed in real-world scenarios due to non-independent and identically distributed (non-IID) data distributions. While existing domain adaptation (DA) approaches attempt to address this issue, they either require access to source data or struggle with large domain shifts. This paper presents a novel source-free domain adaptation framework for HAR that effectively handles substantial domain discrepancies across different datasets. Our approach introduces two key innovations: (1) a Discriminative Information Gramian (DIG) method that quantifies the relationship between target-domain samples and the source domain without requiring access to source data, and (2) an unsupervised domain generalization technique that ensures consistent feature extraction across augmented data samples, enhancing the model's effectiveness in the target domain. We evaluate our framework on five diverse HAR datasets comprising 87 users with varying demographics, devices, and environmental conditions. In single-source scenarios, our method achieves 76.77% accuracy and 67.03% F1-score, surpassing state-of-the-art solutions by 9.77% and 17.43%, respectively. For multi-source scenarios, we attain 85.33% accuracy and 79.55% F1-score, exceeding existing methods by at least 8.8% and 14.8%, respectively. This work represents the first successful attempt at dataset-level HAR domain adaptation without access to source data, marking a significant advancement in practical HAR applications.