21 November 2024 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
The accurate assessment of human body posture through wearable technology has significant implications for sports science, clinical diagnostics, rehabilitation, and VR interaction. Traditional methods often require complex setups or are limited by the environment's constraints. In response to these challenges, this paper presents an innovative approach to human posture estimation under complex motion scenarios through the development of an advanced shoe insole embedded with pressure sensors and an Inertial Measurement Unit (IMU). Coupled with a single wrist-mounted IMU, our system facilitates a comprehensive analysis of human biomechanics by integrating physical kinematics modeling based on pressure data with a multi-region human posture estimation network. To enhance the robustness of our system model, we employed large language models to generate virtual human motion sequences. These sequences were utilized to create synthetic IMU data for data augmentation purposes, addressing the challenge of limited real-world data availability and variability. Our approach uniquely combines physical modeling with data-driven techniques to improve the accuracy and reliability of posture estimation. Experimental results demonstrate that our integrated system significantly advances wearable technology for motion analysis. The Mean Per Joint Position Error (MPJPE) was reduced to 7.75 cm, highlighting the effectiveness of our multi-modal modeling and virtual data augmentation in refining posture estimation.
3 September 2025 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Human locomotion is a fundamental aspect of daily life and a critical indicator of overall health. This study proposes a cross-modal learning framework based on inertial measurement units (IMUs) to infer plantar pressure distribution, ground reaction force (GRF), and center of pressure (COP). Traditional methods for pressure measurement rely on expensive and complex laboratory equipment, limiting their applicability in real-world scenarios. By integrating IMU data with deep learning techniques, this research achieves real-time, cost-effective, and accurate pressure estimation. The proposed method leverages IMU sensors to reconstruct foot pressure distribution in real-time, with high precision and adaptability to individual characteristics. A robust data pipeline and pretrained model enable accurate inference of GRF and COP using minimal IMU data, validated through case studies. Furthermore, The study builds a high-quality training dataset and optimizes the system for lightweight deployment, enhancing suitability for low-resource environments. Experimental results demonstrate the superior performance of the proposed approach across multiple evaluation metrics, particularly in individual adaptability, spatiotemporal feature capture, and deployment in resource-constrained settings. This work provides a novel and practical solution for clinical gait analysis, sports training, and health monitoring, while paving the way for advanced applications of intelligent devices in motion analysis.
3 September 2025 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Current human motion analysis primarily emphasizes kinematics, often neglecting underlying biomechanical factors such as muscle activation and force generation. This restricts a comprehensive understanding of movement mechanics, particularly in rehabilitation and sports science. To bridge this gap, we propose KineticsSense, a multimodal framework integrating inertial measurement units (IMUs) and plantar pressure data to predict lower-limb electromyography (EMG) signals, capturing the kinetics of human motion. By leveraging the complementary strengths of IMU-derived kinematic features and pressure-based ground reaction force estimations, KineticsSense provides a richer biomechanical representation beyond conventional kinematic analyses. Through extensive experiments covering a diverse range of lower-limb activities --- including walking, running, squats, and jumps --- we demonstrate the robustness and adaptability of KineticsSense in accurately estimating muscle activation patterns. Our results further indicate that the system generalizes well across individuals with varying physical attributes, highlighting its potential for real-world applications. Furthermore, case studies in rehabilitation assessment and athletic performance analysis showcase the practical value of our approach in monitoring neuromuscular function and optimizing movement strategies. By bridging kinematics and biomechanics, this work enhances understanding of human motion dynamics and lays a foundation for advances in rehabilitation, sports science, and human-computer interaction.
1 December 2025 in IEEE Internet of Things Journal
Silent speech interfaces (SSIs) promise private and noise-immune communication, but current solutions often sacrifice user comfort, mobility, or privacy. This article introduces PPGSpeech, a novel SSI that overcomes these limitations by pioneering the use of photoplethysmography (PPG) acquired from a comfortable, necklace-style wearable device. Our core discovery is that subtle neck muscle movements during silent articulation induce distinct, measurable modulations in the underlying PPG signal. To harness this phenomenon, we developed a complete end-to-end system featuring: 1) a custom neck-worn sensor for multiwavelength PPG acquisition; 2) a deep learning pipeline that converts 1-D PPG signals into 2-D time–frequency images via a continuous wavelet transform (CWT) and classifies them using a lightweight CNN; and 3) a Pix2Pix GAN model to reconstruct audible speech from the captured signals. In a 16-participant study covering a vocabulary of 15 commands and four confounding actions, our user-dependent model achieved a recognition accuracy of 81.41% $\pm ~9.74$ %. Furthermore, our speech reconstruction achieved a mean opinion score (MOS) of 3.48 and a word correct rate (WCR) of 60.67%, demonstrating that the PPG signal is sufficiently rich to recover intelligible speech. By establishing the viability of neck-based PPG for silent speech, PPGSpeech offers a discreet, privacy-preserving, and continuously wearable paradigm for next-generation human–computer interaction.
30 September 2026 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Strength training often demands attention to form, balance, and pacing, leaving little bandwidth to notice muscle-level deviations such as asymmetric loading or compensatory coordination. We present MuscleSense , a deployable audio-first closed-loop biofeedback system that makes muscle effort perceivable and actionable during movement without requiring visual attention. Using a minimal wearable setup (bilateral front-thigh IMU + EMG) and on-device inference, MuscleSense estimates a feedback-aligned effort state designed for control rather than signal reconstruction: overall intensity ( I ) plus three descriptors capturing symmetry ( S ), co-contraction ( C ), and temporal stability ( D ). This state drives a two-layer sonification policy: continuous pitch supports moment-to-moment effort grounding, while rate-limited event cues highlight persistent deviations with bounded listening load.; AB@We evaluate MuscleSense on N =17 participants, using multi-channel reference EMG only to construct and evaluate labels. Results show strong agreement with EMG-derived ground truth across users and activities, stable smartphone operation with bounded end-to-end latency, and a deployable rate-limited cue policy designed to reduce listening burden. A short within-subject closed-loop squat study further shows significant condition effects on pacing-related outcomes and favorable MuscleSense trends compared with no-feedback, mirror, and IMU-only audio baselines. A supplementary lunge study provides initial subjective evidence that participants perceived deviation-related feedback as useful for noticing inter-limb effort differences, instability, and compensation. Together, these results suggest that audio-first muscle-effort feedback can support eyes-free in-the-loop regulation, while dedicated behavioral validation of individual deviation cues remains an important next step.