3 September 2025 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Human locomotion is a fundamental aspect of daily life and a critical indicator of overall health. This study proposes a cross-modal learning framework based on inertial measurement units (IMUs) to infer plantar pressure distribution, ground reaction force (GRF), and center of pressure (COP). Traditional methods for pressure measurement rely on expensive and complex laboratory equipment, limiting their applicability in real-world scenarios. By integrating IMU data with deep learning techniques, this research achieves real-time, cost-effective, and accurate pressure estimation. The proposed method leverages IMU sensors to reconstruct foot pressure distribution in real-time, with high precision and adaptability to individual characteristics. A robust data pipeline and pretrained model enable accurate inference of GRF and COP using minimal IMU data, validated through case studies. Furthermore, The study builds a high-quality training dataset and optimizes the system for lightweight deployment, enhancing suitability for low-resource environments. Experimental results demonstrate the superior performance of the proposed approach across multiple evaluation metrics, particularly in individual adaptability, spatiotemporal feature capture, and deployment in resource-constrained settings. This work provides a novel and practical solution for clinical gait analysis, sports training, and health monitoring, while paving the way for advanced applications of intelligent devices in motion analysis.
15 June 2026 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Hand-object interactions are central to everyday activities, yet most intelligent assistants today remain blind to users' physical actions. Existing IMU-based recognition approaches focus on classifying predefined gestures, but they lack the semantic expressiveness required for contextual support in real-world scenarios such as office work and home routines. In this paper, we introduce a semantic tokenization pipeline that bridges continuous inertial signals and large language models (LLMs), enabling assistants to “read” hand movements as naturally as words. We first collected a multimodal dataset of dual-hand activities across office and home environments capturing long-horizon action chains that span multiple interrelated sub-tasks. Using self-supervised representation learning, we discretize IMU embeddings into action tokens that approximate a vocabulary of hand interactions. These tokens are then aligned with natural language through instruction-tuned LLMs, supporting tasks such as action captioning, intent inference, and contextual feedback. Evaluation shows that our tokenization improves semantic consistency with language distributions, and the LLM produces accurate, human-preferred descriptions of actions across diverse activities. We further demonstrate a proof-of-concept assistant prototype that generates contextual reminders. Our findings highlight the potential of transforming raw hand motions into a “language of actions,” paving the way for everyday intelligent assistants that are aware of users' physical interactions. The Project page, source code, and dataset are publicly available at https://scut-hai.github.io/HMotionGPT/.