17 August 2020 in IEEE Access
Recently, gait attracts attention as a practical biometric for devices that naturally possess walking pattern sensing. In the present study, we explored the feasibility of using a multimodal smart insole for identity recognition. We used sensor insoles designed and implemented by us to collect kinetic and kinematic data from 59 participants that walked outdoors. Then, we evaluated the performance of four neural network architectures, which are a baseline convolutional neural network (CNN), a CNN with a multi-stage feature extractor, a CNN with an extreme learning machine classifier using sensor-level fusion and CNN with extreme learning machine classifier using feature-level fusion. The networks were trained with segmented insole data using 0%, 50%, and 70% segmentation overlap, respectively. For 70% segmentation overlap and both-side data, we obtained mean accuracies of 72.8% ±0.038, 80.9% ±0.036, 80.1% ±0.021 and 93.3% ±0.009, for the four networks, respectively. The results suggest that multimodal sensor-enabled footwear could serve biometric purposes in the next generation of body sensor networks.
15 November 2025 in IEEE Sensors Journal
29 April 2026 in IEEE Sensors Journal
Gait recognition using multi-modal data is a promising yet challenging field. In appearance-based gait recognition, set-based methods have become a dominant paradigm, achieving state-of-the-art performance by treating visual sequences as unordered sets to enhance robustness. However, extending this paradigm to a multi-modal context by incorporating inherently sequential data like skeletons introduces a series of fundamental challenges, including: (1) The Sampling Conflict: How can the disparate sampling needs of set-based visual models and sequence-based skeleton models be reconciled? (2) The Heterogeneity Challenge: How can features from sparse, graph-based skeletons be effectively fused into a dense, visual-centric set-based framework? (3) The Projection Problem: What is the optimal strategy for projecting the multi-modal features into a discriminative embedding space? This paper presents a investigation into these three interconnected challenges. To facilitate this study, we propose TriFusionGait, a unified framework built upon a set-based paradigm to integrate silhouette, parsing, and skeleton modalities. Our investigation yields three key design principles that address the core challenges. First, our analysis of the sampling conflict shows that the optimal skeleton strategy is scenario-dependent, following a non-linear relationship between data and robustness. Second, we introduce the Tri-modal Cross-stage Attention (TMCSA) mechanism and Early-Late Fusion (ELF) to address modality heterogeneity. Third, we show that the choice between Single-head and Multi-head mappings is guided by feature granularity. By integrating these principles, TriFusionGait establishes a new state-of-the-art, achieving 99.4% Rank-1 accuracy on CASIA-B (NM), 56.5%/63.0% on SUSTech1K (Clothing/Night). Beyond the proposed framework, this work provides design principles for multi-modal gait recognition systems. The source code is available at: https://github.com/zc513008927/TriFusionGait.