27 September 2023 in IEEE Transactions on Multimedia
Skeleton-based action recognition has been substantially driven by the development of artificial intelligence technology and deep sensors. Recently, graph convolutional networks (GCNs) have achieved excellent performances in skeleton-based action recognition. However, the performances of GCN-based methods are impaired by inappropriate node partitioning strategy and obstructed long-range information flow. To solve these issues, a novel Select-Assemble-Normalize Graph Convolution Network (SAN-GCN) is proposed to model the spatio-temporal features of skeleton. First, all skeleton joints are selected as root nodes, and the neighborhoods of the root joints are assembled and normalized according to the body structure, which explicitly and interpretably expresses the spatial geometric relation of the skeleton joints. Second, we propose an attention-based assembly and normalization strategy to adaptively capture non-local joints. The adaptive assembly and normalization can avoid the dilution of key long-range features. Moreover, a bi-level aggregation strategy is introduced to learn spatio-temporal dependencies of joints, where the low-level aggregation aligns the normalized neighborhood graphs, and the high-level aggregation aggregates the features of neighbor nodes by a standard convolution kernel. In high-level aggregation, it is convenient to realize factorized spatio-temporal aggregation or unified spatio-temporal aggregation. Extensive experiments on four datasets with different numbers of action patterns demonstrate that our model achieves comparable performance with the state-of-the-art works.
10 January 2024 in IEEE Transactions on Neural Systems and Rehabilitation Engineering
Gait impairment in Parkinson’s Disease (PD) is quantitatively assessed using the Movement Disorder Society Unified Parkinson’s Disease Rating Scale (MDS-UPDRS), a well-established clinical tool. Objective and efficient PD gait assessment is crucial for developing interventions to slow or halt its advancement. Skeleton-based PD gait MDS-UPDRS score estimation has attracted increasing interest in improving diagnostic efficiency and objectivity. However, previous works ignore the important cross-spacetime dependencies between joints in PD gait. Moreover, existing PD gait skeleton datasets are very small, which is a big issue in deep learning-based gait studies. In this work, we collect a sizable PD gait skeleton dataset by multi-view Azure Kinect sensors. The collected dataset contains 102 PD patients and 30 healthy older adults. In addition, gait data from 16 young adults (aged 24–50 years) are collected to further examine the effect of age on PD gait assessment. For skeleton-based automatic PD gait analysis, we propose a novel cross-spatiotemporal graph convolution network (CST-GCN) to learn complex features of gait patterns. Specifically, a gait graph labeling strategy is designed to assemble and group cross-spacetime neighbors of the root node according to the spatiotemporal semantics of the gait skeleton. Based on this strategy, the CST-GCN module explicitly models the cross-spacetime dependencies among joints. Finally, a dual-path model is presented to realize the modeling and fusion of spatial, temporal, and cross-spacetime gait features. Extensive experiments validate the effectiveness of our method on the collected dataset.
18 June 2024 in IEEE Sensors Journal
Extracting stride length and velocity from wearable sensors is traditionally based on the double integration of accelerometer data with zero velocity update (ZUPT) technique. However, this approach might not be suitable for institutionalized older adults, whose clear zero-velocity phase can not be detected accurately. While deep learning models have been proposed to overcome this limitation, these approaches need subject-specific labeled data, which are difficult to collect in practice, to calibrate the models. We show that two-dimensional (2D) deep convolutional neural networks (DCNNs) can be used to extract accurate estimates of stride length and velocity with instrumented footwear. Leave-one-subject-out cross-validation is used to avoid overfitting of the results to deep learning models on data collected from N=95 institutionalized older adults with two different stride definitions during the 6-minute walk test (6MWT). When stride is defined from the initial contact to the next initial contact, DCNNs result in better inter-rater reliability and improved performance by 38.8% and 31.2% relative to conventional techniques (i.e., the double integration method with ZUPT technique) for stride length and velocity, respectively. The performance of DCNNs does not degrade substantially when the stride is defined from the foot-flat phase (i.e., zero-velocity phase) to the next foot-flat phase. Deep learning models are robust to inter-subject variability without requiring subject-specific labeled data, indicating the potential of their use for out-of-the-lab gait analysis. Their performance is independent of stride definitions, making them more suitable for institutionalized older adults than conventional techniques, where initial contacts can be reliably detected even when they do not occur under the hindfoot.