Vision Transformer, which performs well in various vision tasks, encounters a bottleneck in skeleton-based action recognition and falls short of advanced GCN-based methods. The root cause is that the current skeleton transformer depends on the self-attention mechanism of the complete channel of the global joint, ignoring the highly discriminative differential correlation within the channel, so it is challenging to learn the expression of the multivariate topology dynamically. To tackle this, we present Skeleton MixFormer, an innovative spatio-temporal architecture to effectively represent the physical correlations and temporal interactivity of the compact skeleton data. Two essential components make up the proposed framework: 1) Spatial MixFormer. The channel-grouping and mix-attention are utilized to calculate the dynamic multivariate topological relationships. Compared with the full-channel self-attention method, Spatial MixFormer better highlights the channel groups' discriminative differences and the joint adjacency's interpretable learning. 2) Temporal MixFormer, which consists of Multiscale Convolution, Temporal Transformer and Sequential Holding Module. The multivariate temporal models ensure the richness of global difference expression and realize the discrimination of crucial intervals in the sequence, thereby enabling more effective learning of long and short-term dependencies in actions. Our Skeleton MixFormer demonstrates state-of-the-art (SOTA) performance across seven different settings on four standard datasets, namely NTU-60, NTU-120, NW-UCLA, and UAV-Human. Related code will be available on https://github.com/ElricXin/Skeleton-MixFormer.

Human action recognition (HAR), which aims to recognize and understand individual actions and intentions, has rapidly become a research hotspot in computer vision. Compared with other data modalities, skeleton data offers more efficient node semantics and more coherent spatio-temporal motion patterns, effectively reducing the impact of lighting and background changes. In recent years, many researchers have focused on skeleton-based action recognition methods and have made significant progress. However, we believe that the current skeleton-based action recognition methods still face three major challenges: 1) reducing reliance on expensive labeled data while maintaining model performance; 2) enabling the model to understand and recognize new behavior classes with a limited number of samples; and 3) addressing the challenges posed by the lack of skeleton information in single-modality spatio-temporal motion representation learning. Based on these challenges, we conduct a comprehensive review of the existing skeleton-based action recognition methods. Additionally, we provide an extensive review and analysis of publicly available action recognition datasets. This review aims to offer researchers a comprehensive perspective, stimulate more innovative ideas, and promote the application and breakthrough of skeleton action recognition in a wider range of computer vision tasks.

Skeleton-based action recognition is a crucial but challenging task in the application of engineering algorithms. However, due to the inaccurate estimation quality, certain joints that should theoretically lack dynamic information show irregular jitter, which affects the recognition accuracy. In this paper, we propose a Cascade Interaction Spatial-temporal transformer Network ( CI-STFormer ) to address abnormal joint jitter and multi-scale aggregation of features effectively. The network comprises three parts: (1) Spatial sparse-level optimization and interaction module. Relying on the topology map, the sparse-level interaction adjacency matrix is constructed through cascading interaction to mask the jitter non-discriminative joints, and then the transformer is applied to realize the sparse-level internal feature update. (2) Spatial primal-level fusion and interaction module. Multi-head self-attention is employed to enhance the original feature representation. Then, the sparse-level update features are cascaded to construct the scale-cascaded interaction adjacency matrix to achieve the balance of noise caused by the jitter and the fusion of different scale features. (3) Temporal domain scale-level TCN combines the multi-scale filter and the temporal self-attention channel interaction algorithm to extract different temporal features and the interaction and update of global temporal information. The experimental results show that the proposed method performs excellently on the four datasets of NTU RGB+D 60, NTU RGB+D 120, UAV-Human, and NW-UCLA and achieves state-of-the-art performance on the transformer-based track. Related code will be available on https://github.com/Xdu-Liu/CI-STFormer.git.

Graph convolution networks (GCNs) have achieved remarkable performance in skeleton-based action recognition by exploiting the adjacency topology of body representation. However, the adaptive strategy adopted by the previous methods to construct the adjacency matrix is not balanced between the performance and the computational cost. We assume this concept of Adaptive Trap, which can be replaced by multiple autonomous submodules, thereby simultaneously enhancing the dynamic joint representation and effectively reducing network resources. To effectuate the substitution of the adaptive model, we unveil two distinct strategies, both yielding comparable effects: (1) Optimization. Individuality and Commonality GCNs (IC-GCNs) is proposed to specifically optimize the construction method of the associativity adjacency matrix for adaptive processing. The uniqueness and co-occurrence between different joint points and frames in the skeleton topology are effectively captured through methodologies like preferential fusion of physical information, extreme compression of multi-dimensional channels, and simplification of self-attention mechanism. (2) Replacement. AutoLearning GCNs (AL-GCNs) is proposed to boldly remove popular adaptive modules and cleverly utilize human key points as motion compensation to provide dynamic correlation support. AL-GCNs construct a fully learnable group adjacency matrix in both spatial and temporal dimensions, resulting in an elegant and efficient GCN-based model. In addition, three effective tricks for skeletonbased action recognition (Skip-Block, Bayesian Weight Selection Algorithm, and Simplified Dimensional Attention) are exposed and analyzed in this paper. Finally, we employ the variable channel and grouping method to explore the hardware resource bound of the two proposed models. IC-GCN and AL-GCN exhibit impressive performance across NTU-RGB+D 60, NTU-RGB+D 120, NW-UCLA, and UAV-Human datasets, with an exceptional parameter-cost ratio. Code will be available on Adaptive-Pitfall.

Semantically guided skeleton-based action recognition is a crucial but challenging task in the data processing with RGB and depth sensors. The generation of action descriptions is facilitated by the pretrained language model, thereby imbuing the skeleton with essential contextual semantics. However, it also presents two challenges that require careful consideration: (1) efficiently and automatically generating the core semantics of actions, and (2) extending the applicability of supervised learning when only partial labels are available. Therefore, we propose Critical Elements Prompts for Partial Supervision in Skeleton-based Action Contrastive Learning (CEPCLR) as a significant attempt to break through the existing limitations of skeleton and semantic multimodal fusion. Firstly, we design action-specific prompts that enable large language models (LLMs) to generate general semantic descriptions both efficiently and accurately, followed by iterative prompting that mines discriminative descriptions through controlled, rule-based emergence. Secondly, we adopt Language-Skeleton Contrastive Learning for multimodal fusion, which significantly enhances the integration of positive and negative action pairs. Notably, we introduce a novel partially supervised learning task for skeleton data, where labels are used selectively for semantic generation but are not involved in the classification tasks. The proposed CEPCLR achieves strong results on standard datasets like NTU RGB+D, NTU RGB+D 120, and PKU-MMD, effectively addressing missing contextual information in human skeleton and action expressions.