7 February 2022 in IEEE Transactions on Multimedia
Recent years have witnessed the popularity of using a two-stream architecture and attention mechanism for action recognition with videos. However, it is time-consuming to train two separate convolutional neural networks (ConvNets), especially with the complex attention mechanism. In this paper, we present a novel architecture, termed as Appearance-Motion Fusion Network (AMFNet), to learn efficient and robust action representation from RGB and optical flow data in an end-to-end manner. AMFNet is constructed by connecting a convolutional neural network with an appearance-motion fusion block (AMFB), whose goal is to incorporate appearance and motion streams into a unified framework driven by a cross-modality attention (CMA) mechanism. More specifically, the CMA only relies on optical flow data, which consists of a Key-Frame Adaptive Selection Module (KFASM) and an Optical-Flow-Driven Spatial Attention Module (OFDSAM). The former aims to adaptively identify the discriminative key frames from a sequence, while the latter is able to guide our networks to focus on the action-relevant regions of each frame. We explore two schemes for appearance and motion streams fusion in AMFB from hierarchical and comprehensive levels. The proposed AMFNet is extensively evaluated on five action recognition data sets, including HMDB-51, UCF-101, JHMDB, Penn and Kinetics-400. Compared to the state-of-the-art methods operated at RGB and optical flow, the experimental results validate that our AMFNet achieves a comparable performance with a pure 2D-Single-ConvNet design.
9 March 2020 in IEEE Access
Human action recognition is a challenging task in computer vision. Modeling spatial-temporal information in videos effectively is crucial for the performance improvement of action recognition. In this paper, we introduce an end to end framework, Spatio-Temporal Attention ConvNet (STACNet), which combines two novel attention modules and convolutional neural networks for action recognition. Two novel attention modules, Spatial Attention Module (SAM) and Temporal Attention Module (TAM), are proposed respectively. Spatial Attention Module is established by fusing the value feature and the gradient feature of the feature map, making the representation of ConvNet for action recognition focus on the informative motion regions of actions. Temporal Attention Module is built by combining global average pooling and global max pooling to explore key frames in videos. With the two attention modules, STACNet can adaptively distinguish key frames in a sequence and selectively pay different levels of attention to different spatial motion regions of human actions, at virtually negligible increase in computation cost. We demonstrate the effectiveness of SAM and TAM for action recognition, respectively. The experimental results show that STACNet can obtain superior performance on the datasets of HMDB51 and UCF101.
25 December 2024 in IEEE Transactions on Multimedia
Thanks to the remarkably expressive power for depicting structural data, Graph Convolutional Network (GCN) has been extensively adopted for skeleton-based action recognition in recent years. However, GCN is designed to operate on irregular graphs of skeletons, making it difficult to deal with other modalities represented on regular grids directly. Thus, although existing works have demonstrated the necessity of multi-modality fusion, few methods in the literature explore the fusion of skeleton and other modalities within a GCN architecture. In this paper, we present a novel GCN-based framework, termed GCNbased Multi-modality Fusion Network (GMFNet), to efficiently utilize complementary information in RGB and skeleton data. GMFNet is constructed by connecting a main stream with a GCN-based multi-modality fusion module (GMFM), whose goal is to gradually combine finer and coarse action-related information extracted from skeletons and RGB videos, respectively. Specifically, a cross-modality data mapping method is designed to transform an RGB video into a skeleton-like (SL) sequence, which is then integrated with the skeleton sequence under a gradual fusion scheme in GMFM. The fusion results are fed into the following main stream to extract more discriminative features and produce the final prediction. In addition, a spatio-temporal joint attention mechanism is introduced for more accurate action recognition. Compared to the multi-stream approaches, GMFNet can be implemented within an end-to-end training pipeline and thereby reduces the training complexity. Experimental results show the proposed GMFNet achieves impressive performance on two large-scale data sets of NTU RGB+D 60 and 120.