Spatio-temporal action detection networks, which need to simultaneously extract and fuse spatial and temporal features, often result in existing models becoming bloated and difficult to run in real-time and deploy on edge devices. This paper introduces an efficient and real-time spatio-temporal action detection model, YOWOv3. This model uses efficient 3D and 2D backbone networks to separately extract spatial and spatial-temporal features from sequential information. A lightweight spatio-temporal feature fusion module, designed by deeply integrating convolution and self-attention mechanisms, further enhances the extraction of spatio-temporal features. We refer to this module as the CFACM (Channel Fusion & Attention Convolution Mix) module. Our approach not only outperforms the latest efficient spatio-temporal action detection models in terms of lightness, reducing the model size by 24% compared to the latter, but also improves the mAP accuracy on the UCF101-24 dataset by 1.35%, while maintaining excellent speed performance, thus achieving a balance between accuracy and speed. Furthermore, existing models often use 3D convolutions to extract temporal information, which may be limited on certain devices, such as Apple’s M series processors. To mitigate the potential issue of 3D convolution operators not being supported during edge deployment of spatio-temporal action detection models, we employ a spatio-temporal shift module containing only 2D convolutions. This enables the model to acquire temporal information and inject the obtained temporal features into multi-level spatio-temporal feature extraction models. This not only liberates the model from the constraints of 3D convolution operations but also enhances the model’s balance between accuracy and speed. This results in state-of-the-art performance in lightweight networks using only 2D convolutions.

Compared with traditional methods, the action recognition model based on 3D convolutional deep neural network captures spatio-temporal features more accurately, resulting in higher accuracy. However, the large number of parameters and computational requirements of 3D models make it difficult to deploy on mobile devices with limited computing power. In order to achieve an efficient video action recognition model, we have analyzed and compared classic lightweight network principles and proposed the 3D-ShuffleViT network. By deeply integrating the self-attention mechanism with convolution, we have introduced an efficient ACISA module that further enhances the performance of our proposed model. This has resulted in exceptional performance in both context-sensitive and context-independent action recognition, while reducing deployment costs. It is worth noting that our 3D-ShuffleViT network, with a computational cost of only 6% of that of SlowFast-ResNet101, achieved 98% of the latter’s Top1 accuracy on the EgoGesture dataset. Furthermore, on the same CPU (Intel i5-8300H), its speed was 2.5 times that of the latter. In addition, when we deployed our model on edge devices, our proposed network achieved the best balance between accuracy and speed among lightweight networks of the same order.