2 August 2023 in IEEE Transactions on Industrial Electronics
Human–robot interaction (HRI) systems are crucial in robotics, natural, fast-response, and multimodal are the future trends in their development. However, current interaction methods have the following flaws: 1) slow response of action recognition algorithms for generic scenes, especially at the beginning stage; 2) insufficient feature extraction and fusion capabilities for spatiotemporal graph data; and 3) no good paradigm of body–hand recognition in HRI. To overcome these bottlenecks, we propose a fast-response graph convolutional network (GCN) for body–hand gesture recognition. First, we propose a dynamic-static parallel network for dynamic body gestures that is responsive and accurate. Second, we propose a spatiotemporal graph attention module to improve the graph data fusion effect in the dynamic-static network. Third, we implement a complete command module to form complete commands with body and hand information for interactions and control of the robot. Finally, extensive experiments on four datasets and real-world experiments were conducted to demonstrate that our network is capable of fast response and accurate recognition of dynamic body gestures at the beginning stage, verifying the effectiveness of skeleton-based body-hand gesture recognition, with a clear advantage over the state-of-the-art.
13 May 2024 in IEEE Transactions on Intelligent Transportation Systems
For autonomous driving systems, it is crucial to recognize the actions and gestures of traffic conductors and cyclists on the road to ensure safety. However, traffic gesture recognition is more challenging than action recognition in general scenarios due to the differences in action posture and sample composition between traffic gesture datasets and general action datasets. Therefore, general action recognition methods cannot identify traffic gestures well. To overcome these problems, we propose a novel motion-guided graph convolutional transformer (MG-GCT) for traffic gesture recognition. Firstly, we proposed a two-stream network to fully utilize joint data and motion data for action recognition. Secondly, we designed and implemented a motion-guided module between two streams, which leverages the powerful spatial representation ability of the motion data to guide the learning of the joint data stream in the spatial dimension. Thirdly, we implemented a temporal transformer network to process the temporal features of the skeleton. Finally, we conducted extensive experiments on two public datasets and one dataset presented by us to demonstrate the effectiveness of our network in traffic gesture recognition, which has a significant advantage over the state-of-the-art methods.
6 November 2024 in IEEE Intelligent Systems
For the safety of autonomous driving systems, recognizing the gestures of cyclists and controllers on traffic roads is crucial. However, general action recognition methods are unsuitable for traffic gestures. This is because, compared to general actions, traffic gestures exhibit distinct characteristics, namely: (1) short action duration; (2) atypical action posture; and (3) small dataset sizes . We propose a novel two-stream graph convolutional network (2S-GCN) to address these challenges. Firstly, we introduced a motion-guided module connecting the motion and joint stream. By leveraging the motion stream’s spatial expression capability, we guided the joint stream’s learning in spatial dimensions to enhance the network’s performance to short action durations. Secondly, we utilized a simplified skeleton topology that exclusively captures the upper body during data preprocessing, thereby preserving essential information while eliminating redundancy. This approach enhances the network’s adaptability to atypical action postures. Thirdly, we use a GCN embedded with a multi-channel attention module as the backbone, which is particularly suitable for small datasets, and conduct extensive experiments on two datasets. The experimental results demonstrate that our network can accurately recognize traffic gestures and has significant advantages over state-of-theart methods.
6 August 2025 in IEEE Transactions on Industrial Electronics
Gesture recognition constitutes a core modality for human–robot interaction (HRI). Current studies primarily address short-distance scenarios, whereas long-distance outdoor scenarios remain significantly underexplored. The principal technical challenges arise from three key challenges: First, the predominant separated spatio-temporal processing approach in current network architectures and attention mechanisms. 1) introduces redundant structural complexity and computational demands, severely constraining unmanned aerial vehicle (UAV) platform implementation. 2) Inherently limits the model’s ability to learn weakly-correlated spatio-temporal relationships. 3) Additionally, existing HRI systems demonstrate inherent deficiencies in accurate hand localization and skeleton estimation under long-distance conditions. To address these issues, we propose a lightweight spatio-temporal skeleton (STS) attention transformer for long-distance gesture recognition in UAV control. First, we designed and implemented a lightweight STS Transformer to unify spatial-temporal feature processing while maintaining structural simplicity. Second, we developed an STS attention employing coupled-decoupled operators on skeleton sequences to extract weakly-correlated features. Subsequently, we developed a two-stage RGB+D framework for hand detection and skeleton estimation, achieving capability at extended ranges up to 5 m under UAV onboard camera view and 10 m in third-party camera view. Finally, extensive experiments on two datasets and real-world experiments were conducted to demonstrate the effectiveness of our network in skeleton-based gesture recognition, showing a clear advantage over the state-of-the-art (SOTA) methods.
25 December 2024 in IEEE Transactions on Multimedia
Category-level object pose estimation and tracking has achieved impressive progress in computer vision, augmented reality, and robotics. Existing methods either estimate the object states from a single observation or only track the 6-DoF pose of a single object. In this paper, we focus on category-level multi-object 9-Dimensional (9D) state tracking from the point cloud stream. We propose a novel 9D state estimation network to estimate the 6-DoF pose and 3D size of each instance in the scene. It uses our devised multi-scale global attention and object-level local attention modules to obtain representative latent features to estimate the 9D state of each object in the current observation. We then integrate our network estimation into a Kalman filter to combine previous states with the current estimates and achieve multi-object 9D state tracking. Experiment results on two public datasets show that our method achieves state-of-the-art performance on both category-level multi-object state estimation and pose tracking tasks. Furthermore, we directly apply the pre-trained model of our method to our air-ground robot system with multiple moving objects. Experiments on our collected real-world dataset show our method's strong generalization ability and real-time pose tracking performance.