26 December 2023 in IEEE Transactions on Multimedia
Monocular 3D human pose estimation is an ill-posed problem in computer vision due to its depth ambiguity. Most existing works supplement the depth information by extracting temporal pose features from video frames, and they have made notable progress. However, these approaches divide a long sequence of video frames into multiple short sequences for separate processing, which leads to the loss of complementary information between sequences. Furthermore, the short-term temporal correlation among frames in a sequence is often not fully exploited. To model temporal dependencies efficiently, we propose the frame-padded multiscale transformer approach, which includes a frame-padded video sequence preprocessing step and a multiscale temporal transformer backbone. Our approach addresses the omission of the temporal features of edge frames in existing approaches by padding video frames in the shallow layer. In addition, we extract the temporal information of 3D human poses using a multiscale transformer to enhance the short-term correlation of human pose skeleton keypoints. Extensive experiments validate the effectiveness of our approach on two popular datasets: Human3.6 M and MPI-INF-3DHP. The results show that our approach achieves state-of-the-art performance. The code and model are available at https://github.com/ygx7/FMFormer .
9 September 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Based on its excellent capability to extract temporal features, transformer has been widely used in monocular 3D human pose estimation. However, due to its global perspective, it performs inadequately in extracting spatial features, which hinders breakthroughs in performance. In this paper, we propose a local-global feature fusion method based on GCN and transformer for 3D human pose estimation. Our method integrates GCN with multiscale transformer to extract local spatiotemporal features of poses. These are then integrated with the global spatiotemporal features extracted by vanilla transformer to reconstruct 3D human poses accurately. In addition, we introduce a hierarchical feature fusion method to better capturing the underlying 3D pose structure. It blends deep abstract features with shallow raw features. We evaluate our model on the Human3.6M and MPI-INF-3DHP datasets, and experimental results demonstrate that our approach outperforms existing state-of-the-art methods. We achieve advanced performance on both datasets with errors of 37.7mm and 16.4mm under MPJPE, respectively. The code and model are available at https://github.com/ygx7/LG3DPose.
28 September 2023 in IEEE Photonics Journal
At present, with the original point cloud as input, most of the object detectors use Pointnet++ to extract features of the point cloud based on the Farthest Point Sampling (FPS). However, affected by FPS, feature extraction is incomplete and unstable. Moreover, high-level semantic features lack the internal vertex properties of Regions of Interest (RoI). In order to solve the above problems, we propose the MaskSL RCNN (Mask-Spherical-neighborhood-global-feature-Layer Region-CNN), a feature-enhanced 3D object detection network. It improves sampling of the farthest point through point-level feature enhancement. In addition, we propose Spherical neighborhood global feature Layer (SL) to supplement the global features and improve the learning ability of network. At last, based on semantic-level feature enhancement, we design grid pooling layer based on vertex attention, which fully explores the boundary characteristics of RoI and increases ability to learn advanced features in RoI. Our network improves detection precision of small objects such as pedestrians. Compared with PointRCNN, it has improved the mAP of simple, medium, and difficult object detection in the KITTI dataset by 2.66%, 1.69%, and 0.67%, respectively.