Skeleton-based gesture recognition has attracted extensive attention and has made great progress. However, mainstream methods generally treat all frames as equally important, which may limit performance, especially when dealing with high inter-class variance in gesture. To tackle this issue, we propose an approach that models a Markov decision process to identify keyframes while discarding irrelevant ones. This article proposes a deep reinforcement learning double-feature double-motion network comprising two main components: a baseline gesture recognition model and a frame selection network. These two components mutually influence each other, resulting in enhanced overall performance. Following the evaluation of the SHREC-17 and F-PHAB datasets, our proposed method demonstrates superior performance. Find our code at https://github.com/Voldaf/DRL-DDNet .
Although skeleton-based gesture recognition using supervised learning has achieved promising results, the reliance on extensive annotated data poses significant costs. This paper addresses the challenge of semi-supervised skeleton-based gesture recognition, to effectively learn feature representations from labeled and unlabeled data. To resolve this problem, we propose a novel multimodal multilevel attention network designed for semi-supervised learning. This model utilizes the self-attention mechanism to polymerize multimodal and multilevel complementary semantic information of the hand skeleton, designing a multimodal multilevel contrastive loss to measure feature similarity. Specifically, our method explores the relationships between joint, bone, and motion to learn more discriminative feature representations. Considering the hierarchy of the hand skeleton, the skeleton data is divided into multilevel to capture complementary semantic information. Furthermore, the multimodal contrastive loss measures similarity among these multilevel representations. The proposed method demonstrates improved performance in semi-supervised skeleton-based gesture recognition tasks, as evidenced by experiments on the SHREC-17 and DHG 14/28 datasets.
The abundant visual information contained in multi-view images is widely used in computer vision tasks. Existing visual relationship detection frameworks have extended the feature vector to improve model performance. However, single view information can not fully reveal the visual relationships in complex visual scenes. To solve this problem and explore the multi-view information in a visual relationship detection (VRD) model, a novel multi-view VRD framework based on a monocular RGB image and an estimated depth map is proposed. The contributions of this paper are threefold. First, we construct a novel multi-view framework which fuses information of different views extracted from estimated RGB-D images. Second, a multi-view image generation method is proposed to transfer flat visual space to 3D multi-view space. Third, we redesign the visual relationship balanced classifier which can process multi-view feature vectors simultaneously. Detailed experiments were conducted on two datasets to demonstrate the effectiveness of the multi-view VRD framework. The experimental results showed that the multi-view VRD framework resulted in state-of-the-art zero-shot learning performance in specific depth conditions.