Speech-driven gesture generation is an emerging field within virtual human creation. However, a significant challenge lies in accurately determining and processing the multitude of input features (such as acoustic, semantic, emotional, personality, and even subtle unknown features). Traditional approaches, reliant on various explicit feature inputs and complex multimodal processing, constrain the expressiveness of resulting gestures and limit their applicability. To address these challenges, we present Persona-Gestor , a novel end-to-end generative model designed to generate highly personalized 3D full-body gestures solely relying on raw speech audio. The model combines a fuzzy feature extractor and a non-autoregressive Adaptive Layer Normalization (AdaLN) transformer diffusion architecture (DiTs-based). The fuzzy feature extractor harnesses a fuzzy inference strategy that automatically infers implicit, continuous fuzzy features. These fuzzy features, represented as a unified latent feature, are fed into the AdaLN transformer. The AdaLN transformer introduces a conditional mechanism that applies a uniform function across all tokens, thereby effectively modeling the correlation between the fuzzy features and the gesture sequence. This module ensures a high level of gesture-speech synchronization while preserving naturalness. Finally, we employ the diffusion model to train and infer various gestures. Extensive subjective and objective evaluations on the Trinity, ZEGGS, and BEAT datasets confirm our model's superior performance to the current state-of-the-art approaches. Persona-Gestor improves the system's usability and generalization capabilities, setting a new benchmark in speech-driven gesture synthesis and broadening the horizon for virtual human technology. Supplementary videos and code can be accessed at https://zf223669.github.io/Diffmotion-v2-website/.

As a challenging computer vision task, Scene Graph Generation (SGG) finds the latent semantic relationships among objects from a given image, which may be limited by the datasets and real-world scenarios. In this paper, we consider a novel incremental learning task called Relationship-Incremental Scene Graph Generation (RISGG) that learns the semantic relationships among objects in an incremental way. Compared with classic Class-Incremental Learning (CIL) problem, RISGG suffers from its special issues: (1) Old class shift – the relationship-labeled object pair may have different labels during different learning sessions. (2) Background shift – the relationship-unlabeled object pair may not be a real unlabeled one. In this work, we address the above issues from the following aspects. First, we present a Divide-and-Conquer (DaC) pipeline to deal with the old class shift via decoupling the recognition of relationship classes and recognizing relationships individually. In this way, label confusion and interaction among different relationships are eliminated during training. Second, we propose a Feature Adapter (FA) to bridge the feature space gap between the current session and the previous one and use our extra supervision to mine old relationship information in the current session. Our proposed network combined DaC and FA, abbreviated DaCFA-Net, for RISGG. Experimental results on the benchmark dataset demonstrate the significant performance gain of DaCFA-Net in RISGG. It gains about 20% improvement against the SGG baselines on the popular VG dataset.

Monocular 3D human pose estimation presents a considerable challenge owing to the intrinsic depth ambiguity associated with single-camera observations. Existing methods primarily rely on mean per joint position error (MPJPE) loss to train models for the conversion from 2D to 3D coordinates. However, empirical analysis reveals that models trained solely with point-based supervision may produce biomechanically implausible poses or exhibit significant depth ambiguity, even when achieving low MPJPE. This limitation arises from the fact that point-based loss only considers individual joint locations without accounting for inter-joint relationships. Fortunately, edges of human pose encode critical prior knowledge, including skeleton connectivity and biomechanical distributions. Explicitly modeling edge representations enables the model to overcome the constraints associated with point-only approaches, reducing the uncertainty in the optimization process of the 2D-3D inverse mapping and directly constraining depth ambiguity. Therefore, we propose the G raph- A ware M ulti-representation A ggregation (GAMA-Pose) framework that jointly predicts points and edges, with their fusion serving as the final output. To ensure the accuracy of edge predictions and mitigate depth ambiguity, A nti- D epth- A mbiguity Loss (ADA-Loss) is introduced to supervise the properties of edges and give direct supervision on depth ambiguity. Correspondingly, edge-based metrics are proposed to quantify the error of predicted edges. Experiments conducted on Human3.6M and MPI-INF-3DHP datasets demonstrate that GAMA-Pose effectively addresses the limitations of models relying solely on point constraints, mitigates depth ambiguity, enhances the accuracy of both point and edge predictions, and achieves state-of-the-art (SOTA) performance on both datasets.