9 February 2024 in IEEE Transactions on Visualization and Computer Graphics
Throughout history, static paintings have captivated viewers within display frames, yet the possibility of making these masterpieces vividly interactive remains intriguing. This research paper introduces 3DArtmator, a novel approach that aims to represent artforms in a highly interpretable stylized space, enabling 3D-aware animatable reconstruction and editing. Our rationale is to transfer the interpretability and 3D controllability of the latent space in a 3D-aware GAN to a stylized sub-space of a customized GAN, revitalizing the original artforms. To this end, the proposed two-stage optimization framework of 3DArtmator begins with discovering an anchor in the original latent space that accurately mimics the pose and content of a given art painting. This anchor serves as a reliable indicator of the original latent space local structure, therefore sharing the same editable predefined expression vectors. In the second stage, we train a customized 3D-aware GAN specific to the input artform, while enforcing the preservation of the original latent local structure through a meticulous style-directional difference loss. This approach ensures the creation of a stylized sub-space that remains interpretable and retains 3D control. The effectiveness and versatility of 3DArtmator are validated through extensive experiments across a diverse range of art styles. With the ability to generate 3D reconstruction and editing for artforms while maintaining interpretability, 3DArtmator opens up new possibilities for artistic exploration and engagement.
28 January 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
The vulnerability of 3D point cloud analysis to unpredictable rotations poses an open yet challenging problem: orientation-aware 3D domain generalization. Cross-domain robustness and adaptability of 3D representations are crucial but not easily achieved through rotation augmentation. Motivated by the inherent advantages of intricate orientations in enhancing generalizability, we propose an innovative rotation-adaptive domain generalization framework for 3D point cloud analysis. Our approach aims to alleviate orientational shifts by leveraging intricate samples in an iterative learning process. Specifically, we identify the most challenging rotation for each point cloud and construct an intricate orientation set by optimizing intricate orientations. Subsequently, we employ an orientation-aware contrastive learning framework that incorporates an orientation consistency loss and a margin separation loss, enabling effective learning of categorically discriminative and generalizable features with rotation consistency. Extensive experiments and ablations conducted on 3D cross-domain benchmarks firmly establish the state-of-the-art performance of our proposed approach in the context of orientation-aware 3D domain generalization.
31 July 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
We introduce GenPoly, a novel generalized 3D prior model designed for multiple 3D generation tasks, focusing on preserving fine details. While previous works learn generalizable representations by decomposing objects into coarse-grained components to reassemble a coherent global structure, this approach sacrifices small-scale details. In this paper, we take a different perspective, formulating 3D prior modeling as a bottom-up polymorphic evolving process. Our key insight is that, beyond global structures, intricate local geometry variations hold rich contextual information that should be incorporated into the modeling process to learn fine-grained, generalizable representations. This allows coarse shapes to progressively evolve through multi-granular local geometry refinements, enabling high-fidelity 3D generation. To this end, we first introduce a polymorphic variational autoencoder (PolyVAE), which constructs a versatile shape residual codebook via a polymorphic quantization mechanism. This codebook strategically encodes intricate local geometry representations from tesselated shapes within the latent space. Building on these representations, a 3D polymorphic evolving scheme is further developed to refine local details in a coarse-to-fine manner progressively. In this way, visually compelling 3D shapes with rich and complex details can be ultimately generated. The effectiveness of our method is demonstrated through extensive qualitative and quantitative evaluations, where GenPoly consistently surpasses state-of-the-art methods across various downstream tasks, particularly in local detail preservation. The code and more visualizations will be available on our project website.
5 June 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Interactive 3D segmentation in radiance fields is crucial for advanced 3D scene understanding and manipulation. However, existing methods often struggle to achieve both volumetric completeness and segmentation accuracy, primarily because they fail to consider the critical links between 2D prompt-based segmentations across multiple views. Motivated by this gap, we introduce Gaussian Prompter, a novel approach specifically designed for 3D Gaussian Splatting. The core idea behind Gaussian Prompter is to seamlessly integrate a Gaussian-centric segmentation paradigm by effectively linking various 2D prompts from multi-view segmentations to ensure consistent 3D segmentation. To realize this, we employ two tailored approaches: GaussBlend and PinPrompt. GaussBlend aggregates multi-view 2D segmentation masks into a cohesive 3D segmentation, ensuring both accuracy and completeness. PinPrompt leverages high-confidence prompts from adjacent views to enhance segmentation precision further. Additionally, to address the lack of complex datasets in 3D segmentation, we introduce the SegMip-360 dataset, which includes over 350 precisely annotated masks across seven scenes. Extensive experiments demonstrate that the Gaussian Prompter significantly outperforms state-of-the-art methods in both segmentation accuracy and completeness. Our code and video demonstrations can be found at our repository and project page.
1 January 2025 in IEEE Transactions on Circuits and Systems for Video Technology