20 April 2022 in IEEE Transactions on Circuits and Systems for Video Technology
Existing research on the 2D image-based 3D model retrieval task focuses on learning transferable representations directly to narrow the domain discrepancy. However, it is not easy to achieve in practice due to the significant variations across two domains. In addition, some methods design a domain discriminator to distinguish the feature arising from source or target domains for transferable feature representations learning, which will lead to an unexpected deterioration of the feature discriminability. To settle these problems, we propose jointly learning transferable and discriminative representations for 2D image-based 3D model retrieval. Specifically, we extract features from the 2D images and 3D models (described as multiple views) by CNN. Considering the difficulty of directly narrowing the discrepancy of two domains, we are prone to connect 2D image and 3D model domains to an intermediate domain, where the domain gap aims to be eliminated. However, the feature transferability does not denote well discriminability. Based on the batch spectral penalization (BSP) theory, the feature transferability is dominated by feature vectors with higher singular values, while the feature discriminability depends on more eigenvectors with lower singular values to convey rich discriminative structures. Therefore, we penalize the largest singular values so that the feature vectors with lower singular values are appropriately enhanced, thereby strengthening feature discriminability. A series of experiments on two challenging datasets, MI3DOR and MI3DOR-2, indicate that our method can significantly improve performance.
1 August 2021 in Journal of Visual Communication and Image Representation
6 August 2023 in Information Fusion
8 May 2023 in Information Processing & Management
28 January 2026 in Information Processing & Management
26 February 2026 in IEEE Transactions on Circuits and Systems for Video Technology
2D-3D cross-modal retrieval (2D-3DCMR) aims at retrieving the most matching 3D models by leveraging 2D query images. However, the inherent modal discrepancy makes the 2D-3DCMR task still largely challenging. Besides, the scarcity of 3D labels in real-world applications severely hinders learning discriminative representations. To address these limitations, we present a prompt tuning guided dual distribution alignment (PTG-DDA) framework based on the CLIP model for the 2D-3DCMR task. Specifically, we design a learnable multi-view adaptive representation learning (MARL) module that adaptively integrates 3D features to merge complementary information and filter out redundant information across views, thereby improving the representation capability of 3D models. To mitigate the feature distribution shift between the 2D and 3D data, we design an attention-guided heterogeneous feature alignment (AHFA) module to guide the 2D and 3D inputs attend to feature banks by adopting the attention mechanism, thereby achieving heterogeneous feature alignment. Furthermore, to learn discriminative 3D features, we employ a multi-modal semantic prompt synergy (MSPS) module, which integrates class-related representations into learnable prompts to progressively learn the cross-modal synergy via a prompt synergy adapter, thereby achieving semantic feature alignment. Comprehensive experimental results on popular 2D-3DCMR benchmarks, i.e., MI3DOR and MI3DOR-2, demonstrate the superiority and effectiveness of PTG-DDA.