Restoring high-fidelity textures for 3D reconstructed models are an increasing demand in AR/VR, cultural heritage protection, entertainment, and other relevant fields. Due to geometric errors and camera pose drifting, existing texture mapping algorithms are either plagued by blurring and ghosting or suffer from undesirable visual seams. In this paper, we propose a novel tri-directional similarity texture synthesis method to eliminate the texture inconsistency in RGB-D 3D reconstruction and generate visually realistic texture mapping results. In addition to RGB color information, we incorporate a novel color image texture detail layer serving as an additional context to improve the effectiveness and robustness of the proposed method. First, we select an optimal texture image for each triangle face of the reconstructed model to avoid texture blurring and ghosting. During the selection procedure, the texture details are weighted to avoid generating texture chart partitions across high-frequency areas. Then, we optimize the camera pose of each texture image to align with the reconstructed 3D shape. Next, we propose a tri-directional similarity function to resynthesize the image context within the boundary stripe of texture charts, which can significantly diminish the occurrence of texture seams. Finally, we introduce a global color harmonization method to address the color inconsistency between texture images captured from different viewpoints. The experimental results demonstrate that the proposed method outperforms state-of-the-art texture mapping methods and effectively overcomes texture tearing, blurring, and ghosting artifacts.
Monocular object 6D pose estimation is a fundamental yet challenging task in computer vision. Recently, deep learning has been proven to be capable of predicting remarkable results in this task. Existing works often adopt a two-stage pipeline with establishing 2D-3D correspondences and utilizing a PnP/RANSAC or differentiable PnP algorithm to recover 6 degrees-of-freedom (6DoF) pose parameters. However, most of them hardly consider the geometric features in 3D space, and ignore the topological cues when performing differentiable PnP algorithms. To this end, we present an improved end-to-end monocular 6D pose estimation method (DGECN++) that incorporates depth estimation and a geometric-aware learnable PnP network. Our method is based on keypoints. First we detect the 2D keypoints that correspond to the 3D model. We then integrate differentiable PnP/RANSAC algorithm to create an end-to-end pipeline for 6D pose estimation. We focuses on the following three key aspects: 1) We utilize the estimated depth information to guide the process of extracting 2D-3D correspondences and refine the results using a cascaded differentiable PnP/RANSAC algorithm that incorporates geometric information. 2) We leverage the uncertainty of the estimated depth map to enhance the accuracy and robustness of the predicted 6D pose. 3) We propose a differentiable Perspective-n-Point (PnP) algorithm based on edge convolution and self-attention to explore the topological relationships between 2D-3D correspondences. Experimental results demonstrate that our proposed network surpasses existing methods in terms of both effectiveness and efficiency.
Recently, large-scale visual language pre-trained (VLP) models have demonstrated impressive performance across various downstream tasks. Motivated by these advancements, pioneering efforts have emerged in multi-label image recognition with missing labels, leveraging VLP prompt-tuning technology. However, they usually cannot match text and vision features well, due to complicated semantics gaps and missing labels in a multi-label image. To tackle this challenge, we propose Text-Region Matching for optimizing Multi-Label prompt tuning, namely TRM-ML, a novel method for enhancing meaningful cross-modal matching. Compared to existing methods, we advocate exploring the information of category-aware regions rather than the entire image or pixels, which contributes to bridging the semantic gap between textual and visual representations in a one-to-one matching manner. Concurrently, we further introduce multimodal contrastive learning to narrow the semantic gap between textual and visual modalities and establish intra-class and inter-class relationships. Additionally, to deal with missing labels, we propose a multimodal category prototype that leverages intra- and inter-category semantic relationships to estimate unknown labels, facilitating pseudo-label generation. Extensive experiments on the MS-COCO, PASCAL VOC, Visual Genome, NUS-WIDE, and CUB-200-211 benchmark datasets demonstrate that our proposed framework outperforms the state-of-the-art methods by a significant margin. Our code is available here.