Deep networks have made remarkable progress in Multi-View Stereo (MVS) task in recent years. However, the problem of finding accurate correspondences across different views under ill-posed matching situations remains unresolved and crucial. To address this issue, this paper proposes a Geometry-enhanced Attentive Multi-View Stereo (GA-MVS) network, which can access multi-view consistent feature representation and achieve accurate depth estimation in challenging situations. Specifically, we propose a geometry-enhanced feature extractor to explore illumination-invariant geometric features and incorporate them with common texture features to improve matching accuracy when dealing with view-dependent photometric effects, such as shadow and specularity. Then, we design a novel attentive learning framework to explore per-pixel adaptive supervision, effectively improving the depth estimation performance of textureless regions. The experimental results on the DTU and Tanks & Temples benchmarks demonstrate that our method achieves state-of-the-art results compared to other advanced MVS models.
11 April 2025 in Proceedings of the AAAI Conference on Artificial Intelligence
Guided depth super-resolution (GDSR) has demonstrated impressive performance across a wide range of domains, with numerous methods being proposed. However, existing methods often treat depth maps as images, where shading values are computed discretely, making them struggle to effectively restore the continuity inherent in the depth map. In this paper, we propose a novel approach that maximizes the utilization of spatial characteristics in depth, coupled with human abstract perception of real-world substance, by transforming the GDSR issue into deformation of a roughcast with ideal plasticity, which can be deformed by force like a continuous object. Specifically, we firstly designed a cross-modal operation, Continuity-constrained Asymmetrical Pixelwise Operation (CAPO), which can mimic the process of deforming an isovolumetrically flexible object through external forces. Utilizing CAPO as the fundamental component, we develop the Pixelwise Cross Gradient Deformation (PCGD), which is capable of emulating operations on ideal plastic objects (without volume constraint). Notably, our approach demonstrates state-of-the-art performance across four widely adopted benchmarks for GDSR, with significant advantages in large-scale tasks and generalizability.
With the gradual application of 3D reconstruction technology in underwater scenes, the design of vision-based reconstruction models has become an important research direction for human ocean exploration and development. The underwater laser triangulation method is the most commonly used approach, yet it misses details during the reconstruction of sparse point clouds, which do not meet the requirements of practical applications. On the other hand, existing underwater photometric stereo methods can accurately reconstruct local details of target objects, but they require relative stillness to be maintained between the camera and the target, which is practically difficult to achieve in underwater imaging environments. In this paper, we propose an underwater target reconstruction algorithm that combines laser triangulation and multispectral photometric stereo (MPS) to address the aforementioned practical problems in underwater 3D reconstruction.This algorithm can obtain more comprehensive 3D surface data of underwater objects through mobile measurement. At the same time, we propose to optimize the laser place calibration and laser line separation processes, further improving the reconstruction performance of underwater laser triangulation and multispectral photometric stereo. The experimental results show that our method achieves higher-precision and higher-density 3D reconstruction than current state-of-the-art methods.
Accurate land-use classification is critical for urban planning and environmental monitoring, yet effectively integrating heterogeneous data sources such as hyperspectral imagery and laser radar (LiDAR) remains challenging. To address this, we propose MixtureRS, a compact multimodal network that effectively integrates hyperspectral imagery and LiDAR data for land-use classification. Our approach employs a 3-D plus heterogeneous convolutional stack to extract rich spectral–spatial features, which are then tokenized and fused via a cross-modality transformer. To enhance model capacity without incurring significant computational overhead, we replace conventional dense feed-forward blocks with a sparse Mixture-of-Experts (MoE) layer that selectively activates the most relevant experts for each token. Evaluated on a 15-class urban benchmark, MixtureRS achieves an overall accuracy of 88.6%, an average accuracy of 90.2%, and a Kappa coefficient of 0.877, outperforming the best homogeneous transformer by over 12 percentage points. Notably, the largest improvements are observed in water, railway, and parking categories, highlighting the advantages of incorporating height information and conditional computation. These results demonstrate that conditional, expert-guided fusion is a promising and efficient strategy for advancing multimodal remote sensing models.
24 March 2023 in IEEE Transactions on Instrumentation and Measurement
Multi-view stereo (MVS) aims to measure the precise surface depth of a scene from observations at multiple photography angles and then densely reconstruct its 3D geometry information. Learning-based MVS approaches have been dominantly popular for their robustness to low texture areas and non-Lambertian surfaces. However, most existing methods focus on estimating depth maps for input images by constructing global cost volumes and designing ingenious yet large variance-based 3D-CNN networks for cost volume regularization. Such approaches ignore the co-visible relation embedded in multiple views, resulting in heavy computation, erroneous cost aggregation from invisible views, and finally inaccurate 3D reconstruction results. In this paper, we propose a Co-visibility Reasoning Multi-View Stereo Network (CR-MVSNet) to explore the co-visible relations hidden in multiple views for reliable multi-view similarity measurement and efficient reconstruction. Precisely, the proposed Co-visibility Reasoning Cost Aggregation (CRCA) module includes the adaptive inter-cost volume aggregation via mining the uncertainty of co-visibility relations in multiple views and the adaptive intra-cost volume aggregation by exploiting spatial contextual information. Moreover, the cost volumes are constructed via the proposed global-to-patch manner to speed up computation. Experimental results show that our approach achieves the best overall performance on the DTU, Tanks and Temples, and ETH3D-test datasets over recent state-of-the-art MVS algorithms. The consistently favorable results on three datasets with completely different depth ranges proved the superiority and generalizability of CR-MVSNet.
20 May 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Learning-based Multi-View Stereo (MVS) methods, typically reliant on cascaded cost volume formulations, perform well on small-scale scenes. However, as the depth range of captured images becomes broader and more varied, the coarse-to-fine depth sampling process, which depends solely on feature matching, is increasingly prone to local optima. Despite recent advancements in feature representation, depth sampling patterns, and cost aggregation techniques, challenges related to model generalization and computational efficiency persist. In this paper, we propose SR-MVSNet, a novel framework that integrates multi-view feature matching and RGB-D cross-modal structural consistency learning to achieve high-quality 3D reconstruction. Our approach begins with the construction of Low-Resolution (LR) cost volumes for initial LR depth estimation, which are then enhanced to full-resolution via a tailored uncertainty-aware guided depth super-resolution module. To ensure cross-view consistency, the depth maps undergo further refinement through multi-view feature matching. By avoiding high-resolution cost volume processing, our framework improves depth estimation robustness and efficiency. Additionally, we introduce an iterative depth fusion post-processing strategy during inference to improve reconstruction in ambiguous matching regions, a critical challenge for MVS methods. Experiments show that our method achieves top-3 performance on the DTU and Tanks & Temples datasets and ranks first on the ETH3D dataset. Furthermore, it uses significantly fewer GPU resources than most high performing methods, offering a favorable trade-off between reconstruction quality and computational efficiency.