Underwater images suffer from quality degradation due to the underwater light absorption and scattering. It remains challenging to enhance underwater images using deep learning-based methods since the scarcity of real-world underwater images and their enhanced counterparts. Although existing works manually select well-enhanced images as reference images to train enhancement networks in an end-to-end manner, their performance tends to be inferior in some scenarios. We argue that the manually selected reference images cannot approximate their ground truth perfectly, leading to imbalanced learning and domain shift in enhancement networks. To address this issue, we analyse widely used underwater datasets from the perspective of color spectrum distribution and surprisingly find the sound color spectrum distribution of the enhanced reference images compared to in-air datasets. Based on this perceptive observation, instead of directly learning the enhancement mapping, we propose a novel methodology to learn color compensation for general purposes. Specifically, we present a probabilistic color compensation network that estimates the probabilistic distribution of colors by multi-scale volumetric fusion of texture and color features. We further propose a novel two-stage enhancement framework that first performs color compensation and then enhancement, which is highly flexible to be integrated with an existing enhancement method without tuning. Extensive experiments on underwater image enhancement across various challenging scenarios show that our proposed approach consistently improves the results of the popular conventional and learning-based methods by a significant margin. Moreover, our enhanced images achieve superior performance on underwater salient object detection and visual 3D reconstruction, demonstrating that our method can successfully break through the generalization bottleneck of existing learning-based enhancement models. Our implementation will be made available at https://github.com/Ray2OUC/P2CNet.
Recommending a limited number of Point-of-Interests (POIs) a user will visit next has become increasingly important to both users and POI holders for Location-Based Social Networks (LBSNs). However, POI recommendation is a challenging task since complex sequential patterns and rich contexts are contained in extremely sparse user check-in data. Recent studies show that embedding techniques effectively incorporate POI contextual information to alleviate the data sparsity issue, and Recurrent Neural Network (RNN) has been successfully employed for sequential prediction. Nevertheless, existing POI recommendation approaches are still limited in capturing user personalized preference due to separate embedding learning or network modeling. To this end, we propose a novel unified spatio-temporal neural network framework, named PPR, which leverages users’ check-in records and social ties to recommend personalized POIs for querying users by joint embedding and sequential modeling. Specifically, PPR first learns user and POI representations by joint modeling User-POI relation, sequential patterns, geographical influence, and social ties in a heterogeneous graph and then models user personalized sequential patterns using the designed spatio-temporal neural network based on LSTM model for the personalized POI recommendation. Furthermore, we extend PPR to an end-to-end recommendation model by jointly learning node representations and modeling user personalized sequential preference. Extensive experiments on three real-world datasets demonstrate that our model significantly outperforms state-of-the-art baselines for successive POI recommendation in terms of Accuracy, Precision, Recall and NDCG. The source code is available at:https://www.anonymous.4open.science/r/DSE-1BEC.
20 July 2021 in IEEE Journal of Oceanic Engineering
Photometric stereo is a widely used optical 3-D reconstruction method, which has advantages for high-resolution and well-detailed 3-D reconstruction even in weak texture regions. However, in underwater photometric stereo, the light refraction and attenuation using the camera and point light sources usually result in significant errors in shape recovery. In this article, we present a novel solution to address these challenges for improving the performance of underwater photometric stereo by combining it with underwater laser triangulation. First, we employ underwater laser triangulation as references to correct the global surface-shape distortion caused by nonuniform close-range illumination. Second, we propose to use an underwater camera refractive model to eliminate nonlinear refraction distortion. Third, we build a device implementing the proposed method for 3-D seabed reconstruction in on-site sea trials. Experimental results demonstrate that the proposed approach is able to produce accurate 3-D reconstruction results in the underwater environment.
9 February 2023 in IEEE Transactions on Instrumentation and Measurement
Structure-from-Motion (SfM) is a prevalent solution for vision measurement. Traditional SfM methods rely on sparse feature matching to establish visual associations between different views. However, both the incremental-based and global-based SfM show sensitiveness to outliers generated by existing hand-crafted features under challenging environments. Planar marker-assisted SfM is an alternative solution that employs artificial landmarks to help robust pose estimation. Nevertheless, using planar markers also exists the planar pose ambiguity problem, which means there may be more than one reasonable solution for pose estimation; it is a great challenge for camera pose estimation. This paper proposes an efficient planar marker-assisted global SfM method and resolves the planar pose ambiguity problem by examining the rotation consistency across multiple views. In particular, we construct two complementary graphs and present a graph filter method to filter out the views with pose ambiguity according to our designed rotation consistency checking. Afterward, we perform sparse feature matching following the proposed graph structure and make a pose-fixed bundle adjustment for high-quality 3D reconstruction. Extensive experiments have been conducted on synthetic and real datasets, and the results verify the effectiveness of the proposed method. Our method outperforms several state-of-the-art methods with encouraging efficiency, especially in challenging scenes.
18 March 2024 in IEEE Transactions on Circuits and Systems for Video Technology
Deep networks have made remarkable progress in Multi-View Stereo (MVS) task in recent years. However, the problem of finding accurate correspondences across different views under ill-posed matching situations remains unresolved and crucial. To address this issue, this paper proposes a Geometry-enhanced Attentive Multi-View Stereo (GA-MVS) network, which can access multi-view consistent feature representation and achieve accurate depth estimation in challenging situations. Specifically, we propose a geometry-enhanced feature extractor to explore illumination-invariant geometric features and incorporate them with common texture features to improve matching accuracy when dealing with view-dependent photometric effects, such as shadow and specularity. Then, we design a novel attentive learning framework to explore per-pixel adaptive supervision, effectively improving the depth estimation performance of textureless regions. The experimental results on the DTU and Tanks & Temples benchmarks demonstrate that our method achieves state-of-the-art results compared to other advanced MVS models.
3 March 2023 in IEEE Transactions on Instrumentation and Measurement
Recent feature descriptor research has witnessed tremendous progress with the development of the deep neural network. However, most existing descriptors solely focus on learning strong discriminativeness with deep-invariant features, neglecting their representation ability and rich hierarchical clues hidden in images, which could further establish high-quality matches via implicit hierarchical comparisons in rich representative descriptor space. In this article, we consider both the discriminative and representation ability of feature descriptors to enrich the descriptor space with a novel representative learning framework. On the one hand, we introduce histogram of oriented gradient (HOG) as a prior term to guide our descriptor to learn a powerful representation and robustness in a self-supervised manner. On the other hand, we present an adaptive triplet loss (ATL), which penalizes the triplet loss (TL) according to the descriptor matching distances in order to encourage our descriptor to learn strong discriminativeness. Moreover, to fully use the information encapsulated in images and boost the representation ability, we propose a novel HIerarchical Feature Transformer Network (HIFT), which derives dense descriptions from the semantic and cross-scale-enhanced hierarchical features in a local-to-global manner. Extensive experiments on popular feature matching and visual localization benchmarks show that the HIFT achieves highly competitive performance compared with the state-of-the-art methods. Applications on visual measurement tasks of visual 3-D reconstruction and ego-motion estimation also demonstrate the high generalization ability of our method. Our model is available at https://github.com/Ray2OUC/HIFT .