Mismatching removal is at the core yet still a challenging problem in the photogrammetry and computer vision field. In this paper, we propose a coordinate embedding network (named CE-Net). We consider the mismatching problem as a graph node classification problem, and generate node descriptors by embedding point coordinates and aggregating geometric information from neighboring nodes based on self-attention and cross-attention mechanism. Finally, a binary classifier is used to separate node descriptors into two classes, namely matching inliers and outliers. Benefiting from the attention mechanism, firstly the node descriptors can get geometric information from “good neighbors” (i.e., matching inliers) and keep away from “bad neighbors” (i.e., matching outliers), improving the exactness of the descriptors; secondly the node descriptors can contain the information from both intra-graph and inter-graph, improving their distinctiveness. Experiments in testing datasets show that our proposed CE-Net achieves the state-of-the-art performance with a precision of 0.972, an outlier recall of 0.984, and an inlier recall of 0.963. Furthermore, CE-Net also outperforms the compared methods in real mismatching removal tasks in terms of positional accuracy, dispersion, and number of remaining point pairs, showing great potentials in practical applications. Our codes and data are available on https://github.com/csyhy1986 .

Due to radiometric and geometric distortions between images, mismatches are inevitable. Thus, a mismatch removal process is required for improving matching accuracy. Although deep learning methods have been proved to outperform handcraft methods in specific scenarios, including image identification and point cloud classification, most learning methods are supervised and are susceptible to incorrect labeling, and labeling data is a time-consuming task. This paper takes advantage of deep reinforcement leaning (DRL) and proposes a framework named unsupervised learning for mismatch removal (ULMR). Resorting to DRL, ULMR firstly scores each state–action pair guided by the output of classification network; then, it calculates the policy gradient of the expected reward; finally, through maximizing the expected reward of state–action pairings, the optimal network can be obtained. Compared to supervised learning methods (e.g., NM-Net and LFGC), unsupervised learning methods (e.g., ULCM), and handcraft methods (e.g., RANSAC, GMS), ULMR can obtain higher precision, more remaining correct matches, and fewer remaining false matches in testing experiments. Moreover, ULMR shows greater stability, better accuracy, and higher quality in application experiments, demonstrating reduced sampling times and higher compatibility with other classification networks in ablation experiments, indicating its great potential for further use.

Due to the inherent limitations of matching algorithms and the complexities associated with image contents, mismatches are inevitable and can have detrimental effects on downstream tasks in computer vision and remote sensing. Researchers have published numerous reviews on mismatch removal, which may suffer from two primary deficiencies. Firstly, these reviews are often embedded within studies that primarily focus on image matching, thereby limiting the detailed and comprehensive analysis of mismatch removal methods. Secondly, reviews of deep learning (DL)-based methods, despite their numerous existence and interconnection, tend to be fragmentary and lack a systematic approach. To address these two shortcomings, this paper presents a comprehensive survey of DL-based mismatch removal principles and methods. We provide a summary of network architectures, techniques for extracting geometrical information, and various training modes. Specifically, we highlight the importance of permutation invariance in mining operations, enumerate a majority of existing mining methods, and provide an explanation of their permutation invariant properties. Furthermore, we present both the intuitive motivation and mathematical analysis of commonly used methods, elucidating their underlying principles and efficacy. In the conclusion, we predict upcoming trends based on the findings of our review, aiming to provide valuable insights into mismatch removal techniques and guide their practical applications.

Geographic features in maps consist of a mixture of points, polylines, and polygons, generally including POIs, roads, buildings, and other geographic features. Due to the differing dimensionality of these various types of geographic data, traditional geometric similarity measurement methods that rely on a single type of feature are not applicable to mixed scenes. The traditional solution to this issue is to treat points as projections of polylines and polylines as projections of polygons. Through neural networks, projection matrices can be learned to convert points, polylines, and polygons into the same type of object, thereby enabling the use of single-scene geometric measurement methods (e.g., Graph Neural Networks) to solve the problem. However, the key challenge in using Graph Neural Networks for similarity measurement is learning the adjacency relationships between geometric features. It is evident that the adjacency relationships between different feature pairs, such as polyline–polygon, polyline–polyline, and polygon–polygon, require different approaches for measurement, and these diverse relationships cannot be captured by a simple GNN. Heterogeneous Graph Neural Networks (HGNNs) are suited to address this problem: different adjacency relationships between feature pairs can be learned using distinct embedded networks, the new node characteristics can be calculated through the information aggregation and propagation framework of HGNNs, and these new characteristics can be used for geometric similarity measurement. Finally, the effectiveness of the proposed method was verified through practical experiments.

Most learning-based methods require labelling the training data, which is time-consuming and gives rise to wrong labels. To address the labelling issues thoroughly, we propose an unsupervised learning framework to remove mismatches by maximizing the expected score of sample consensuses (MESAC). The proposed MESAC can train various permutation invariant networks (PINs) based on training data with no labels, and has three distinct merits: (1) the framework can train various PINs in an unsupervised mode such that these are immune to wrong labels; (2) the gradients of the expected score are explicitly calculated by a revised score-function estimator, which can avoid gradient explosion; (3) the distribution of matching probabilities is learned from the PIN and precisely modelled by a categorical distribution, which can decrease the sampling times and improve the computational efficiency accordingly. Experiments of testing datasets disclose that mean recall is increased by at most 77% when pure PINs are embedded in MESAC, and mean precision is also improved by 16%. Applications in pose recovery indicate that the success rates of MESAC-integrated PINs outperform the compared methods when training with neither matching labels nor ground truth epipolar geometry (EG) constraints, showing the great potential of MESAC in mismatch removal.