Fourier-based image correlation is a powerful area-based image registration technique, which involves aligning images based on a translation model or similarity model by means of the image information and operation in the frequency domain. In recent years, Fourier-based image correlation has made significant progress and attracted extensive research interest in a variety of applications, especially in the field of photogrammetry and remote sensing, leading to the development of a number of subpixel methods that have improved the accuracy and robustness. However, to date, a detailed review of the literature related to Fourier-based image correlation is still lacking. In this review, we aim at providing a comprehensive overview of the fundamentals, developments, and applications of image registration with Fourier-based image correlation methods. Specifically, this review introduces the principal laws underlying these methods, presents a survey of the existing subpixel methods calculated both in the spatial domain and in the frequency domain, summarizes the major applications from three aspects, and discusses the challenges and possible directions of future research. This review is expected to be beneficial for researchers working in the relevant fields to obtain an insight into the current state of the art, to develop new variants, to explore potential applications, and to suggest promising future trends of image registration with Fourier-based image correlation.

Three-dimensional (3D) digital technology is essential to the maintenance and monitoring of cultural heritage sites. In the field of bridge engineering, 3D models generated from point clouds of existing bridges is drawing increasing attention. Currently, the widespread use of the unmanned aerial vehicle (UAV) provides a practical solution for generating 3D point clouds as well as models, which can drastically reduce the manual effort and cost involved. In this study, we present a semi-automated framework for generating structural surface models of heritage bridges. To be specific, we propose to tackle this challenge via a novel top-down method for segmenting main bridge components, combined with rule-based classification, to produce labeled 3D models from UAV photogrammetric point clouds. The point clouds of the heritage bridge are generated from the captured UAV images through the structure-from-motion workflow. A segmentation method is developed based on the supervoxel structure and global graph optimization, which can effectively separate bridge components based on geometric features. Then, recognition by the use of a classification tree and bridge geometry is utilized to recognize different structural elements from the obtained segments. Finally, surface modeling is conducted to generate surface models of the recognized elements. Experiments using two bridges in China demonstrate the potential of the presented structural model reconstruction method using UAV photogrammetry and point cloud processing in 3D digital documentation of heritage bridges. By using given markers, the reconstruction error of point clouds can be as small as 0.4%. Moreover, the precision and recall of segmentation results using testing date are better than 0.8, and a recognition accuracy better than 0.8 is achieved.

The semantic labeling of the urban area is an essential but challenging task for a wide variety of applications such as mapping, navigation, and monitoring. The rapid advance in Light Detection and Ranging (LiDAR) systems provides this task with a possible solution using 3D point clouds, which are accessible, affordable, accurate, and applicable. Among all types of platforms, the airborne platform with LiDAR can serve as an efficient and effective tool for large-scale 3D mapping in the urban area. Against this background, a large number of algorithms and methods have been developed to fully explore the potential of 3D point clouds. However, the creation of publicly accessible large-scale annotated datasets, which are critical for assessing the performance of the developed algorithms and methods, is still at an early age. In this work, we present a large-scale aerial LiDAR point cloud dataset acquired in a highly-dense and complex urban area for the evaluation of semantic labeling methods. This dataset covers an urban area with highly-dense buildings of approximately 1 km² and includes more than three million points with five classes of objects labeled. Moreover, experiments are carried out with the results from several baseline methods, demonstrating the feasibility and capability of the dataset serving as a benchmark for assessing semantic labeling methods.

Matching multi-modal remote sensing images (MRSI) is a challenging task. Due to significant nonlinear radiation differences (NRD), traditional image matching methods cannot achieve satisfactory results. Current research shows that structural information can get more robust matching results compared with texture information (i.e., gradient features) from images. In order to better explore the structural information of images, this paper proposes a MRSIs matching method using structure saliency features, called weighted structure saliency feature (WSSF). Two strategies are investigated and integrated in WSSF to improve the matching performance. The scale space is constructed based on the pointwise shape-adaptive texture scale filtering, which can better retain the structure features, and the second-order Gaussian steerable filtering, edge confidence map and phase features are combined to establish the structural saliency map combined with second-order Gaussian steerable filtering, which is much more robust to NRD than traditional gradient map. The performance of the proposed method was evaluated on a total of 120 image pairs from two MRSI datasets, and compared with state-of-the-art matching methods including the histogram of the orientation of weighted phase (HOWP), locally normalized image feature transform (LNIFT), co-occurrence filter space matching (CoFSM), radiation-variation insensitive feature transform (RIFT), local phase sharpness orientation (LPSO), and position-scale-orientation SIFT (PSO-SIFT). Experimental results indicate that WSSF obtains satisfactory and reliable results in terms of success rate and matching accuracy. Compared with the above six methods, the matching accuracy of WSSF is improved by more than 20.275%, and the success rate is improved by over 5.833%. The source code will be publicly available in https://github.com/WGY-RS/WSSF.

Compared with traditional manned airborne photogrammetry, unmanned aerial vehicle remote sensing (UAVRS) has the advantages of lower cost and higher flexibility in data acquisition. It has, therefore, found various applications in fields such as three-dimensional (3D) mapping, emergency management, and so on. However, due to the instability of the UAVRS platforms and the low accuracy of the onboard exterior orientation (EO) observations, the use of direct georeferencing image data leads to large location errors. Light detection and ranging (LiDAR) data, which is highly accurate 3D information, is treated as a complementary data source to the optical images. This paper presents a semi-automatic approach for the registration of UAVRS images and airborne LiDAR data based on linear control features. The presented approach consists of three main components, as follows. (1) Buildings are first separated from the point cloud by the integrated use of height and size filtering and RANdom SAmple Consensus (RANSAC) plane fitting, and the 3D line segments of the building ridges and boundaries are semi-automatically extracted through plane intersection and boundary regularization with manual selections; (2) the 3D line segments are projected to the image space using the initial EO parameters to obtain the approximate locations, and all the corresponding 2D line segments are semi-automatically extracted from the UAVRS images. Meanwhile, the tie points of the UAVRS images are generated using a Förstner operator and least-squares image matching; and (3) by use of the equations derived from the coplanarity constraints of the linear control features and the colinear constraints of the tie points, block bundle adjustment is carried out to update the EO parameters of the UAVRS images in the coordinate framework of the LiDAR data, achieving the co-registration of the two datasets. Experiments were performed to demonstrate the validity and effectiveness of the presented method, and a comparison with the traditional registration method based on LiDAR intensity images showed that the presented method is more accurate, and a sub-pixel accuracy level can be achieved.