Multimodal image registration is the fundamental technique for scene analysis with series remote sensing images of different spectrum region. Due to the highly nonlinear radiometric relationship, it is quite challenging to find common features between images of different modal types. This paper resorts to the deep neural network, and tries to learn descriptors for multimodal image patch matching, which is the key issue of image registration. A Siamese fully convolutional network is set up and trained with a novel loss function, which adopts the strategy of maximizing the feature distance between positive and hard negative samples. The two branches of the Siamese network are connected by the convolutional operation, resulting in the similarity score between the two input image patches. The similarity score value is used, not only for correspondence point location, but also for outlier identification. A generalized workflow for deep feature based multimodal RS image registration is constructed, including the training data curation, candidate feature point generation, and outlier removal. The proposed network is tested on a variety of optical, near infrared, thermal infrared, SAR, and map images. Experiment results verify the superiority over other state-of-the-art approaches.

Automatic registration of optical and synthetic aperture radar (SAR) images is one of the most challenging tasks due to the influence of speckle noise and nonlinear radiation differences. In this article, we propose a compound registration method for optical and SAR images based on feature decoupling network (FDNet), which consists of a residual denoising network (RDNet) and a pseudo-Siamese fully convolutional network (PSFCN). First, we propose a fast compound matching algorithm, which can overcome the respective weaknesses of the registration accuracy and computational complexity of the feature-based and area-based methods. Specifically, FAST keypoint detection is used to generate the center of the initial template. The extraction of local feature descriptors and the matching of the initial templates are implemented by PSFCN. Second, we design an RDNet to learn the statistical model of speckle noise in SAR images and define a new loss function based on mean-square error (mse) and total variation (TV) to achieve the propagation of speckle noise. PSFCN and RDNet are used to learn deep representations of semantic and noise information, respectively. Then, the semantic and noise features are decoupled on spaced convolutional layers. Finally, the optimal matching templates are searched in a small search window around the initial matching templates. In addition, we propose a strategy for adaptively selecting the template size based on 2-D entropy, which can select the appropriate template size according to the content richness of SAR images. Registration results on a public registration dataset show that our proposed method achieves better performance than other state-of-the-art methods.

To fully explore the complementary information from optical and synthetic aperture radar (SAR) imageries, they need firstly to be co-registered with high accuracy. Due to the vast radiometric and geometric disparity, the problem to match high resolution optical and SAR images is quite challenging. The present deep learning based methods have shown advantages over the traditional approaches, but the performance increment is not significant. In this article, we explore better network framework for high resolution optical and SAR image matching from three aspects. Firstly, we propose an effective multilevel feature fusion method which helps to take advantage of both the low-level fine-grained features for precious feature location and the high-level semantic features for better discriminative ability. Secondly, a feature channel excitation procedure is conducted using a novel multi-frequency channel attention module, which is able to make image features of different types and multiple levels effectively collaborate with each other, and produce image matching features with high diversity. Thirdly, the self-adaptive weighting loss is introduced, with which, each sample is assigned with an adaptive weighting factor, and therefore information buried in all nearby samples can be better exploited. Under a pseudo-Siamese architecture, the proposed optical and SAR image matching network (OSMNet) is trained and tested on a large and diverse high resolution optical and SAR dataset. Extensive experiments demonstrate that each component of the proposed deep framework helps to improve the matching accuracy. Also, the OSMNet shows overwhelming superior to the state-of-the-art handcrafted approaches on imageries of different land-cover types.

Image matching is a primary technology to fuse the complementary information from optical and SAR images. Due to the high nonlinear radiometric and geometric relationship, the optical and SAR image matching task remains a widely unsolved challenge. In this study, we propose to use a Siamese convolutional neural network (CNN) architecture to learn pixelwise deep dense features. The proposed network is able to balance the learning of high-level semantic information and low-level fine-grained information, which is nonnegligible for feature matching task. Under the local searching framework, the loss function is defined based on the score map produced by the sum of squared differences (SSDs) between the learned pixelwise dense features of local optical and the SAR image patches, with a fast implementation in the frequency domain. The hardest negative mining strategy is adopted to increase the discrimination of the network. Extensive experiments are conducted on optical and SAR image pairs of different spatial resolution and different landcover types, verifying the superiority and robustness of the proposed method in terms of matching accuracy and matching precision.

The co-registration of optical and SAR imageries is the bottleneck in exploring the complementary information from the two multi-modal datasets. The difficulties lie in not only the complex radiometric relationship between them, but also the distinct geometrical models of the optical and SAR imaging systems, which cause it nontrivial to explicitly depict the spatial relationship between the corresponding image regions when elevation fluctuations exist. This article aims to investigate the optical flow technique for the pixelwise dense registration of the high resolution optical and SAR images, so as to get rid of the outlier removal and geometric mapping procedures, which have to be conducted in the classical image registration approaches that are based on sparse feature point matching. Herein, a deep optical flow framework is designed. Firstly, a dilated feature concatenation method is proposed to enhance the discriminability of the pixelwise features for similarity measurement. An effective network training strategy is used, based on a smoothed flow loss, and also a training dataset that contains simulated elevation fluctuations. Secondly, we propose a self-supervised optical flow finetuning strategy. It incorporates the strength of the blockwise matching approach, which produces better matching precision, into the proposed pixelwise matching procedure. In this way, the accuracy of the optical-SAR dense registration is substantially improved. Extensive experiments conducted on the 1m resolution optical-SAR image pairs of different land-cover types and distinct topographic conditions indicate that, the proposed OSFlowNet-Ft framework is quite robust, and has the potential to perform the optical-SAR image dense registration in practical applications. The Python code of the proposed deep optical flow network will be made available.