Registration for multisensor or multimodal image pairs with a large degree of distortions is a fundamental task for many remote sensing applications. To achieve accurate and low-cost remote sensing image registration, we propose a multiscale framework with unsupervised learning, named MU-Net. Without costly ground truth labels, MU-Net directly learns the end-to-end mapping from the image pairs to their transformation parameters. MU-Net stacks several deep neural network (DNN) models on multiple scales to generate a coarse-to-fine registration pipeline, which prevents the backpropagation from falling into a local extremum and resists significant image distortions. We design a novel loss function paradigm based on structural similarity, which makes MU-Net suitable for various types of multimodal images. MU-Net is compared with traditional feature-based and area-based methods, as well as supervised and other unsupervised learning methods on the optical-optical, optical-infrared, optical-synthetic aperture radar (SAR), and optical-map datasets. Experimental results show that MU-Net achieves more comprehensive and accurate registration performance between these image pairs with geometric and radiometric distortions. We share the code implemented by Pytorch at https://github.com/yeyuanxin110/MU-Net .

Most traditional approaches classify hyperspectral image (HSI) pixels relying only on the spectral values of the input channels. However, the spatial context around a pixel is also very important and can enhance the classification performance. In order to effectively exploit and fuse both the spatial context and spectral structure, we propose a novel two-stream deep architecture for HSI classification. The proposed method consists of a two-stream architecture and a novel fusion scheme. In the two-stream architecture, one stream employs the stacked denoising autoencoder to encode the spectral values of each input pixel, and the other stream takes as input the corresponding image patch and deep convolutional neural networks are employed to process the image patch. In the fusion scheme, the prediction probabilities from two streams are fused by adaptive class-specific weights, which can be obtained by a fully connected layer. Finally, a weight regularizer is added to the loss function to alleviate the overfitting of the class-specific fusion weights. Experimental results on real HSIs demonstrate that the proposed two-stream deep architecture can achieve competitive performance compared with the state-of-the-art methods.

The supervised deep networks have shown great potential in improving the classification performance. However, training these supervised deep networks is very challenging for hyperspectral image given the fact that usually only a small amount of labeled samples are available. In order to overcome this problem and enhance the discriminative ability of the network, in this paper, we propose a deep network architecture for a super-resolution (SR)-aided hyperspectral image classification with classwise loss (SRCL). First, a three-layer SR convolutional neural network (SRCNN) is employed to reconstruct a high-resolution image from a low-resolution image. Second, an unsupervised triplet-pipeline CNN (TCNN) with an improved classwise loss is built to encourage intraclass similarity and interclass dissimilarity. Finally, SRCNN, TCNN, and a classification module are integrated to define the SRCL, which can be fine-tuned in an end-to-end manner with a small amount of training data. Experimental results on real hyperspectral images demonstrate that the proposed SRCL approach outperforms other state-of-the-art classification methods, especially for the task in which only a small amount of training data are available.

Variants of deep networks have been widely used for hyperspectral image (HSI)-classification tasks. Among them, in recent years, recurrent neural networks (RNNs) have attracted considerable attention in the remote sensing community. However, complex geometries cannot be learned easily by the traditional recurrent units [e.g., long short-term memory (LSTM) and gated recurrent unit (GRU)]. In this article, we propose a geometry-aware deep recurrent neural network (Geo-DRNN) for HSI classification. We build this network upon two modules: a U-shaped network (U-Net) and RNNs. We first input the original HSI patches to the U-Net, which can be trained with very few images and obtain a preliminary classification result. We then add RNNs on the top of the U-Net so as to mimic the human brain to refine continuously the output-classification map. However, instead of using the traditional dot product in each gate of the RNNs, we introduce a Net-Gated GRU that increases the nonlinear representation power. Finally, we use a pretrained ResNet as a regularizer to improve further the ability of the proposed network to describe complex geometries. To this end, we construct a geometry-aware ResNet loss, which leverages the pretrained ResNet's knowledge about the different structures in the real world. Our experimental results on real HSIs and road topology images demonstrate that our approach outperforms the state-of-the-art classification methods and can learn complex geometries.

Keypoint detection is a crucial step for feature-based image registration. The traditional detectors only extract one type of keypoint such as a corner or a blob, which is not quite beneficial to image registration. Accordingly, this letter presents a novel keypoint detector that aims to simultaneously extract corners and blobs. The proposed detector is named as Harris-Difference of Gaussian (DoG), which combines the advantages of the Harris-Laplace corner detector and the DoG blob detector. In the definition of Harris-DoG, we first build an image scale space and extract the corners by using the multiscale Harris detector. Then, these corners are screened by an automatic scale selection technique based on their DoG responses. This can make the corners robust to scale changes. Meanwhile, DoG also is used to detect the blobs in the image scale space by a nonmaxima suppression scheme. Finally, the scale invariant feature transform (SIFT) descriptors are computed for the detected corners and blobs, and they are applied together for image registration. The proposed Harris-DoG has been tested by using three pairs of multisensor remote sensing images. The experimental results show that Harris-DoG can effectively increase the number of correct matches and improve the registration accuracy compared with the state-of-the-art keypoint detectors.

In recent years, generative adversarial networks (GAN) have made great progress in the field of hyperspectral image classification (HIC), which alleviates the problem of insufficient training samples to a large extent. At present, GAN in the field of HIC are all based on Convolutional Neural Network (CNN). But CNN cannot extract sequence information well, and it is difficult to model remote dependencies. However, hyperspectrum is rich in spectral sequence information, and Transformer has been proven to be good at processing sequence information. Therefore, in order to process spectral information and alleviate the problem of insufficient training samples of hyperspectral images (HSI), we put forward a new frame Transformer with residual upscale GAN (TRUG). TRUG includes a generator G and a discriminator D. In the G, we propose residual upscale (RU) to improve the resolution of generated features, while also extracting texture features and capturing context relationships. In addition, we visualized the generated fake images for more intuitive analysis. In the D, we adopt Transformer block with progressively decreasing scale, and use grid self-attention mechanism in the first layer to better extract features for classification. In addition, GAN are prone to the problem of unstable training. In order to solve this problem, we improve the normalization algorithm and add relative position coding. We applied a pure Transformer based GAN to HIC datasets. Experimental results show that the proposed TRUG model has better performance than other models on the three datasets.