The problem of effectively exploiting the information multiple data sources has become a relevant but challenging research topic in remote sensing. In this article, we propose a new approach to exploit the complementarity of two data sources: hyperspectral images (HSIs) and light detection and ranging (LiDAR) data. Specifically, we develop a new dual-channel spatial, spectral and multiscale attention convolutional long short-term memory neural network (called dual-channel A³CLNN) for feature extraction and classification of multisource remote sensing data. Spatial, spectral, and multiscale attention mechanisms are first designed for HSI and LiDAR data in order to learn spectral- and spatial-enhanced feature representations and to represent multiscale information for different classes. In the designed fusion network, a novel composite attention learning mechanism (combined with a three-level fusion strategy) is used to fully integrate the features in these two data sources. Finally, inspired by the idea of transfer learning, a novel stepwise training strategy is designed to yield a final classification result. Our experimental results, conducted on several multisource remote sensing data sets, demonstrate that the newly proposed dual-channel A³CLNN exhibits better feature representation ability (leading to more competitive classification performance) than other state-of-the-art methods.
4 March 2021 in IEEE Journal of Selected Topics in Signal Processing
Recurrent neural networks, especially the convolutional long short-term memory (ConvLSTM), have attracted plenty of attention and shown promising results due to their ability in modeling long-term dependencies in many research fields. In this paper, a lightweight tensor attention-driven ConvLSTM neural network (TACLNN) is proposed for hyperspectral image (HSI) classification. Firstly, to reduce the trainable parameters and memory requirements of ConvLSTM (specifically, the 2-D version of LSTM, i.e., ConvLSTM2D), a lightweight ConvLSTM2D cell is developed by utilizing tensor-train decomposition, resulting in a TT-ConvLSTM2D cell, with which a spatial-spectral TT-ConvLSTM 2-D neural network (SSTTCL2DNN) is built. However, it is inevitable for SSTTCL2DNN to obtain lower accuracies for HSI classification. To recover the accuracy loss caused by the TT-ConvLSTM2D cell in SSTTCL2DNN, a learnable tensor attention residual block (TARB) module is built to further enhance its geometrical structure. When applied to three widely used HSI benchmarks, the proposed TACLNN model outperforms several state-of-the-art methods for HSI classification. In addition, the proposed TACLNN can effectively reduce the number of parameters and storage requirements achieving higher classification accuracies as compared to other competitive baselines.
30 November 2021 in IEEE Geoscience and Remote Sensing Letters
Recently, graph convolutional networks (GCNs) have been developed to explore the spatial relationship between pixels, achieving better classification performance of hyperspectral images (HSIs). However, these methods fail to sufficiently leverage the relationship between spectral bands in HSI data. As such, we propose an adaptive cross-attention-driven spatial–spectral graph convolutional network (ACSS-GCN), which is composed of a spatial GCN (Sa-GCN) subnetwork, a spectral GCN (Se-GCN) subnetwork, and a graph cross-attention fusion module (GCAFM). Specifically, Sa-GCN and Se-GCN are proposed to extract the spatial and spectral features by modeling the correlations between spatial pixels and between spectral bands, respectively. Then, by integrating attention mechanism into information aggregation of the graph, the GCAFM, including three parts, i.e., the spatial graph attention block, the spectral graph attention block, and the fusion block, is designed to fuse the spatial and spectral features, and suppress noise interference in Sa-GCN and Se-GCN. Moreover, the idea of the adaptive graph is introduced to explore an optimal graph through backpropagation during the training process. Experiments on two HSI datasets show that the proposed method achieves better performance than other classification methods.
4 August 2021 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
As a basic research topic in the field of remote sensing, semantic segmentation of high-resolution aerial images has broad application prospects. However, most existing semantic segmentation methods usually extract multiscale features of images in a hierarchical manner and fail to make full use of the surface information from high-resolution remote sensing images. To address the above problem, we propose a novel dual-channel scale-aware segmentation network with position and channel attentions (DSPCANet) for high-resolution aerial images, which contains an Xception branch and a digital surface model-based position and channel attention fusion (DSMPCF) branch to process the near-infrared, red, and green (IRRG) spectral images and DSM images, respectively. First, the inner residual block (R2_Block) module represents the multiscale features at the granularity level and increases the range of the receptive field of each network layer. Furthermore, channel attention module (CAM) and improved position attention module (IPAM) are developed to embed into the DSMPCF branch to learn the geographic feature representation from the DSM images, while the Xception branch is applied to process the IRRG spectral images. Finally, in the fusion part of the proposed model, IPAM and CAM are further utilized to effectively model the fusion features from the spatial and channel dimensions, obtain the class-based correlation, and recalibrate the class-level information. The proposed DSPCANet model is evaluated on the ISPRS Vaihingen and Potsdam datasets, and the extensive experiments demonstrate that it is more accurate and efficient than other state-of-the-art methods.
6 July 2022 in IEEE Transactions on Geoscience and Remote Sensing
Convolutional long short-term memory (ConvLSTM) has received much attention for hyperspectral image (HSI) classification due to its ability of modeling long-range correlations, which, however, is vulnerable to too many parameters and insufficient training, limiting its classification accuracy, especially for small samples. Different from it, traditional hand-crafted methods extract the features with basic attributes of HSIs, which can provide the lack of details and interpretability of deep semantic features. However, existing methods fail to incorporate their complementarity for HSI classification. As such, a Pseudo complex-valued (CV) Deformable ConvLSTM Neural Network with mutual Attention learning (APDCLNN) is proposed, providing a new way to realize the collaborative learning of hand-crafted and deep features for HSI classification. First, a 2-D pseudo CV deformable ConvLSTM (PDConvLSTM2D) cell is designed using deformable convolution and complex operations, with which a spatial–spectral PDConvLSTM2D neural network (SSPDCL2DNN) is built to extract scale- and spectral-enhanced deep spatial–spectral features. Then, 3-D Gabor filter is used to extract hand-crafted features, and a mutual attention-based multimodality feature learning and fusion (MAMLF) module is designed to integrate them into deep features for training and optimization of SSPDCL2DNN. Finally, an attention loss subnetwork is designed to refine the classification results. As we know, this is the first attempt to apply the idea of mutual attention learning to fuse hand-crafted and deep features for HSI classification. Extensive experiments on three widely used HSI datasets show the advantages of our model over other deep methods in terms of both quantitative and visual quality.
18 March 2021 in IEEE Geoscience and Remote Sensing Letters
Deep neural networks have achieved promising performance for hyperspectral image (HSI) classification. However, due to the limitation of the available labeled samples, the traditional deeper and wider neural networks usually cause the overfitting problem and lose the detailed information. To solve this problem, a brain-like structure, namely spatial attention-driven recurrent feedback convolutional neural network (SARFNN), is proposed by utilizing the recurrent feedback and attention mechanism structures, from which two deep models are further developed for HSI classification. First, a 2-D SARFNN (SARF2DNN) model is developed to learn the spatial features from HSI data. After that, to better exploit the 3-D characteristic, the 3-D version is extended from SARF2DNN, thus constructing an SARF3DNN model to extract joint spatial-spectral features. Moreover, with the help of the idea of brain-likeness, the recurrent feedback module is designed to recover information loss caused by deeper structure and the dimension reduction operation. The experimental results conducted on two HSI data sets show that our SARFNN architecture can achieve more competitive performance than other state-of-the-art algorithms.