Convolutional neural networks (CNNs) are powerful in extracting local information but lack the ability to model long-range dependencies. In contrast, transformer relies on multihead self-attention mechanisms to effectively extract the global contextual information and thus model long-range dependencies. In this paper, we propose a novel encoder-decoder structured semantic segmentation network, named as CNN and multiscale transformer fusion network (CMTFNet), to extract and fuse local information and multiscale global contextual information of high-resolution remote sensing images. Specifically, to further process the output features from the CNN encoder, we build a transformer decoder based on the multiscale multihead self-attention (M2SA) module for extracting rich multiscale global contextual information and channel information. Additionally, the transformer block introduces an efficient feed-forward network (E-FFN) to enhance the information interaction between different channels of the feature. Finally, the multiscale attention fusion (MAF) module fully fuses the feature information from different levels. We have conducted extensive comparison experiments and ablation experiments on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam datasets. The extensive experimental results demonstrate that our proposed CMTFNet can obtain superior performance compared to the currently popular methods. The codes will be available at https://github.com/DrWuHonglin/CMTFNet.
14 March 2024 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
The characteristics of remote sensing images, such as complex ground objects, rich feature details, large intra-class variance and small inter-class variance, usually require deep learning semantic segmentation methods to have strong feature learning representation ability. Due to the limitation of convolutional operation, Convolutional Neural Networks (CNNs) are good at capturing local details, but perform poorly at modelling long-range dependencies. Transformers rely on multi-head selfattention mechanisms to extract global contextual information, but it usually leads to high complexity. Therefore, this paper proposes CNN and Multi-scale Local-context Transformer network (CMLFormer), a novel encoder-decoder structured network for remote sensing image semantic segmentation. Specifically, for the features extracted by the lightweight ResNet18 encoder, we design a transformer decoder based on Multi-scale Local-context Transform Block (MLTB) to enhance the ability of feature learning. By using a self-attention mechanism with non-overlapping windows and with the help of multi-scale horizontal and vertical interactive stripe convolution, MLTB is able to capture both local feature information and global feature information at different scales with low complexity. Additionally, the Feature Enhanced Module (FEM) is introduced into the encoder to further facilitate the learning of global and local information. Experimental results show that our proposed CMLFormer exhibits excellent performance on the Vaihingen and Potsdam datasets. The code is available at https://github.com/DrWuHonglin/CMLFormer .
In remote sensing, it is quite necessary to fuse spectral information of low-resolution multispectral (LRMS) images and spatial information of panchromatic (PAN) images for obtaining high-resolution multispectral (HRMS) images. In this paper, an effective fusion method integrating multispectral (MS) image super-resolution and convolutional sparse representation (CSR) fusion is proposed to make full use of the spatial information of remote sensing images. Firstly, for enhancing the spatial information of LRMS images with suitable sizes, a fast iterative image super-resolution algorithm based on the learned iterative shrinkage and thresholding algorithm (LISTA) is exploited in the first stage. It employs a feed-forward neural network to simplify the solution of sparse coefficients in the process of super-resolution. In the fusion stage, we propose a CSR-based image fusion framework, in which each MS super-resolution image and PAN image is decomposed into a basic layer and a detail layer, then we fuse the basic layers and the detail layers of the images, respectively. This hierarchical fusion strategy guarantees the great performance in detail preservation. Experimental results on QuickBird, WorldView-2 and Landsat ETM+ datasets demonstrate that the proposed method outperforms other methods in terms of both objective evaluation and visual effect.
23 November 2023 in IEEE Geoscience and Remote Sensing Letters
Remote-sensing image semantic segmentation is usually based on convolutional neural networks (CNNs). CNNs demonstrate powerful local feature extraction capabilities through stacked convolution and pooling. However, the locality of the convolution operation limits the ability of CNNs to directly extract global information. Relying on the multihead self-attention (MHSA) mechanism, transformer shows great advantages in modeling global information. In this letter, we propose a CNN-transformer fusion network (CTFNet) for remote-sensing image semantic segmentation. CTFNet applies a U-shaped encoder-decoder structure to achieve the extraction and adaptive fusion of local features and global context information. Specifically, a lightweight W/P transformer block is proposed as the decoder to obtain global context information with low complexity and connected to the encoder through the skip connection. Finally, the channel and spatial attention fusion module (AFM) is exploited to adaptively fuse deep semantic features and shallow detail features. On the Vaihingen and Potsdam datasets of the International Society for Photogrammetry and Remote Sensing (ISPRS), the effectiveness of each module is demonstrated by ablation experiments. Compared with several classical networks, our proposed CTFNet can obtain superior performance.
Deep learning, especially convolutional neural network (CNN), has been widely applied in remote sensing scene classification in recent years. However, such approaches tend to extract features from the whole image rather than discriminative regions. This article proposes a self-attention network model with joint loss that effectively reduce the interference of complex backgrounds and the impact of intra-class diversity in remote sensing image scene classification. In this model, self-attention mechanism is integrated into ResNet18 to extract discriminative features of images, which makes CNN focus on the most salient region of each image and suppress the interference of the irrelevant information. Moreover, for reducing the influence of intra-class diversity on scene classification, a joint loss function combining center loss with cross-entropy loss is proposed to further improve the accuracy of complex scene classification. Experiments carried out on AID, NWPU-NESISC45 and UC Merced datasets show that the overall accuracy of the proposed model is higher than that of most current competitive remote sensing image scene classification methods. It also performs well in the case of fewer data samples or complex backgrounds.