High-resolution remote sensing images contain rich color and texture information, but due to the inherent limitations of 2D data, achieving high-quality semantic segmentation remains a challenge. Multimodal data fusion technology has emerged as an effective approach to overcome this issue. To accurately capture the semantic information in remote sensing images, this study designs a multimodal fusion transformer-based DeepLabv3+ model for remote sensing semantic segmentation, named FTransDeepLab. Specifically, the network learns features from two modalities and is inspired by the DeepLab architecture. We extended the encoder by stacking the multi-scale Segformer, encoding the input images into highly representative spatial features. Additionally, we introduced the Multimodal Feature Rectification (MFR) module and the Multimodal Feature Fusion (MFF) module. The MFR, composed of a channel attention module and a spatial attention module, enhances the model’s ability to capture essential features and improves performance by focusing on both global and local contexts. The MFF module utilizes a cross-attention mechanism to optimize the feature fusion process, which enhances representation learning by facilitating the interaction between diverse information and integrates features from different modalities. Finally, in the decoding path, the extracted high-level features are concatenated with low-level features to optimize the feature representation and upsampled to restore the size of input image. Extensive results on two datasets, ISPRS Vaihingen and Potsdam, have confirmed that the proposed FTransDeepLab can achieve superior performance compared to state-of-the-art segmentation methods.

The semantic segmentation of high-resolution remote sensing imagery (RSI) is an essential task for many applications. As a promising unsupervised learning method, unsupervised domain adaptation (UDA) methods remarkably contribute to the advancement of high-resolution RSI semantic segmentation. Previous methods focus on reducing domain shift of orthophotos, suffering from some limitations because the available information in orthophotos is relatively homogeneous. This paper proposes a framework to introduce digital surface model (DSM) data for the unsupervised semantic segmentation of RSI. The proposed method combines RSI with DSM through two modules, namely, multipath encoder (MPE) and multitask decoder (MTD), and aligns global data distribution in the source and target domains with a UDA module. A refined post fusion (RPF) module is proposed in the inference phase to exploit the height information fully for refining the segmentation results. Specifically, MPE is designed to utilize RSI and DSM to train the segmentation network jointly, which iteratively fuses RSI and DSM features at multiple levels to enhance their feature representations. MTD is designed to produce fusion prediction maps by filtering interference information of DSM and yielding accurate segmentation masks of DSM and RSI. Experimental results show that the proposed method substantially improves the semantic segmentation performance on high-resolution RSI and outperforms state-of-the-art methods. This paper provides a methodological reference for fusing multimodal data in various RSI-based unsupervised tasks.

Benefiting from free labeling pixel-level samples, weakly supervised semantic segmentation (WSSS) is making progress in automatically extracting building from high-resolution (HR) remote sensing (RS) imagery. For WSSS methods, generating high-quality pseudomasks is crucial for accurate building extraction.To improve the performance of generating pseudomasks by using image-level labels, this article proposes a weakly supervised building extraction method by combining adversarial climbing and gated convolution. The proposed method optimizes class activation maps (CAMs) by using adversarial climbing strategy, generates accurate class boundary maps by introducing a gated convolution module, and further refines building pseudomasks by fusing pairing semantic affinities and CAMs with a random walk strategy. Experimental results on three datasets—two ISPRS datasets and a self-annotated dataset—demonstrate that the proposed approach outperformed SOTA WSSS methods, leading to improvement of building extraction from HR RS imager. This article provides a new approach for optimizing pseudomasks generation, and a methodological reference for the applications of weakly supervised on RS images.

Building instances extraction is an essential task for surveying and mapping. Challenges still exist in extracting building instances from high-resolution remote sensing imagery mainly because of complex structures, variety of scales, and interconnected buildings. This study proposes a coarse-to-fine contour optimization network to improve the performance of building instance extraction. Specifically, the network contains two special sub-networks: attention-based feature pyramid sub-network (AFPN) and coarse-to-fine contour sub-network. The former sub-network introduces channel attention into each layer of the original feature pyramid network (FPN) to improve the identification of small buildings, and the latter is designed to accurately extract building contours via two cascaded contour optimization learning. Furthermore, the whole network is jointly optimized by multiple losses, that is, a contour loss, a classification loss, a box regression loss and a general mask loss. Experimental results on three challenging building extraction datasets demonstrated that the proposed method outperformed the state-of-the-art methods’ accuracy and quality of building contours.

Image-level weakly supervised semantic segmentation (WSSS) methods have greatly facilitated the extraction of buildings from remote sensing (RS) images. However, the lack of the locations and extents of individual buildings in image-level labels results in some limitations of the methods, especially in the cases of cluttered backgrounds, diverse building shapes and sizes. By utilizing bounding box annotations, a novel WSSS model is developed to improve building extraction from RS images in this paper. Specifically, during the training phase, a multiscale feature retrieval (MFR) module is designed to learn multiscale building features and suppress the background noise inside the bounding box. In the inference phase, multiscale class activation maps (CAM) are generated from multiscale features to achieve accurate building localization. Finally, a pseudo mask generation and correction (PGC) module refines the CAMs to generate and correct the building pseudo masks. Experiments are conducted to examine the proposed model in three datasets, namely, the WHU aerial building dataset, the CrowdAI building dataset, and a self-annotated building dataset. Experimental results demonstrate that the proposed method outperforms baselines, achieving 76.99%, 75.51% and 67.35% in terms of IoU scores on the three challenging datasets, respectively. This paper provides a methodological reference for the application of weakly supervised learning on RS images.