The annotations used during the training process are crucial for the inference results of remote sensing images (RSIs) based on a deep learning framework. Unlabeled RSIs can be obtained relatively easily. However, pixel-level annotation is a process that necessitates a high level of expertise and experience. Consequently, the use of small sample training methods has attracted widespread attention as they help alleviate reliance on large amounts of high-quality labeled data and current deep learning methods. Moreover, research on small sample learning is still in its infancy owing to the unique challenges faced when completing semantic segmentation tasks with RSI. To better understand and stimulate future research that utilizes semantic segmentation tasks with small data, we summarized the supervised learning methods and challenges they face. We also reviewed the supervised approaches with data that are currently popular to help elucidate how to efficiently utilize a limited number of samples to address issues with semantic segmentation in RSI. The main methods discussed are self-supervised learning, semi-supervised learning, weakly supervised learning and few-shot methods. The solution of cross-domain challenges has also been discussed. Furthermore, multi-modal methods, prior knowledge constrained methods, and future research required to help optimize deep learning models for various downstream tasks in relation to RSI have been identified.

Synthetic aperture radar (SAR) and optical images provide highly complementary ground information. The fusion of SAR and optical data can significantly enhance semantic segmentation inference results. However, the fusion methods for multimodal data remains a challenge for current research due to significant disparities in imaging mechanisms from diverse sources. Our goal was to bridge the significant gaps between optical and SAR images by developing a dual-input model that utilizes image-level fusion. To improve most existing state-of-the-art image fusion methods, which often assign equal weights to multiple modalities, we employed the principal component analysis (PCA) transform approach. Subsequently, we performed feature-level fusion on shallow feature maps, which retain rich geometric information. We also incorporated a channel attention module to highlight channels rich in features and suppress irrelevant information. This step is crucial due to the substantial similarity between SAR and optical images in shallow layers such as geometric features. In summary, we propose a generic multimodal fusion strategy that can be attached to most encoding–decoding structures for feature classification tasks, designed with two inputs. One input is the optical image, and the other is the three-band fusion data obtained by combining the PCA component of the optical image with the SAR. Our feature-level fusion method effectively integrates multimodal data. The efficiency of our approach was validated using various public datasets, and the results showed significant improvements when applied to several land cover classification models.

Building extraction (BE) and change detection (CD) from remote sensing (RS) imagery are significant yet highly challenging tasks with substantial application potential in urban management. Learning representative multi-scale features from RS images is a crucial step toward practical BE and CD solutions, as in other DL-based applications. To better exploit the available labeled training data for representation learning, we propose a multi-task learning (MTL) network for simultaneous BE and CD, comprising the state-of-the-art (SOTA) powerful Swin transformer as a shared backbone network and multiple heads for predicting building labels and changes. Using the popular CD dataset the Wuhan University building change detection dataset (WHU-CD), we benchmarked detailed designs of the MTL network, including backbone and pre-training choices. With a selected optimal setting, the intersection over union (IoU) score was improved from 70 to 81 for the WHU-CD. The experimental results of different settings demonstrated the effectiveness of the proposed MTL method. In particular, we achieved top scores in BE and CD from optical images in the 2021 Gaofen Challenge. Our method also shows transferable performance on an unseen CD dataset, indicating high label efficiency.

Stereo matching is a fundamental task in three-dimensional scene reconstruction. Recently, deep learning-based methods have proven effective on some benchmark datasets, such as KITTI and SceneFlow. Unmanned aerial vehicles (UAVs) are commonly used for surface observation, and the images captured are frequently used for detailed 3D reconstruction because of their high resolution and low-altitude acquisition. Currently, mainstream supervised learning networks require a significant amount of training data with ground-truth labels to learn model parameters. However, owing to the scarcity of UAV stereo-matching datasets, learning-based stereo matching methods in UAV scenarios is not fully investigated yet. To facilitate further research, this study proposes a pipeline for generating accurate and dense disparity maps using detailed meshes reconstructed based on UAV images and LiDAR point clouds. Through the proposed pipeline, we constructed a multi-resolution UAV scenario dataset called UAVStereo, with over 34,000 stereo image pairs covering three typical scenes. To the best of our knowledge, UAVStereo is the first stereo matching dataset for UAV low-altitude scenarios. The dataset includes synthetic and real stereo pairs to enable generalization from the synthetic domain to the real domain. Furthermore, our UAVStereo dataset provides multi-resolution and multi-scene image pairs to accommodate various sensors and environments. In this study, we evaluated traditional and state-of-the-art deep learning methods, highlighting their limitations in addressing challenges in UAV scenarios and offering suggestions for future research. Our dataset is available at https://github.com/rebecca0011/UAVStereo.git.

Building extraction is a challenging task in Remote Sensing Images (RSI) interpretation. Fusing RSI from different sources, such as high-resolution RSI and LiDAR, is a common strategy to improve the building extraction accuracy. However, the acquisition of multi-source registered RSI of the same region is time-consuming and laborious, which limits the practical application of multi-source fusion strategy. Benefiting from the success of large models in different fields in recent years, we proposed to extract depth information from large models and fuse it with corresponding RSI to improve the building extraction accuracy. Directly introducing the extracted depth information into the original image is not helpful for improving the building extraction accuracy. Therefore, we designed a dedicated Deep Feature Fusion Module (DFFM) for deep feature fusion. Specifically, we designed a Depth Feature Fusion Network (DFF-Net) composed of a Depth Information Extraction Module (DIEM), a DFFM and a U-Net. The DFF-Net successfully improves building extraction accuracy by fusing the extracted depth information. To evaluate the validity of this method, we conducted abundant building extraction experiments on the WHU, Massachusetts, and Inria building datasets. Compared with comparative methods, our method obtained the highest IoU on all these three RSI building datasets, which demonstrates its simplicity and effectiveness.