Deep learning approaches have contributed to the rapid development of Remote Sensing (RS) image interpretation. The most widely used training paradigm is to utilize ImageNet pre-trained models to process RS data for specified tasks. However, there are issues such as domain gap between natural and RS scenes, and the poor generalization capacity of RS models. It makes sense to develop a foundation model with general RS feature representation. Since a large amount of unlabeled data is available, the self-supervised method has more development significance than the fully supervised method in remote sensing. However, most of the current self-supervised methods use contrastive learning, whose performance is sensitive to data augmentation, additional information, and selection of positive and negative pairs. In this paper, we leverage the benefits of generative self-supervised learning for RS images, and propose a Remote sensing foundation Model framework called RingMo, which consists of two parts. First, a large-scale dataset is constructed by collecting two million RS images from satellite and aerial platforms, covering multiple scenes and objects around the world. Second, we propose an RS foundation model training method designed for dense and small objects in complicated RS scenes. We show that the foundation model trained on our dataset with RingMo method achieves state-of-the-art on eight datasets across four downstream tasks, demonstrating the effectiveness of the proposed framework. Through in-depth exploration, we believe it is time for RS researchers to embrace generative self-supervised learning and leverage its general representation capabilities to speed up the development of RS applications.

The proposal of Segment Anything Model (SAM) has created a new paradigm for deep learning-based semantic segmentation field, and has shown amazing generalization performance. However, we find it may fail or perform poorly on multimodal remote sensing scenarios, especially the Synthetic Aperture Radar (SAR) images. Besides, SAM does not provide category information of objects. In this paper, we propose a foundation model for multimodal remote sensing image segmentation called RingMo-SAM, which can not only segment anything in optical and SAR remote sensing data, but also identify object categories. First, a large-scale dataset containing millions of segmentation instances is constructed by collecting multiple open-source datasets in this field to train the model. Then, by constructing an instance-type and terrain-type category-decoupling mask decoder, the category-wise segmentation of various objects is achieved. In addition, a prompt encoder embedded with the characteristics of multimodal remote sensing data is designed. It not only supports multi-box prompts to improve the segmentation accuracy of multi-objects in complicated remote sensing scenes, but also supports SAR characteristics prompts to improve the segmentation performance on SAR images. Extensive experimental results on several datasets including iSAID, ISPRS Vaihingen, ISPRS Potsdam, AIR-PolSAR-Seg, etc. have demonstrated the effectiveness of our method.

Few-shot object detection (FSOD) has received numerous attention due to the difficulty and time-consuming of labeling objects. Recent researches achieve excellent performance in a natural scene by only using a few instances of novel classes to fine-tune the last prediction layer of the model well-trained on plentiful base data. However, compared with natural scene objects with a single direction and small size variety, the direction and size of the objects in remote sensing images (RSIs) vary greatly. The methods proposed for the natural scene cannot be directly applied to RSIs. In this article, we first propose a strong baseline for RSIs. It fine-tunes all detector components acting on high-level features and effectively improves the performance of novel classes. Further analyzing the results of the baseline, we find that the error for novel classes is mainly concentrated in classification. It misclassifies novel classes as confusable base classes or backgrounds due to the difficulty in extracting generalized information from limited instances. As is well-known, text-modal knowledge can highly summarize the generalized and unique characteristics of categories. Thus, we introduce text-modal descriptions for each category and propose an FSOD method guided by TExt-MOdal knowledge, called TEMO. Specifically, a text-modal knowledge extractor and a cross-modal assembly module are proposed to extract text features and fuse the text-modal features into visual-modal features. The fused features greatly reduce the classification confusion of novel classes. Furthermore, we introduce a mask strategy and a separation loss to avoid over-fitting and ambiguity of text-modal features. Experimental results on detection in optical remote sensing images (DIOR), Northwestern Polytechnical University (NWPU), and fine-grained object recognition in high-resolution remote sensing imagery (FAIR1M) illustrate that our TEMO achieves state-of-the-art performance in all settings.

Due to the differences in the feature distribution between classes, when the model learns in a continuous data stream, it will encounter catastrophic forgetting. The incremental learning methods have shown great potential to solve this problem. However, most existing methods based on task-incremental learning are difficult to adapt to characteristics of remote sensing scenes with few differences in appearance but large differences in features, which is not conducive to artificially distinguish task-identity document (ID). Thus, we propose a class-incremental learning (CIL) network for small objects enhancing semantic segmentation in aerial imagery. Specifically, considering the superior accuracy of the binary classifier, we propose a twin-auxiliary (TA) model that adds an auxiliary binary classification task. Then, for expansion and contraction at the edge and small object confusion problems, we introduce a diversity distillation loss, using the results of binary-classifier to constrain the multiclass segmentation results and strengthen the attention to the locations of the segmentation results that have changed. Finally, we design a conflict reduction mechanism for multihead classifier to achieve single-head prediction for CIL. Experiments demonstrate that our method has good performance on the Vaihingen and Potsdam datasets by the International Society for Photogrammetry and Remote Sensing (ISPRS), outperforming state-of-the-art (SOTA) incremental learning methods. The code will be available soon.

Remarkable progress based on deep neural networks has been achieved on the semantic segmentation in remote sensing images. However, pixel-level labeling is expensive for remote sensing images. Semi-supervised semantic segmentation becomes an alternative approach to reduce the cost of annotation, and it is crucial to utilize efficiently a large number of unlabeled data. Nevertheless inevitably, there is the unbalanced class distribution between labeled and unlabeled data of remote sensing scene. Existing semi-supervised methods train unlabeled images in isolation from labeled images and only learn reliable pixel pseudo-labels, leading to underutilization of unlabeled images. This article proposes a novel semi-supervised semantic segmentation approach based on label propagation and contrastive regularization for remote sensing images. Specifically, the unlabeled images are augmented by randomly copy-pasting the class regions from labeled images. A prototype feature constraint module is used to enforce the constraint on the pixel features of unlabeled images relying on the prototype features from labeled images, achieving feature alignment on the entire dataset. Furthermore, we present the region contrastive learning module that guides the model to learn feature consistency under different perturbations and compact feature representations over class regions on unlabeled images. Extensive experimental results on multiple remote sensing datasets demonstrate that our proposed approach achieves superior performance compared with state-of-the-art semi-supervised semantic segmentation methods.

The essence of improving the effect of cross-modal image–text retrieval (CIR) lies in the finer-grained modeling of homogeneous features between modalities. However, in remote sensing (RS) scenarios, existing methods usually apply the image–sentence granular feature alignment paradigm, bringing significant difficulties to the fine-grained representation of homogeneous features between modalities. Besides, more complex background noise and extreme scale ranges of foreground targets are hard to distinguish, causing the feature mottle problem. To address the above issues, we propose a novel Semantic-guided Image–text Retrieval framework with Segmentation (SIRS). It is a multitask joint learning framework for plug-and-play and end-to-end training RS CIR models efficiently, including semantic-guided spatial attention (SSA) and adaptive multiscale weighting (AMW) modules. First, SSA introduces a background reconstruction (BR) branch based on noise perception and a semantic segmentation (SS) branch based on pixel-level prediction. It explores a joint learning strategy that concisely filters background noise and refines foreground features considerably. Second, AMW performs multiscale weighting on various layers of feature map output by the encoder, effectively improving the learning efficiency of foreground targets at different scales. It is worth mentioning that SIRS outputs combination results with image and segmentation mask, which is not available in other methods. Based on the RSITMD dataset, we complete the SS annotation RSITMD-SS to verify the performance of the proposed method. Sufficient and complete experiments verify the effectiveness of the proposed method. With SIRS, the mainstream SVP and CLIP-based methods improve about 7 mR and derive segmentation prediction with acceptable computational cost optionally. The code and associated dataset will be available at https://github.com/StarBurstStream0/SIRS .