Instance segmentation in remote sensing (RS) imagery aims to predict the locations of instances and represent them with pixel-level masks. Thanks to the more accurate pixel-level information for each instance, instance segmentation has enormous potential applications in resource planning, urban surveillance, and military reconnaissance. However, current RS imagery instance segmentation methods mostly follow the fully supervised paradigm, relying on expensive pixel-level labels. Moreover, remote sensing imagery suffers from cluttered backgrounds and significant variations in target scales, making segmentation challenging. To accommodate these limitations, we propose a semantic attention enhancement and structured model-guided multi-scale weakly supervised instance segmentation network (SASM-Net). Building upon the modeling of spatial relationships for weakly supervised instance segmentation, we further design the multi-scale feature extraction module (MSFE module), semantic attention enhancement module (SAE module), and structured model guidance module (SMG module) for SASM-Net to enable a balance between label production costs and visual processing. The MSFE module adopts a hierarchical approach similar to the residual structure to establish equivalent feature scales and to adapt to the significant scale variations of instances in RS imagery. The SAE module is a dual-stream structure with semantic information prediction and attention enhancement streams. It can enhance the network’s activation of instances in the images and reduce cluttered backgrounds’ interference. The SMG module can assist the SAE module in the training process to construct supervision with edge information, which can implicitly lead the model to a representation with structured inductive bias, reducing the impact of the low sensitivity of the model to edge information caused by the lack of fine-grained pixel-level labeling. Experimental results indicate that the proposed SASM-Net is adaptable to optical and synthetic aperture radar (SAR) RS imagery instance segmentation tasks. It accurately predicts instance masks without relying on pixel-level labels, surpassing the segmentation accuracy of all weakly supervised methods. It also shows competitiveness when compared to hybrid and fully supervised paradigms. This research provides a low-cost, high-quality solution for the instance segmentation task in optical and SAR RS imagery.

The interpretation of optical and synthetic aperture radar (SAR) images in remote sensing is general for many tasks, such as environmental monitoring, marine management, and resource planning. Instance segmentation of optical and SAR images, which can simultaneously provide instance-level localization and pixel-level classification of objects of interest, is a crucial and challenging task in image interpretation. Considering that most current methods for instance segmentation of optical and SAR images rely on expensive pixel-level annotation, we develop a weakly supervised instance segmentation (WSIS) method to balance the visual processing requirements with the annotation cost. First, we decompose the prior knowledge of the mask-aware task in WSIS into three meta-knowledge components: fundamental knowledge, apparent knowledge, and detailed knowledge inspired by human visual perception habits of “whole to part” and “coarse to detailed.” Then, a meta-knowledge-guided weakly supervised instance segmentation network (MGWI-Net) is proposed. In this network, the weakly supervised mask (WSM) head can instantiate both fundamental knowledge and apparent knowledge to perform mask awareness without any annotations at the pixel level. The network also includes a mask information awareness assist (MIAA) head, which can implicitly guide the network to learn detailed information about edges through the boundary-sensitive feature of the fully connected conditional random field (CRF), facilitating the instantiation of detailed knowledge. The experimental results show that the MGWI-Net can efficiently generate instance masks for optical and SAR images and achieve the approximate instance segmentation results of the fully supervised method with about one-eighth of the annotation production time. The model parameters and processing speed of our network are also competitive. This study can provide inexpensive and convenient technical support for applying and promoting instance segmentation methods for optical and SAR images.

Instance segmentation (IS) of remote sensing (RS) images can not only determine object location at the box-level but also provide instance masks at the pixel-level. It plays an important role in many fields, such as ocean monitoring, urban management, and resource planning. Compared with natural images, RS images usually pose many challenges, such as background clutter, significant changes in object size, and complex instance shapes. To this end, we propose a query-based RS image cascade IS network (QCIS-Net). The network mainly includes key components, such as the efficient feature extraction (EFE) module, multistage cascade task (MSCT) head, and joint loss function, which can characterize the location and visual information of instances in RS images through efficient queries. Among them, the EFE module combines global information from the Transformer architecture to solve the problem of long-term dependencies in visual space. The MSCT head uses a dynamic convolution kernel based on the query representation to focus on the region of interest, which facilitates the association between detection and segmentation tasks through a multistage structural design that benefits both tasks. The elaborately designed joint loss function and the use of the transfer-learning technique based on a well-known dataset (MS COCO) can guide the QCIS-Net in training and generating the final instance mask. Experimental results show that the well-designed components of the proposed method have a positive impact on the RS image instance segmentation task. It achieves mask average precision (AP) values of 75.2% and 73.3% on the SAR ship detection dataset (SSDD) and Northwestern Polytechnical University Very-High-Resolution dataset (NWPU-VHR-10 dataset), outperforming the other competitive models. The method proposed in this paper can enhance the practical application efficiency of RS images.