With the advent of the era of high-resolution remote sensing, semantic segmentation methods for solving pixel-level classification have been widely studied. Deep learning has significantly advanced deep feature extraction methods, becoming widely employed in remote sensing image analysis. Deep feature fusion methods is able to effectively combine features from different sources. Optical and synthetic aperture radar (SAR) images stand out as primary data sources in remote sensing, offering complementary and consistent information. Fusion of deep semantic features of optical and SAR images can alleviate the limitations of single-source images in application and improve semantic segmentation accuracy. Therefore, this article reviews the research on deep fusion of optical and SAR images in semantic segmentation tasks from four aspects. Firstly, we provide a summary of challenges and research methods pertinent to semantic segmentation of remote sensing images. Then the challenges and urgent needs of deep feature fusion of optical and SAR images are analyzed, and current research is summarized from the perspective of structural design by studying various feature fusion strategies. Additionally, the compilation and in-depth analysis of open-source optical and SAR datasets suitable for semantic segmentation are undertaken, serving as fundamental resources for future research endeavors. Finally, the article identifies the major challenges summarized from the literature review in this field, outlining expectations and potential future directions for researchers.

In practical SAR target detection applications, it is often encountered that the training and testing data come from different SAR sensors, leading to a decline in SAR target detection performance. Although domain adaptation methods can achieve model generalization through feature transfer, the change of scattering characteristics for the same target and the difference of feature distribution, caused by cross-sensor, cannot be ignored in SAR images. It is inevitable to lead to the escalation of the offset in the bounding box regression and deviation of the feature alignment. To address these issues, a cross-sensor SAR image target detection method based on dynamic feature discrimination and center-aware calibration is proposed. Based on the domain adaptation framework, initially, a Dynamic Feature Discrimination Module (DFDM) is introduced to address the exacerbated offset in the regression. A bidirectional spatial feature aggregation mechanism is employed to aggregate features in both horizontal and vertical directions and a multi-scale structure is adopted to enhance the scattering and semantic features, which can dynamically constrain the target position while improving target discrimination capability. Then, the Center-Aware Calibration Module (CACM) is designed to address the alignment deviation in feature transfer. The target salience relationship is modeled based on the distance between different positions and the target center to suppress background clutter interference. The perception center of the target is focused by combining the centerness map and classification map, which can calibrate the domain-invariant features and alleviate misalignment. Finally, the proposed method is tested on two datasets, MiniSAR and FARAD, and compared with the latest domain adaption methods. Both mAP and F1 values have improved by more than 6%-20%, verifying the effectiveness of the proposed method.

SAR-optical images from different sensors can provide consistent information for scene classification. However, the utilization of unlabeled SAR-optical images in deep learning-based remote sensing image interpretation remains an open issue. In recent years, contrastive self-supervised learning (CSSL) methods have shown great potential for obtaining meaningful feature representations from massive amounts of unlabeled data. This paper investigates the effectiveness of CSSL-based pretraining models for SAR-optical remote-sensing classification. Firstly, we analyze the contrastive strategies of single-source and multi-source SAR-optical data augmentation under different CSSL architectures. We find that the CSSL framework without explicit negative sample selection naturally fits the multi-source learning problem. Secondly, we find that the registered SAR-optical images can guide the Siamese self-supervised network without negative samples to learn shared features, which is also the reason why the CSSL framework outperforms the CSSL framework with negative samples. Finally, we apply the CSSL pretrained network without negative samples that can learn the shared features of SAR-optical images to the downstream domain adaptation task of optical transfer to SAR images. We find that the choice of a pretrained network is important for downstream tasks.

Applying synthetic aperture radar automatic target recognition (SAR ATR) in open scenario based on deep learning (DL) is challenging due to the difficulty in incrementally recognizing new targets with limited samples. To address this challenge, we introduce simulated data that reflects the structure and scattering features of the new target to supplement measured data for better performance, and then, we propose a novel class incremental SAR ATR method guided by simulated data through feature aggregation (SGFA). Due to the gap between simulated and measured data, DL-model prefers extracting simulated-specific features in incremental learning, resulting in misclassification of new targets. In order to avoid the bias learning of simulated data, SGFA utilizes feature aggregation to extract scattering and structural features that are present in both simulated and measured images, which consists of measured data-anchored mini-batch construction strategy (MDA) and feature-level contrastive loss. Specifically, the MDA can reduce the high sampling probability of a large number of simulated samples in each mini-batch. The feature-level contrastive loss can aggregate the feature distributions of simulated and measured data, which is obtained by automatically constructing sample pairs through cyclic shifts of feature vectors in the mini-batch. In addition, a small amount portion of simulated data is retained to resist severe forgetting caused by the difficulty of adequately representing the data distribution with limited measured data. The experiments on SAMPLE dataset demonstrate the effectiveness of the proposed method.

In Synthetic Aperture Radar (SAR) applications, the continuous emergence of new target classes poses a significant challenge to Automatic Target Recognition (ATR) systems. Adapting to the distribution of new data can induce drastic alterations in the feature space of deep models, resulting in a decline in their ability to recognize old data, termed catastrophic forgetting. To address this challenge, we propose a novel class-incremental SAR ATR method based on Physical Attributes Embedded Prototypical Network (PAEPN). PAEPN embeds physical attributes derived from electromagnetic scattering and geometric priors into the deep model to achieve stable representations. These physical attributes, determined by the target's shape, structure, and material composition, remain invariant throughout the incremental learning process, thereby enhancing the stability and interpretability of deep models. Specifically, PAEPN first extracts and integrates physical attribute priors to establish feature anchors, guiding the deep model in extracting physically consistent features and preventing drastic changes in the feature space. Second, a spatial attention enhancement strategy is introduced to enable the deep model to reliably focus on the key regions of SAR targets. Finally, feature relations that represent semantic similarity are distilled to further mitigate catastrophic forgetting. During testing, PAEPN employs the cosine distance between the sample feature and class prototypes for recognition. Comprehensive experiments on three datasets demonstrate that PAEPN outperforms existing state-of-the-art methods.