Cloud contamination greatly limits the potential utilization of optical images for geoscience applications. An effective alternative is to extract data from synthetic aperture radar (SAR) images to remove clouds due to the strong penetration ability of microwaves. In this article, we propose a novel unified spatial–spectral residual network that utilizes SAR images as auxiliary data to remove clouds from optical images. The method can better establish the relationship between SAR and optical images and be divided into two modules: feature extraction and fusion module and reconstruction module. In the feature extraction and fusion module, a gated convolutional layer is introduced to discriminate cloud pixels from clean pixels, which makes up for the lack of distinguishing ability of vanilla convolutional layers and avoids the error of cloud areas in feature extraction. In the reconstruction module, spatial and channel attention mechanisms are introduced to obtain global spatial and spectral information. The network is tested on three datasets with different spatial resolutions and compositions of land covers to verify the effectiveness and applicability of the method. The results show that the method outperforms other mainstream algorithms that simultaneously use SAR images as auxiliary data with a gain of about 2.3 dB in terms of peak signal-to-noise ratio PSNR on the SEN12MS-CR dataset.

In observations of Earth, the existence of clouds affects the quality and usability of optical remote sensing images in practical applications. Many cloud removal methods have been proposed to solve this issue. Among these methods, synthetic aperture radar (SAR)-based methods have more potential than others because SAR imaging is hardly affected by clouds, and can reflect ground information differences and changes. While SAR images used as auxiliary information for cloud removal may be blurred and noisy, the similar non-local information of spectral and electromagnetic features cannot be effectively utilized by traditional cloud removal methods. To overcome these weaknesses, we propose a novel cloud removal method using SAR-optical data fusion and a graph-based feature aggregation network (G-FAN). First, cloudy optical images and contemporary SAR images are concatenated and transformed into hyper-feature maps by pre-convolution. Second, the hyper-feature maps are inputted into the G-FAN to reconstruct the missing data of the cloud-covered area by aggregating the electromagnetic backscattering information of the SAR image, and the spectral information of neighborhood and non-neighborhood pixels in the optical image. Finally, post-convolution and a long skip connection are adopted to reconstruct the final predicted cloud-free images. Both the qualitative and quantitative experimental results from the simulated data and real data experiments show that our proposed method outperforms traditional deep learning methods for cloud removal.

Optical remote sensing images, as a significant data source for Earth observation, are often impacted by cloud cover, which severely limits their widespread application in Earth sciences. Synthetic aperture radar (SAR), with its all-weather, all-day observation capabilities, serves as a valuable auxiliary data source for cloud removal (CR) tasks. Despite substantial progress in deep learning (DL)-based CR methods utilizing SAR data in recent years, challenges remain in preserving fine texture details and maintaining image visual authenticity. To address these limitations, this study proposes a novel diffusion-based CR method called the Dual-branch Multimodal Conditional Guided Diffusion Model (DMDiff). Considering the intrinsic differences in data characteristics between SAR and optical images, we design a dual-branch feature extraction architecture to enable adaptive feature extraction based on the characteristics of the data. Then, a cross-attention mechanism is employed to achieve deep fusion of the multimodal feature extracted above, effectively guiding the progressive diffusion process to restore cloud-covered regions in optical images. Furthermore, we propose an image adaptive prediction (IAP) strategy within the diffusion model, specifically tailored to the characteristics of remote sensing data, which achieves a nearly 20 dB improvement in PSNR compared to the traditional noise prediction (NP) strategy. Extensive experiments on the airborne, WHU-OPT-SAR, and LuojiaSET-OSFCR datasets demonstrate that DMDiff outperforms SOTA methods in terms of both signal fidelity and visual perceptual quality. Specifically, on the LuojiaSET-OSFCR dataset, our method achieves a remarkable 17% reduction in the FID metric over the second-best method, while also yielding significant enhancements in quality assessment metrics such as PSNR and SSIM.

Due to the all-time and all-weather characteristics of synthetic aperture radar (SAR) data, they have become an important input for optical image restoration, and various cloud removal datasets based on SAR-optical have been proposed. Currently, the construction of multi-source cloud removal datasets typically employs single-polarization or dual-polarization backscatter SAR feature images, lacking a comprehensive description of target scattering information and polarization characteristics. This paper constructs a high-resolution remote sensing dataset, AIR-POLSAR-CR1.0, based on optical images, backscatter feature images, and polarization feature images using the fully polarimetric synthetic aperture radar (PolSAR) data. The dataset has been manually annotated to provide a foundation for subsequent analyses and processing. Finally, this study performs a performance analysis of typical cloud removal deep learning algorithms based on different categories and cloud coverage on the proposed standard dataset, serving as baseline results for this benchmark. The results of the ablation experiment also demonstrate the effectiveness of the PolSAR data. In summary, AIR-POLSAR-CR1.0 fills the gap in polarization feature images and demonstrates good adaptability for the development of deep learning algorithms.

Optical remote sensing images play a crucial role in the observation of the Earth's surface. However, obtaining complete optical remote sensing images is challenging due to cloud cover. Reconstructing cloud-free optical images has become a major task in recent years. This paper presents a two-flow Polarimetric Synthetic Aperture Radar (PolSAR)-Optical data fusion cloud removal algorithm (PODF-CR), which achieves the reconstruction of missing optical images. PODF-CR consists of an encoding module and a decoding module. The encoding module includes two parallel branches that extract PolSAR image features and optical image features. To address speckle noise in PolSAR images, we introduce dynamic filters in the PolSAR branch for image denoising. To better facilitate the fusion between multimodal optical images and PolSAR images, we propose fusion blocks based on cross-skip connections to enable interaction of multimodal data information. The obtained fusion features are refined through an attention mechanism to provide better conditions for the subsequent decoding of the fused images. In the decoding module, multi-scale convolution is introduced to obtain multi-scale information. Additionally, to better utilize comprehensive scattering information and polarization characteristics to assist in the restoration of optical images, we use a dataset for cloud restoration called OPT-BCFSAR-PFSAR, which includes backscatter coefficient feature images and polarization feature images obtained from PoLSAR data and optical images. Experimental results demonstrate that this method outperforms existing methods in both qualitative and quantitative evaluations.