Clouds are a very important factor in the availability of optical remote sensing images. Recently, deep learning (DL)-based cloud detection methods have surpassed classical methods based on rules and physical models of clouds. However, most of these deep models are very large, which limits their applicability and explainability, while other models do not make use of the full spectral information in multispectral images, such as Sentinel-2. In this article, we propose a lightweight network for cloud detection, fusing multiscale spectral and spatial features (CD-FM3SFs) and tailored for processing all spectral bands in Sentinel-2A images. The proposed method consists of an encoder and a decoder. In the encoder, three input branches are designed to handle spectral bands at their native resolution and extract multiscale spectral features. Three novel components are designed: a mixed depthwise separable convolution (MDSC) and a shared and dilated residual block (SDRB) to extract multiscale spatial features, and a concatenation and sum (CS) operation to fuse multiscale spectral and spatial features with little calculation and no additional parameters. The decoder of CD-FM3SF outputs three cloud masks at the same resolution as input bands to enhance the supervision information of small, middle, and large clouds. To validate the performance of the proposed method, we manually labeled 36 Sentinel-2A scenes evenly distributed over mainland China. The experiment results demonstrate that CD-FM3SF outperforms traditional cloud detection methods and state-of-the-art DL-based methods in both accuracy and speed.

Cloud detection is a crucial step in the optical satellite image processing pipeline for Earth observation. Clouds in optical remote sensing images seriously affect the visibility of the background and greatly reduce the usability of images for land applications. Traditional methods based on thresholding, multi-temporal or multi-spectral information are often specific to a particular satellite sensor. Convolutional Neural Networks for cloud detection often require labeled cloud masks for training that are very time-consuming and expensive to obtain. To overcome these challenges, this paper presents a hybrid cloud detection method based on the synergistic combination of generative adversarial networks (GAN) and a physics-based cloud distortion model (CDM). The proposed weakly-supervised GAN-CDM method (available online https://github.com/Neooolee/GANCDM) only requires patch-level labels for training, and can produce cloud masks at pixel-level in both training and testing stages. GAN-CDM is trained on a new globally distributed Landsat 8 dataset (WHUL8-CDb, available online doi:https://doi.org/10.5281/zenodo.6420027) including image blocks and corresponding block-level labels. Experimental results show that the proposed GAN-CDM method trained on Landsat 8 image blocks achieves much higher cloud detection accuracy than baseline deep learning-based methods, not only in Landsat 8 images (L8 Biome dataset, 90.20% versus 72.09%) but also in Sentinel-2 images (“S2 Cloud Mask Catalogue” dataset, 92.54% versus 77.00%). This suggests that the proposed method provides accurate cloud detection in Landsat images, has good transferability to Sentinel-2 images, and can quickly be adapted for different optical satellite sensors.

Thin clouds seriously affect the availability of optical remote sensing images, especially in visible bands. Short-wave infrared (SWIR) bands are less influenced by thin clouds, but usually have lower spatial resolution than visible (Vis) bands in high spatial resolution remote sensing images (e.g., in Sentinel-2A/B, CBERS04, ZY-1 02D and HJ-1B satellites). Most cloud removal methods do not take advantage of the spectral information available in SWIR bands, which are less affected by clouds, to restore the background information tainted by thin clouds in Vis bands. In this paper, we propose CR-MSS, a novel deep learning-based thin cloud removal method that takes the SWIR and vegetation red edge (VRE) bands as inputs in addition to visible/near infrared (Vis/NIR) bands, in order to improve cloud removal in Sentinel-2 visible bands. Contrary to some traditional and deep learning-based cloud removal methods, which use manually designed rescaling algorithm to handle bands at different resolutions, CR-MSS uses convolutional layers to automatically process bands at different resolution. CR-MSS has two input/output branches that are designed to process Vis/NIR and VRE/SWIR, respectively. Firstly, Vis/NIR cloudy bands are down-sampled by a convolutional layer to low spatial resolution features, which are then concatenated with the corresponding features extracted from VRE/SWIR bands. Secondly, the concatenated features are put into a fusion tunnel to down-sample and fuse the spectral information from Vis/NIR and VRE/SWIR bands. Third, a decomposition tunnel is designed to up-sample and decompose the fused features. Finally, a transpose convolutional layer is used to up-sample the feature maps to the resolution of input Vis/NIR bands. CR-MSS was trained on 28 real Sentinel-2A image pairs over the globe, and tested separately on eight real cloud image pairs and eight simulated cloud image pairs. The average SSIM values (Structural Similarity Index Measurement) for CR-MSS results on Vis/NIR bands over all testing images were 0.69, 0.71, 0.77, and 0.81, respectively, which was on average 1.74% higher than the best baseline method. The visual results on real Sentinel-2 images demonstrate that CR-MSS can produce more realistic cloud and cloud shadow removal results than baseline methods.

Cloud detection is an important step in the processing of remote sensing images. Most methods based on convolutional neural networks (CNNs) for cloud detection require pixel-level labels, which are time-consuming and expensive to annotate. To overcome this challenge, this letter proposes a novel semisupervised algorithm for cloud detection by training a self-attentive generative adversarial network (SAGAN) to extract the feature difference between cloud images and cloud-free images. Our main idea is to introduce visual attention into the process of generating ``real'' cloud-free images. The training of SAGAN is based on three guiding principles: expansion of attention maps of cloud regions which will be replaced with translated cloud-free images, reduction of attention maps to coincide with cloud boundaries, and optimization of a self-attentive network to handle the extreme cases. The inputs for SAGAN training are the images and image-level labels, which are easier, cheaper, and more time-saving than the existing methods based on CNN. To test the performance of SAGAN, experiments are conducted on the Sentinel-2A Level 1C image data. The results show that the proposed method achieves very promising results with only the image-level labels of training samples.

Multi-spectral remote sensing images are widely used for monitoring the globe. Although thin clouds can affect all optical bands, the influences of thin clouds differ with band wavelength. When processing multi-spectral bands at different resolutions, many methods only remove thin clouds in visible/near infrared bands or rescale multi resolution bands to the same resolution then process them together. The former can not make full use of multi-spectral information, and in the latter the rescaling process will introduce noise. In this paper, a deep learning-based thin cloud removal method which fuses full spectral and spatial features in original Sentinel-2 bands is proposed, named CR4S2. A multi input and output architecture is designed for better fusing information in all bands, and reconstructing the background at original resolutions. In addition, two parallel down-sampling residual blocks are designed to transfer features extracted from different depths to the bottom of the network. Experiments were conducted on a new globally distributed Sentinel-2 thin cloud removal dataset called WHUS2-CRv. The results show that the best averaged PSNR, SSIM, nRMSE and SAM of the proposed method over 12 bands in all 20 testing images were 39.55, 0.9443, 0.0245 and 2.5676°, respectively. Compared with baseline methods, the proposed CR4S2 method can better restore not only the spatial features but also spectral features. This indicates that the proposed method is very promising for removing thin clouds in multispectral remote sensing image at different resolutions.