In recent years, convolutional neural network (CNN)-based spatiotemporal fusion (STF) models for remote sensing images have made significant progress. However, existing STF models may suffer from two main drawbacks. Firstly, multi-band prediction often generates a hybrid feature representation that includes information from all bands. This blending of features can lead to the loss or blurring of high-frequency details, making it challenging to reconstruct multi-spectral remote sensing images with significant spectral differences between bands. Another challenge in many STF models is the limited preservation of spectral information during 2D convolution operations. Combining all input channels’ convolution results into a single-channel output feature map can lead to the degradation of spectral dimension information. To address these issues and to strike a balance between avoiding hybrid features and fully utilizing spectral information, we propose a remote sensing image STF model that combines single-band and multi-band prediction (SMSTFM). The SMSTFM initially performs single-band prediction, generating separate predicted images for each band, which are then stacked together to form a preliminary fused image. Subsequently, the multi-band prediction module leverages the spectral dimension information of the input images to further enhance the preliminary predictions. We employ the modern ConvNeXt convolutional module as the primary feature extraction component. During the multi-band prediction phase, we enhance the spatial and channel information captures by replacing the 2D convolutions within ConvNeXt with 3D convolutions. In the experimental section, we evaluate our proposed algorithm on two public datasets with 16x resolution differences and one dataset with a 3x resolution difference. The results demonstrate that our SMSTFM achieves state-of-the-art performance on these datasets and is proven effective and reasonable through ablation studies.
This paper presents a novel scale and gradient aware image smoothing method, particularly effective for removing high-contrast detailed textures while preserving boundary sharpness and fine structures. The core idea of the proposed method is based on an observation that small-scale textures can be removed only depending on a down-then-up scaling(DTUS) operation. Accordingly, we present a selective edge smoothing framework by jointly considering scale and gradient measurements. Specifically, we first compute the structural similarity(SSIM) of the DTUS image and the input image to distinguish the textures and structures from the input image. Then we use the SSIM map as weights to fuse the two images together to achieve a scale aware smoothing result. Furthermore, we use the fusion image as guidance to confine the number of non-zero gradients and perform a guided L0 gradient minimization to achieve gradient aware smoothing. Since our method makes full use of image scale and gradient, it outperforms state-of-the-art image smoothing algorithm, especially in removing high-contrast textures. Since our proposed method can remove the insignificant details while preserving sharp and undistorted structural edges, it is also adaptable to many application scenarios, such as boundary extraction, image abstraction, JPEG artifact removal, and layer decomposition-based editing.
20 August 2011 in JOURNAL OF ELECTRONIC MEASUREMENT AND INSTRUMENT