26 December 2024 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Convolutional neural networks and attention mechanisms have greatly benefited remote sensing change detection (RSCD) because of their outstanding discriminative ability. Existent RSCD methods often follow a paradigm of using a non-interactive Siamese neural network for multi-temporal feature extraction and change detection heads for feature fusion and change representation. However, this paradigm lacks the contemplation of the characteristics of RSCD in temporal and spatial dimensions, and causes the drawback on spatial-temporal interaction that hinders high-quality feature extraction. To address this problem, we present STeInFormer, a spatial-temporal interaction Transformer architecture for multi-temporal feature extraction, which is the first general backbone network specifically designed for RSCD. In addition, we propose a parameter-free multifrequency token mixer to integrate frequency-domain features that provide spectral information for RSCD. Experimental results on three datasets validate the effectiveness of the proposed method, which can outperform the state-of-the-art methods and achieve the most satisfactory efficiency-accuracy trade-off.
1 February 2025 in Information Fusion
23 December 2024 in IEEE Transactions on Geoscience and Remote Sensing
Pan-sharpening reconstructs the high resolution multispectral (HR-MS) image from its corresponding panchromatic (PAN) image and low-resolution multispectral (LR-MS) image. However, existing deep learning-based pan-sharpening methods typically suffer from three challenges: (1) the vanilla LR-MS image upsampling employed by them fail to consider domain knowledge, thereby disregarding crucial information; (2) while remote sensing images exhibit multi-scale complex land features, existing methods fail to fully exploit crucial multi-scale spatial information, i.e., scale transformation layers within their models are not effective; and (3) existing CNN and transformer-based pan-sharpening backbones are constrained by inherent local receptive fields or quadratic computational complexity, making them difficult to balance their effectiveness and efficiency. To address these issues, we propose a novel Supervised Detail-guided Multi-scale State Space Model for Pansharpening, namely SDMSPan. Our SDMSPan consists of three Residual State Space Modules (Res-SSMs) that are responsible for handling image information at three spatial scales, where each Res-SSM aims to model both local and long-range dependencies between PAN and LR-MS images at a specific spatial scale with lower computational cost. Between each pair of Res-SSMs, a novel Detail-Guided Upsampling Block (DGUB) is proposed to apply spatial details of the PAN image to guide effective and taskaware intermediate feature upsampling, where a novel multi-scale intermediate spatial-spectral supervision strategy is also proposed to supervise the training of every DGUB. Experimental results demonstrate that our proposed approach significantly outperforms other state-of-the-art methods in performance. Our code is provided at https://github.com/zhaomengjiao123/SDMSPan.
23 July 2025 in IEEE Transactions on Geoscience and Remote Sensing
Pansharpening aims to generate high-resolution multispectral (HR-MS) images by fusing the corresponding low-resolution multispectral (LR-MS) and high-resolution panchromatic (PAN) images. While the spatial-spectral properties modeling plays crucial roles in generating high-quality HR-MS images, existing approaches suffer from: 1) directly modeling the entangled spatial-spectral properties; and 2) lacking task-specific priors for spatial and spectral properties modeling. This paper proposes a novel Fourier domain-based approach (Fourier-SSR) to address these problems, where phase and amplitude components of PAN and LR-MS are considered to individually model spatial and spectral properties for the target HR-MS image. Our Fourier-SSR is motivated by the crucial findings in Fourier domain: (i) manifestation of spatial-spectral properties, i.e., the spatial and spectral properties of remote sensing images can be individually manifested in their phase and amplitude components in Fourier domain; (ii) spatial property-related prior, i.e., only reconstructing the phase component of PAN image in Fourier domain, could generate spatial property required for target HR-MS image; and (iii) spectral property-related prior, i.e., jointly models the amplitude components of PAN and LR-MS images could generate the required spectral property for target HR-MS image. Based on the aforementioned findings, we design a Fourier-guided spatial Mixer and a Fourier-guided spectral Mixer, which innovatively employ complex feature interaction strategies to individually reconstruct the phase and amplitude components for target HR-MS. Experiments show that our methods unleash Fourier domain potential in individually modeling spatial and spectral properties for the target HR-MS image, leading to superior performance over previous state-of-the-art. Our code is provided in Supplementary Material.
23 December 2024 in arXiv
Convolutional neural networks and attention mechanisms have greatly benefited remote sensing change detection (RSCD) because of their outstanding discriminative ability. Existent RSCD methods often follow a paradigm of using a non-interactive Siamese neural network for multi-temporal feature extraction and change detection heads for feature fusion and change representation. However, this paradigm lacks the contemplation of the characteristics of RSCD in temporal and spatial dimensions, and causes the drawback on spatial-temporal interaction that hinders high-quality feature extraction. To address this problem, we present STeInFormer, a spatial-temporal interaction Transformer architecture for multi-temporal feature extraction, which is the first general backbone network specifically designed for RSCD. In addition, we propose a parameter-free multi-frequency token mixer to integrate frequency-domain features that provide spectral information for RSCD. Experimental results on three datasets validate the effectiveness of the proposed method, which can outperform the state-of-the-art methods and achieve the most satisfactory efficiency-accuracy trade-off. Code is available at https://github.com/xwmaxwma/rschange.