29 November 2024 in IEEE Transactions on Circuits and Systems for Video Technology
Constrained by imaging systems, hyperspectral images (HSIs) always have a low spatial resolution. Deep learning-based HSI super-resolution methods have achieved impressive results through learning the nonlinear mapping between low-resolution (LR) and high-resolution (HR) images. However, most of them take the LR image or its upsampled version through bicubic interpolation as input, leading to low-quality features and limited details captured by the network. As a powerful generative model, diffusion model has the ability to learn both contextual semantics and textual details from distinct timesteps, enabling the effective exploration of spatial-spectral distributions in high-dimensional data. In this paper, we propose a novel method that extracts high-quality prior information from original images to assist in super-resolution through pretraining a diffusion model. Specifically, we first train a diffusion model using original HSI patches in a self-supervised manner and then obtain prior features from the pretrained denoising U-Net decoder. To efficiently incorporate the prior features into the super-resolution model, we propose an adaptive fusion module based on spatial and spectral attention mechanisms, which enhances features in both dimensions while preserving the original characteristics. Additionally, to leverage the complementarity of spatial and spectral information, we design a spatial-spectral aggregation Transformer module that incorporates an adaptive interaction module to facilitate information exchange across different dimensions, thereby enhancing the representation capability. Extensive experiments on three public hyperspectral datasets demonstrate that the proposed method achieves excellent super-resolution performance and outperforms the state-of-the-art methods in terms of quantitative quality and visual results.
10 April 2024 in IEEE Transactions on Geoscience and Remote Sensing
The existing few-shot scene classification (FSSC) algorithms have achieved satisfactory results, but they are limited by the paradigm of centralized machine learning, i.e., private remote sensing data need to be centralized on a certain server for training. However, remote sensing images generally contain sensitive information such as national security and company privacy, so it is realistically difficult to collect remote sensing data from all the parties. Therefore, there is a pressing requirement in FSSC for a novel paradigm to achieve multi-party collaborative learning without compromising remote sensing data privacy. In this paper, we formulate a novel personalized multi-party few-shot learning (PMPFSL) paradigm for remote sensing scene classification. In PMPFSL, different participants can achieve multi-party collaborative learning without sacrificing the privacy of their local data, and their respective local models are able to recognize the unseen remote sensing scene categories with a small number of labeled samples. Importantly, the proposed PMPFSL is applicable to various multi-party learning algorithms and few-shot scene classification networks. Moreover, to address the problems of local model overfitting and poor discriminability of few-shot metrics, we propose the personalized adaptive distillation (PAD) scheme and multi-scale feature matching network (MSFMNet) on PMPFSL, respectively. Specifically, each participant obtains the MSFMNet with initialization parameters, and implements a certain number of local training on their respective private machines. Global aggregation is subsequently achieved by uploading only the local models to the central server. In a new round of local training, the participants realize personalized data adaptation to the global model based on the PAD. Overall, the proposed PMPFSL customizes a personalized few-shot model for each participant that is more tailored to their respective remote sensing scenarios. The experimental results demonstrate that our PMPFSL is superior in three benchmark FSSC datasets. We also extensively studied and analyzed the contributions of PAD and MSFMNet in the proposed PMPFSL framework.
27 January 2025 in IEEE Transactions on Geoscience and Remote Sensing
Hyperspectral Images (HSIs) capture extensive spatial and spectral information, facilitating detailed Change Detection (CD) of complex land covers. However, the high correlation among spectral data can lead to information redundancy, increasing processing dimensions and introducing irrelevant or detrimental data to CD. To address these challenges, we propose an Adaptive Center-Focused Hybrid Attention Network (ACFHAN) for CD in HSIs. This network adaptively emphasizes the spatial regions and spectral channels most pertinent to CD while suppressing irrelevant information. The architecture establishes an end-to-end mapping from the two HSIs to the change results, featuring multiple Center-Focused Hybrid Attention Blocks (CFHABs). Each CFHAB integrates two different attention modules, including an Adaptive Spatial-Spectral Hybrid Self-Attention (S2HSA) module that dynamically adjusts spatial-spectral feature weights and a Center-Focused Attention (CFA) module that enhances the area most relevant to the center pixel to be classified. Additionally, to tackle the challenges of expensive labeling, we further designed a multi-scale superpixel-based data augmentation method which combines traditional unsupervised and supervised methods to provide sufficient low-cost but high-confidence labeled data for CD. Experimental results across various HSI CD datasets validate the effectiveness of our proposed method.
29 July 2024 in IEEE Transactions on Geoscience and Remote Sensing
Self-supervised learning, which leverages unlabeled data to learn useful feature representations by constructing auxiliary tasks, has been widely explored in few-shot scene classification to improve the feature representation and generalization capabilities of deep models in scarce data. However, most of the current related work adopts specific self-supervised auxiliary tasks (SSATs) for combinatorial improvement, and does not explore the intrinsic connection between different pretext tasks. In practice, the linkage of SSATs is complex, and the optimization of task-sharing parameters by minimizing linear combinations of losses can be conflicting. In addition, although a single combination of SSAT can improve certain performance on the baseline, it is not the personalized optimal solution on various remote sensing datasets with diverse properties. In this paper, we propose a collaborative self-supervised evolution (so-called CSENet) framework for few-shot remote sensing scene classification to automatically search for appropriate weights in balancing the task conflicts. In contrast to most existing methods, which consider all SSATs to be equally efficacious or fixed-weighted for the few-shot main task, CSENet achieves autonomous co-evolutionary optimization by encoding arbitrary self-supervised weights. Specifically, the complex self-supervised combinations for different remote sensing data are transformed into an evolutionary optimization problem, where chromosomes with weighting variables obtain the optimal combination with genetic operators. Based on the transfer learning few-shot training paradigm, CSENet first efficiently searches for optimal self-supervised combinations with potential by the proposed automatic collaborative evolution strategy, and automatically adjusts the weights without manual settings. Importantly, CSENet provides both inductive and transductive inference, and supports the embedding of arbitrary SSATs. The effectiveness of the proposed framework is demonstrated by state-of-the-art (SOTA) results on three benchmark datasets.
12 December 2024 in IEEE Transactions on Geoscience and Remote Sensing
Using multimodal remote sensing images for change detection (CD) can significantly improve the feasibility and reliability in challenging environments. However, the differences in imaging mechanisms make multimodal images highly heterogeneous. A key challenge for multimodal CD (MCD) is that the heterogeneity of the modalities and changes in ground objects are intertwined during processing. To address this issue, this article proposes a change masked modality alignment network (CMMAN), which uses a multitask framework consisting of one CD branch and two image modal transformation (IMT) branches. Specifically, to ensure a unified feature space, bi-temporal multimodal images are first input into the same Swin-Transformer-based encoder. The extracted features are then fed simultaneously into the CD branch and separately into the two IMT branches. In the CD branch, the decoder is also designed based on the Swin-Transformer, and a weakly modality-correlated feature enhancement (WMCFE) module is introduced to mitigate the interference of modality heterogeneity on CD. For the two IMT branches, both employ a generative adversarial network (GAN) to transform between modalities, and the distributions of features from different modalities are aligned through simultaneous optimization. Uniquely, the change probability map predicted by the CD branch is utilized to mask the change regions in IMT, further decoupling ground object changes and modal heterogeneity. Experimental results on multiple public datasets demonstrate that the proposed CMMAN significantly improves MCD performance and shows good compatibility and portability with various common backbone networks.
28 April 2025 in IEEE Transactions on Geoscience and Remote Sensing
Detecting land cover changes from multi-temporal and multi-modal remote sensing images acquired by different sensors at the same location is a complex yet highly valuable task. Recently, diffusion models, exemplified by the Denoising Diffusion Probabilistic Model (DDPM), have garnered significant attention for their remarkable performance and straightforward architecture. These models excel in image generation, distribution modeling, and feature extraction, making them highly promising for advancing Multimodal Change Detection (MCD). In this paper, we propose a Dual-stream Denoising Diffusion Probabilistic Model (D3PM) to address the challenges of MCD. Specifically, D3PM leverages DDPM to design two distinct processing streams, one for each image modality. The first stream employs an unconditional DDPM, whose denoising encoder-decoder network can achieve robust feature extraction. The second stream employs a conditional DDPM to facilitate modal translation, enabling the extracted features to align with the characteristics of the other modality, thereby improving cross-modality comparability. To further enhance performance, we constructed a CD task branch based on the decoder features of the two DDPMs across multiple denoising time steps. Additionally, we designed a collaborative learning optimization strategy with asynchronous time steps, fostering cross-task knowledge sharing and mutual enhancement while preserving the integrity of individual task learning. Experimental results on multiple public datasets demonstrate the effectiveness and superiority of the proposed D3PM, which achieves efficient modal transformation and alignment, mitigates modal heterogeneity interference, and significantly improves detection performance.