9 September 2024 in IEEE Transactions on Geoscience and Remote Sensing
Land use/land cover classification with multimodal data has attracted increasing attention. For hyperspectral images (HSI) and light detection and ranging (LiDAR) data, the combination of them can make the classification more accurate and robust. However, how to effectively utilize their respective strengths and integrate them with the classification task is still a challenging problem. In this paper, a multilevel attention dynamic-scale network (MADNet) is proposed. Firstly, in the feature extraction stage, the two modalities are divided into two branches with different scales, which are then fed into the convolutional neural networks (CNN) to learn shallow features. Then, considering the characteristics of the HSI, a spectral angle attention module (SAAM) with low-level attention is designed to highlight surrounding pixels that have similar spectra to the central pixel of the patch. After that, a dynamic-scale selection module (DSSM) is proposed to screen an appropriate scale for the patches by pixel similarity analysis. Next, combining the Transformer and the CNN, a global-local cross-attention module (GLCAM) is devised to investigate the fused deep-level multimodal features. Distinct from the vanilla Transformer, the GLCAM deploys a distance-weight operator to decrease the redundancies at long distances and effectively reduce misclassifications. Extensive experiments on three paired HSI and LiDAR datasets demonstrate that the proposed MADNet has certain advantages over the existing methods.
15 July 2025 in IEEE Transactions on Geoscience and Remote Sensing
The limited availability of annotated training data significantly constrains the classification accuracy of hyperspectral image (HSI) and LiDAR fusion approaches. Although contrastive learning has emerged as a potential solution, current implementations frequently neglect the critical class imbalance issues during unlabeled sample selection. To address the issue, we introduce a novel semi-supervised balanced contrastive learning (SemiBaCon) framework for multi-modal remote sensing image classification. First, we propose a superpixel-based balanced sampling (SPBS) mechanism that fundamentally addresses class imbalance through intelligent pseudo-label generation. By segmenting the HSI data into homogeneous superpixels and implementing intra-region label propagation, the method ensures statistically balanced pseudo-label selection across categories, effectively overcoming the bias introduced by conventional random sampling strategies. Second, our architecture integrates a dual-stream encoder combining convolutional neural networks (CNNs) with Transformers, enabling hierarchical feature extraction from spectral-spatial characteristics of HSI and elevation patterns of LiDAR. This design facilitates the construction of multi-modal positive sample pairs, achieving enhanced representation learning through inter-modal consistency constraints. Third, we develop a pseudo-label guided contrastive learning (PLCL) paradigm that synergistically combines pseudo-label confidence with feature similarity metrics, which effectively reduces intra-class variance and improves decision boundaries in the latent space. Comprehensive evaluations on three benchmark datasets demonstrate the framework’s superior performance compared to the state-of-the-art methods.
7 May 2024 in IEEE Geoscience and Remote Sensing Letters
Extracting rock objects from the surface of celestial bodies in deep space exploration environments is crucial for self-service path planning, navigation of detectors, and regional information evaluation. Most existing image semantic segmentation frameworks decrease the spatial resolution of the feature maps as networks deepen, resulting in limitations in detecting small targets and the inability to accurately segment boundary regions. In this letter, we propose a novel semantic segmentation network based on U-Net network and Transformer for deep space rock images, referred to as Diamond-Unet. This model integrates overcomplete and undercomplete branches and incorporates a global–local feature extraction (GLFE) module based on Transformer and convolutional neural network (CNN) technologies to effectively capture discriminative information. Furthermore, an innovative feature cross-fusion path (FCFP) is introduced to enhance information exchange between the dual-branch networks, enabling the capture of both fine-grained details and coarse-grained semantics in the full-scale image segmentation architecture. Experimental results demonstrate that the Diamond-Unet achieves the mean intersection over union (MIoU) scores of 79.32% and 93.43% on two public datasets, which are superior to the compared methods.
9 May 2025 in IEEE Transactions on Geoscience and Remote Sensing
Deep learning techniques have recently achieved significant success in semantic segmentation for Earth observation tasks using single-modality data within a specific region. However, such optimal conditions are seldom found in real-world applications, leading to subpar performance and limited generalization of existing models when faced with multimodal cross-domain scenarios. To tackle this issue, we introduce a novel segmentation network for remote sensing imagery (RSI) called MC-Seg, which incorporates multilevel deep cross-fusion and adversarial domain adaptation. The MC-Seg framework begins by performing multilevel deep cross-fusion of multimodal data through a complete feature extraction module and three fusion levels: feature, modal, and layer. This approach ensures the thorough utilization of features from various modalities. Following this, the framework integrates a local-global adversarial domain adaptation module to minimize the domain gap, effectively aligning the source and target domains despite the differences in RSI data representations. Experimental validation on the C2Seg dataset shows that our proposed method outperforms existing state-of-the-art techniques.
30 December 2025 in IEEE Transactions on Geoscience and Remote Sensing
Due to inherent sensor discrepancies, hyperspectral image (HSI) and LiDAR data in fusion classification tasks often suffer from spatial misalignment and limited high-quality annotations, which severely constrain classification performance. While some recent methods attempt to address these issues by introducing semantic alignment strategies or contrastive learning (CL), challenges such as inconsistent cross-modal representations and limited semantic generalization still persist. To address these issues, we propose a novel text-assisted adaptive registration and fusion classification network (TARCNet). First, we develop a feature adaptive alignment (FAA) module, which adaptively adjusts LiDAR features to alleviate semantic inconsistency under misregistration. Second, we introduce a text-assisted contrastive learning (TCL) module, which leverages linguistic priors to strengthen cross-modal consistency and improve the discriminability of the learned representations. Third, we incorporate a multi-loss joint optimization (MLO) module to ensure consistent and stable optimization across heterogeneous modalities. Extensive experiments conducted on three HSI-LiDAR datasets with misregistration and limited annotations demonstrate that our method outperforms several state-of-the-art approaches, validating its effectiveness and generalization capability.
1 January 2026 in IEEE Transactions on Geoscience and Remote Sensing