31 October 2022 in IEEE Transactions on Circuits and Systems for Video Technology
To mine the spectral-spatial information of target pixel in hyperspectral image classification (HSIC), convolutional neural network (CNN)-based models widely adopt patch-based input pattern, where a patch represents its central pixel and the neighbor pixels play auxiliary roles in the classification process. However, compared to the central pixel, its neighbor pixels often have different contributions for classification. Although many existing patch-based CNNs could adaptively emphasize the spatial neighbor information, most of them ignore the latent relationship between the center pixel and its neighbor pixels. Moreover, efficient spectral-spatial feature extraction has been a difficult yet vital topic for HSIC. To address the mentioned problems, a central vector oriented self-similarity network (CVSSN) is proposed for HSIC. Specifically, based on two similarity measures, we firstly design an adaptive weight addition based spectral vector self-similarity module (AWA-SVSS) in input space and a Euclidean distance based feature vector self-similarity module (ED-FVSS) in feature space to fully mine the central vector oriented spatial relationships. Besides, a spectral-spatial information fusion module (SSIF) is formulated as a new pattern to fuse the central 1D spectral vector and the corresponding 3D patch for efficient spectral-spatial feature learning of the subsequent modules. Moreover, we implement a channel spatial separation convolution module (CSS-Conv) and a scale information complementary convolution module (SIC-Conv) for efficient spectral-spatial feature learning. Extensive experimental results on four popular HSI data sets demonstrate the effectiveness and efficiency of the proposed method compared with other state-of-the-art methods.
10 March 2023 in IEEE Transactions on Geoscience and Remote Sensing
Semantic segmentation is an extremely challenging task in high-resolution remote sensing (HRRS) images as objects have complex spatial layouts and enormous variations in appearance. Convolutional neural networks (CNNs) have excellent ability to extract local features and have been widely applied as the feature extractor for various vision tasks. However, due to the inherent inductive bias of convolution operation, CNNs inevitably have limitations in modeling long-range dependencies. Transformer can capture global representations well, but unfortunately ignores the details of local features and has high computational and spatial complexity in processing high-resolution feature maps. In this paper, we propose a novel hybrid architecture for HRRS image segmentation, termed EMRT, to exploit the advantages of convolution operations and Transformer to enhance multi-scale representation learning. We incorporate the deformable self-attention mechanism in the Transformer to automatically adjust the receptive field, and design an encoder-decoder architecture accordingly to achieve efficient context modeling. Specifically, the CNN is constructed to extract feature representations. In the encoder, local features and global representations at different resolutions are extracted by the CNN and Transformer, respectively, and fused in an interactive manner. Moreover, a separate spatial branch is designed to extract multi-scale contextual information as queries, and global dependencies between features at different scales are efficiently established by the decoder. Extensive experiments on three public remote sensing datasets demonstrate the superiority of EMRT and indicate that the overall performance of our method outperforms state-of-the-art methods. Code is available at https://github.com/peach-xiao/EMRT.
17 April 2024 in IEEE Transactions on Industrial Informatics
Image reconstruction-based methods with autoencoder have been widely used for unsupervised anomaly detection. By training the reconstruction on normal samples, autoencoder is supposed to produce higher reconstruction error for anomalous samples, which is used as an indicator for detecting anomalies. However, since autoencoder adopts the bottleneck layer to reconstruct data, it is hard to control its generalization capability. When the generalization capability is high, anomalous features can be confused with normal features, resulting in accurate reconstruction of anomalous regions as well. In this article, we propose a dual-constraint autoencoder to alleviate the problem of feature confusion through the dual constraint of adversarial learning and global memory bank. Given a sample pair consisting of a normal sample and a synthetic anomaly sample generated from it as input, the proposed autoencoder first forces the encoding of the synthetic anomaly image to be close to the normal encoding by adversarial learning. Then, the synthetic anomaly encoding is rerepresented with items in memory bank, which records normal features that facilitate image restoration, and the new feature is fed to the decoder for image inpainting. Under such constraints, the generalization capability of the autoencoder is suppressed and will not reconstruct the anomaly region well. After that, we feed the synthetic anomaly image together with the repaired image into the proposed adaptive weighted similarity spatial attention-based U-Net and produce an accurate anomaly map with the help of our proposed adaptive weighted similarity spatial attention. Results on the a comprehensive real-world dataset for unsupervised anomaly detection (MVTec AD) dataset and the beanTech anomaly detection dataset dataset show that our framework achieves state-of-the-art performance.
7 April 2023 in IEEE Transactions on Geoscience and Remote Sensing
For the abundant spectral and spatial information recorded in hyperspectral images (HSIs), fully exploring spectral-spatial relationships has attracted widespread attention in hyperspectral image classification (HSIC) community. However, there are still some intractable obstructs. For one thing, in the patch based processing pattern, some spatial neighbor pixels are often inconsistent with the central pixel in land-cover class. For another thing, linear and nonlinear correlations between different spectral bands are vital yet tough for representing and excavating. To overcome these mentioned issues, an adaptive mask sampling and manifold to Euclidean subspace learning (AMS-M2ESL) framework is proposed for HSIC. Specifically, an adaptive mask based intra-patch sampling (AMIPS) module is firstly formulated for intra-patch sampling in an adaptive mask manner based on central spectral vector oriented spatial relationships. Then, based on distance covariance descriptor, a dual channel distance covariance representation (DC-DCR) module is proposed for modeling unified spectral-spatial feature representations and exploring spectral-spatial relationships, especially linear and nonlinear interdependence in spectral domain. Furthermore, considering that distance covariance matrix lies on the symmetric positive definite (SPD) manifold, we implement a manifold to Euclidean subspace learning (M2ESL) module respecting Riemannian geometry of SPD manifold for high-level spectral-spatial feature learning. Additionally, we introduce an approximate matrix square-root (ASQRT) layer for efficient Euclidean subspace projection. Extensive experimental results on three popular HSI data sets with limited training samples demonstrate the superior performance of the proposed method compared with other state-of-the-art methods. The source code is available at https://github.com/lms-07/AMS-M2ESL.
7 July 2022 in IEEE Geoscience and Remote Sensing Letters
Semantic segmentation for remote sensing is a crucial but challenging task. Many supervised semantic segmentation methods rely heavily on a large-scale pixelwise annotated dataset, but it is time-consuming and laborious to provide manual annotation. However, due to the common domain shift of remote-sensing images, a direct transfer might not perform well. Therefore, many unsupervised domain adaptation (UDA) methods have been proposed to solve the data distribution discrepancy in remote-sensing datasets, but these methods cannot completely utilize the features extracted in the training process. In addition, the correlations between feature map channels are crucial for the pixelwise classification task. In this letter, a covariance-based channel attention module is proposed to capture correlations by covariance metric and weighting the feature map channels. To further improve the domain adaptation performance, we propose a three-stage UDA semantic segmentation method for remote-sensing images, and we fine-tune the model that has been trained on the source domain on the target domain via self-training and knowledge distillation (KD). To test the effectiveness of the proposed method, experiments are conducted on the ISPRS 2-D Semantic Labeling dataset and an urban drone dataset (UDD). Our method shows a better performance advantage compared with other state-of-the-art methods.
13 June 2024 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Change detection (CD) aims to identify surface changes from bi-temporal remote sensing (RS) images, which is a crucial and challenging topic in RS. In recent years, RS images CD has achieved significant advancements through the use of convolutional neural networks (CNNs). However, existing deep learning-based CD methods still face some challenges such as blurry boundaries, pseudo changes caused by fluctuations in imaging conditions, and complexity of change objects in the scene. In this study, we present a novel perspective for exploring the cross-temporal interaction in the feature fusion stage and propose a bi-temporal feature exchange and enhancement network (ExNet). Specifically, we argue that the heterogeneity of bi-temporal RS images leads to the failure of the change detection model in some cases. Therefore, we attempt to achieve feature alignment in two aspects by feature exchange. On one hand, we exchange the statistical information of bi-temporal features, facilitating the transfer of their style. On the other hand, bi-temporal features are composed of content correlation embeddings (CCEs) and domain correlation embeddings (DCEs). We design a dynamic low-pass filter (DLF) to partially extract and exchange DCEs in bi-temporal features to achieve the feature distribution alignment. Moreover, a frequency separation enhancement module (FSEM) is proposed, which transforms the fused features into the frequency domain and enhances the corresponding frequency representation. Comprehensive experimental results on three popular CD datasets demonstrate the effectiveness and efficiency of ExNet compared with state-of-the-art methods. The source code will be accessible at https://github.com/liuyk97/ExNet.