6 December 2022 in IEEE Geoscience and Remote Sensing Letters
Deep learning has achieved impressive results on Hyperspectral image (HSI) classification, which generally requires sufficient training samples and a huge number of parameters. However, it is challenging to label HSIs, and likely only a few samples are available in practice. Learning a large number of parameters by the model is also resource-intensive. This paper proposes an HSI classification model that achieves promising classification performance with fewer parameters in few-shot settings. The proposed model adopts the residual 3D-CNN as feature extraction network, and contrastive learning is introduced to learn more discriminative representations for HSIs which can conquer the obstacles from HSIs’ high inter-class similarity and large intra-class variance. The proposed few-shot contrastive learning HSI classification model is tested on five popular HSI datasets and outperforms the state-of-the-art models.
22 August 2022 in IEEE Transactions on Instrumentation and Measurement
Medical image fusion aims to derive complementary information from medical images with different modalities and is becoming increasingly important in clinical applications. The design of fusion strategy plays a key role in achieving high-quality fusion results. Existing methods usually employ handcrafted fusion rules or convolution-based networks to fuse multimodal medical images. However, these fusion strategies are insufficiently fine-grained and cannot capture global information of multimodal images effectively. Moreover, for deep learning-based fusion methods, they always concatenate source images/deep features of different modalities to input into the neural network, easily leading to inadequate information utilization from source images. To address these problems, we propose a transformer-based end-to-end framework for medical image fusion, termed TransFusion. The proposed TransFusion introduces the transformer as the fusion strategy, which utilizes its self-attention mechanism to incorporate the global contextual information of multimodal features and fuse them adequately. Besides, unlike traditional parallel multibranch architectures or shared networks used for multiple inputs, we design branch networks that interact through fusion transformers at multiple scales to utilize the information of different modalities more adequately. A natural advantage of our design is the ability to aggregate global multimodal features through self-attention. Both qualitative and quantitative experiments demonstrate the superiority of our method over the state-of-the-art fusion methods.
24 July 2023 in IEEE Transactions on Artificial Intelligence
Domain generalization (DG) aims to train a global model from different but related domains, which can be generalized to an unseen out-of-distribution domain. Most existing DG methods are based on the centralized learning paradigm, raising the privacy leakage concern. In this paper, we propose a contrastive-enhanced domain generalization framework in the federated learning paradigm (FedCDG), where there are a server and multiple clients. Each client owns data from one domain and builds a local model consisting of a domain-invariant feature extractor and a classifier head. The server generates a global model through aggregating and broadcasting local models' parameters, thus achieving knowledge sharing and keeping data confidential. To enhance the discrimination and generalization ability of the local model, we build an improved instance normalization module that focuses on task-relevant features with less domain-specific information. Moreover, for better class-wise alignment in the embedding space, we propose a prototype-based contrastive loss. Given the limited annotation budget in practice, we also extend the proposed framework into the semi-supervised DG setting (i.e., only 10 labelled samples per class). Experimental results on 3 benchmarks and different backbones show that the proposed framework yields promising performances for both DG and semi-supervised DG in the federated learning paradigm.
5 December 2019 in IEEE Transactions on Neural Networks and Learning Systems
With the increasing popularity of deep convolutional neural networks (DCNNs), in addition to achieving high accuracy, it becomes increasingly important to explain how DCNNs make their decisions. In this article, we propose a CHannel-wise disentangled InterPretation (CHIP) model for visual interpretations of DCNN predictions. The proposed model distills the class-discriminative importance of channels in DCNN by utilizing sparse regularization. We first introduce network perturbation to learn the CHIP model. The proposed model is capable to not only distill the global perspective knowledge from networks but also present class-discriminative visual interpretations for the predictions of networks. It is noteworthy that the CHIP model is able to interpret different layers of networks without retraining. By combining the distilled interpretation knowledge at different layers, we further propose the Refined CHIP visual interpretation that is both high-resolution and class-discriminative. Based on qualitative and quantitative experiments on different data sets and networks, the proposed model provides promising visual interpretations for network predictions in an image classification task compared with the existing visual interpretation methods. The proposed model also outperforms the related approaches in the ILSVRC 2015 weakly supervised localization task.
2 March 2020 in IEEE Transactions on Multimedia
20 July 2021 in IEEE Transactions on Image Processing
With the great success of convolutional neural networks (CNNs), interpretation of their internal network mechanism has been increasingly critical, while the network decision-making logic is still an open issue. In the bottom-up hierarchical logic of neuroscience, the decision-making process can be deduced from a series of sub-decision-making processes from low to high levels. Inspired by this, we propose the Concept-harmonized HierArchical INference (CHAIN) interpretation scheme. In CHAIN, a network decision-making process from shallow to deep layers is interpreted by the hierarchical backward inference based on visual concepts from high to low semantic levels. Firstly, we learned a general hierarchical visual-concept representation in CNN layered feature space by concept harmonizing model on a large concept dataset. Secondly, for interpreting a specific network decision-making process, we conduct the concept-harmonized hierarchical inference backward from the highest to the lowest semantic level. Specifically, the network learning for a target concept at a deeper layer is disassembled into that for concepts at shallower layers. Finally, a specific network decision-making process is explained as a form of concept-harmonized hierarchical inference, which is intuitively comparable to the bottom-up hierarchical visual recognition way. Quantitative and qualitative experiments demonstrate the effectiveness of the proposed CHAIN at both instance and class levels.