Infrared and visible image fusion aims to describe the same scene from different aspects by combining complementary information of multi-modality images. The existing Generative adversarial networks (GAN) based infrared and visible image fusion methods cannot perceive the most discriminative regions, and hence fail to highlight the typical parts existing in infrared and visible images. To this end, we integrate multi-scale attention mechanism into both generator and discriminator of GAN to fuse infrared and visible images (AttentionFGAN). The multi-scale attention mechanism aims to not only capture comprehensive spatial information to help generator focus on the foreground target information of infrared image and background detail information of visible image, but also constrain the discriminators focus more on the attention regions rather than the whole input image. The generator of AttentionFGAN consists of two multi-scale attention networks and an image fusion network. Two multi-scale attention networks capture the attention maps of infrared and visible images respectively, so that the fusion network can reconstruct the fused image by paying more attention to the typical regions of source images. Besides, two discriminators are adopted to force the fused result keep more intensity and texture information from infrared and visible image respectively. Moreover, to keep more information of attention region from source images, an attention loss function is designed. Finally, the ablation experiments illustrate the effectiveness of the key parts of our method, and extensive qualitative and quantitative experiments on three public datasets demonstrate the advantages and effectiveness of AttentionFGAN compared with the other state-of-the-art methods.

Deep learning has been successfully applied to infrared and visible image fusion due to its powerful ability of feature representation. Existing most deep learning based infrared and visible image fusion methods mainly utilize pure convolution model or pure transformer model, which leads to that the fused image cannot preserve long-range dependencies (global context) and local features simultaneously. To this end, we propose a convolution-guided transformer framework for infrared and visible image fusion (CGTF), which aims to combine the local features of convolutional network and the long-range dependency features of transformer to produce satisfactory fused image. In CGTF, the local features are calculated by convolution feature extraction module, and then the local features are used to guide the transformer feature extraction module to capture the long-range dependencies of the image, which can overcome not only the lack of long-range dependencies that exists in convolutional fusion methods, but also the deficiency of local feature that exists in transformer models. Moreover, the convolution-guided transformer fusion framework can consider the inherent relationship of local feature and long-range dependencies due to the alternate use of convolution feature extraction module and transformer module. In addition, to strengthen local feature propagation, we employ dense connections among convolution feature extraction modules. Ablation experiments demonstrate the effectiveness of convolution-guided transformer fusion framework and loss function. We employ two datasets to compare our method with other nine methods, which includes three traditional methods, five deep learning based methods and one transformer based method. Qualitative and quantitative experiments demonstrate the advantages of our method.

In this work, we propose two novel quantum walk kernels, namely the Hierarchical Aligned Quantum Jensen-Shannon Kernels (HAQJSK), between un-attributed graph structures. Different from most classical graph kernels, the proposed HAQJSK kernels can incorporate hierarchical aligned structure information between graphs and transform graphs of random sizes into fixed-size aligned graph structures, i.e., the Hierarchical Transitive Aligned Adjacency Matrix of vertices and the Hierarchical Transitive Aligned Density Matrix of the Continuous-Time Quantum Walks (CTQW). With pairwise graphs to hand, the resulting HAQJSK kernels are defined by computing the Quantum Jensen-Shannon Divergence (QJSD) between their transitive aligned graph structures. We show that the proposed HAQJSK kernels not only reflect richer intrinsic whole graph characteristics in terms of the CTQW, but also address the drawback of neglecting structural correspondence information that arises in most R-convolution graph kernels. Moreover, unlike the previous QJSD based graph kernels associated with the QJSD and the CTQW, the proposed HAQJSK kernels can simultaneously guarantee the properties of permutation invariant and positive definiteness, explaining the theoretical advantages of the HAQJSK kernels. The experiment indicates the effectiveness of the new proposed kernels.

Methods based on generative adversarial network (GAN) have been widely used in infrared and visible images fusion. However, these methods cannot perceive the discriminative parts of an image. Therefore, we introduce a multi-grained attention module into encoder-decoder network to fuse infrared and visible images (MgAN-Fuse). The infrared and visible images are encoded by two independent encoder networks due to their diverse modalities. Then, the results of the two encoders are concatenated to calculate the fused result by the decoder. To exploit the features of multi-scale layers fully and force the model focus on the discriminative regions, we integrate attention modules into multi-scale layers of the encoder to obtain multi-grained attention maps, and then the multi-grained attention maps are concatenated with the corresponding multi-scale features of decoder network. Thus, the proposed method can preserve the foreground target information of the infrared image, and capture the context information of the visible image. Furthermore, we design an additional feature loss in the training process to preserve the important features of the visible image, and a dual adversarial architecture is employed to help the model capture enough infrared intensity information and visible details simultaneously. The ablation studies illustrate the validity of multi-grained attention network and feature loss function. Extensive experiments on two infrared and visible image datasets demonstrate that the proposed MgAN-Fuse has better performance than state-of-the-art methods.

Infrared and visible image fusion (IVF) aims to generate a fused image with abundant texture details and salient thermal radiation targets, which can not only preserve the necessary scene information for traffic vision tasks, but also highlight the imperceptible targets that are crucial in intelligent transportation system (ITS). However, the existing image fusion methods often lack the information interactions between cross-modal features and among cross-granularity features, and they usually ignore the importance of edge information to the image, which affects the quality of the fused image. To this end, this study proposes an IVF network based on cross-modal multi-granularity information interaction and edge guidance, termed as CEFusion. On the one hand, a triple-branch scene fidelity module is designed to fuse the different modal features extracted by the encoder. This module can adequately mine difference information and infrared salient information of the cross-modal features through cross-modal information interaction. On the other hand, a progressive cross-granularity interaction feature enhancement module is employed to achieve the information interaction among cross-granularity features, which can further enrich the texture and structure information in the fused features. In addition, a novel edge loss function is proposed to guide the network to retain the edge information from source images. Extensive comparative and generalization experiments demonstrate that our CEFusion superior to the state-of-the-art methods in preserving texture details and thermal radiation targets. More importantly, the performance of our method in the high-level vision task suggests that it can provide reliable assistance for the ITS applications.

Deep learning approaches have been extensively applied to change detection in hyperspectral images (HSIs). However, the majority of them encounter scarcity of training samples or rely on complex structures and learning strategies. Although untrained change detection models have been proved to be effective in relief above problems, they were constructed using regular convolutions and treated spatial locations and channels equally, which are insufficient to extract discriminative features and lead to limited accuracy. Given this, a novel untrained framework using randomly initialized models with spatial-channel augmentation (RICD) is proposed for HSI change detection in this paper. It consists of two major modules: 1) an enhanced feature extraction network using successive dilation-deformable feature extraction blocks, which can extract multiscale spatial-spectral features over unfixed sampling locations. It enlarges the field of view of convolutions and takes arbitrary neighborhood into consideration, which helps to increase the discriminativeness of the extracted features; 2) a change sensitive feature augmentation and comparison module integrating feature selection and spatial-channel augmentation strategies, which can exploit spatial context and channel importance. It magnifies difference between changed pixels and unchanged ones and emphasizes contribution of significant channels of the selected change sensitive features. Despite that convolution operations are included in RICD, all the weights are untrained and fixed once they are randomly initialized, indicating that the RICD can work in an unsupervised manner. Its performance is tested over three widely used hyperspectral datasets. Quantitative and qualitative comparisons with several state-of-the-art unsupervised methods reveal the effectiveness of the RICD method.