18 July 2022 in IEEE Transactions on Instrumentation and Measurement
The existing deep learning fusion methods mainly concentrate on convolutional neural networks (CNNs), and few attempts are made with transformer. Meanwhile, the convolutional operation is a content-independent interaction between the image and the convolution kernel, which may lose some important contexts and further limit fusion performance. Toward this end, we present a simple and strong fusion baseline for infrared and visible images, namely, residual Swin Transformer fusion network, termed SwinFuse. Our SwinFuse includes three parts: the global feature extraction, fusion layer, and feature reconstruction. In particular, we build a fully attentional feature encoding backbone to model the long-range dependencies, which is a pure transformer network and has a stronger representation ability compared with the CNNs. Moreover, we design a novel feature fusion strategy based on the $L_{1}$ -norm for sequence matrices and measure the corresponding activity levels from row and column vector dimensions, which can well retain competitive infrared brightness and distinct visible details. Finally, we testify our SwinFuse with nine state-of-the-art traditional and deep learning methods on three different datasets through subjective observations and objective comparisons, and the experimental results manifest that the proposed SwinFuse obtains surprising fusion performance with strong generalization ability and competitive computational efficiency. The code will be available at https://github.com/Zhishe-Wang/SwinFuse .
3 September 2021 in IEEE Transactions on Circuits and Systems for Video Technology
Infrared image retains typical thermal targets while visible image preserves rich texture details, image fusion aims to reconstruct a synthesized image containing prominent targets and abundant texture details. Most of deep learning-based methods mainly focus on convolution operation to extract the local features, but do not fully consider their multi-scale characteristics and global dependencies, which may cause loss of target regions and texture details in the fused image. Towards this goal, we present a unified multi-scale densely connected fusion network in this paper, named as UNFusion. We carefully design a multi-scale encoder-decoder architecture that can efficiently extract and reconstruct multi-scale deep features. Dense skip connections are employed in both encoder and decoder sub-networks to reuse all the intermediate features of different layers and scales for fusion tasks. In the fusion layer, $L_{p} $ normalized attention models, which include three kinds of different norms, are proposed to highlight and combine these deep features from spatial and channel dimensions, and the combined spatial and channel attention maps are used to reconstruct a final fused image. We conduct extensive experiments on the public TNO and Roadscene datasets, and the results demonstrate that our UNFusion can simultaneously preserve high brightness of typical thermal targets and abundant texture details to obtain superior scene representation and better visual perception. Besides, our UNFusion achieves better fusion performance and transcends other state-of-the-art methods in terms of qualitative and quantitative comparisons. Our code is available at https://github.com/Zhishe-Wang/UNFusion .
13 September 2024 in IEEE Transactions on Circuits and Systems for Video Technology
Traditional fusion methods based on deep learning mainly employ convolutional or self-attention operations to model local or global dependencies, which often lead to the oversight of frequency-domain information. To address this deficiency, we introduce a unified frequency adversarial learning network, termed FreqGAN. Our method involves a frequency-compensated generator that employs discrete wavelet transformation to decompose encoded spatial features into multiple frequency bands. Leveraging skip connections, low and high-frequency components are respectively directed into the encoder and decoder, compensating for additional outline and detail. Moreover, we construct a hybrid frequency aggregation module, which enables a progressive optimization of activity levels across multiple scales and makes the various frequency bands correlated. Complementing our generative model, we devise dual frequency-constrained discriminators. These discriminators are tasked with dynamically adjusting weights for each input frequency band, thereby obligating the generator to accurately reconstruct salient frequency information from different modality images. Additionally, a frequency-supervised function is formulated to further safeguard against the loss of frequency information. Our comprehensive experimental evaluations, encompassing a wide range of fusion tasks and subsequent applications, distinctly highlight FreqGAN’s superior performance, establishing it as a frontrunner in comparison to existing state-of-the-art alternatives. The source codes are forthcoming at: https://github.com/Zhishe-Wang/FreqGAN.
25 January 2023 in IEEE Transactions on Circuits and Systems for Video Technology
Recent existing methods generally adopt a simple concatenation or addition strategy to integrate features at the fusion layer, failing to adequately consider the intrinsic characteristics of different modal images and feature interaction of different scales, which may produce a limited fusion performance. Toward this end, we introduce a cross-scale iterative attentional adversarial fusion network, namely CrossFuse. More specifically, in the generator, we design a cross-modal attention integrated module to merge the intrinsic content of different modal images. The parallel spatial-independent and channel-independent pathways are proposed to calculate the attentional weights, which are assigned to measure the activity levels of source images at the same scale. Moreover, we construct a cross-scale iterative decoder framework to interact with different modality features at different scales, which can constantly optimize their activity levels. By this means, the generator learns to integrate their modality characteristics via attentional weights in an iterative manner, and the generated result characterizes competitive infrared radiant intensity and distinct visible detail description. Extensive experiments on three different benchmarks demonstrate that our CrossFuse outperforms other nine state-of-the-art methods in terms of fusion performance, generalization ability and computational efficiency. Our codes will be released at https://github.com/Zhishe-Wang/CrossFuse.
5 January 2022 in IEEE Transactions on Instrumentation and Measurement
Infrared and visible image fusion intends to generate a synthetic image with superior scene representation and better visual perception. The existing deep learning-based fusion methods merely make use of the convolution operation to extract features with a local receptive field, without fully considering their multiscale and long-range dependency characteristics, which may fail to preserve some essential global context from source images. To this end, we develop a novel and efficient fusion network based on dense Res2net and double nonlocal attention models, termed Res2Fusion. We introduce Res2net and dense connections into the encoder network with multiple available receptive fields, which is used to extract the multiscale features, and can retain as much meaningful information as possible for fusion tasks. In addition, we develop double nonlocal attention models as a fusion layer to model long-range dependencies on the local features. Specifically, these attention models can refine feature maps obtained by the encoder network to more focus on prominent infrared targets and distinct visible details. Finally, the comprehensive attention maps are used to generate a fused result through the simple decoder network. Extensive experiments demonstrate that the proposed method can simultaneously retain highlighted infrared targets and rich visible details and transcends other state-of-the-art fusion methods in terms of subjective and objective evaluation. The corresponding code is publicly available at https://github.com/Zhishe-Wang/Res2Fusion .
12 December 2022 in IEEE Transactions on Multimedia
The existing generative adversarial fusion methods generally concatenate source images or deep features, and extract local features through convolutional operations without considering their global characteristics, which tends to produce a limited fusion performance. Toward this end, we propose a novel interactive compensatory attention fusion network, termed ICAFusion. In particular, in the generator, we construct a multi-level encoder-decoder network with a triple path, and design infrared and visible paths to provide additional intensity and gradient information for the concatenating path. Moreover, we develop the interactive and compensatory attention modules to communicate their pathwise information, and model their long-range dependencies through a cascading channel-spatial model. The generated attention maps can more focus on infrared target perception and visible detail characterization, and are used to reconstruct the fusion image. Therefore, the generator takes full advantage of local and global features to further increase the representation ability of feature extraction and feature reconstruction. Extensive experiments illustrate that our ICAFusion obtains superior fusion performance and better generalization ability, which precedes other advanced methods in the subjective visual description and objective metric evaluation. Our codes will be public at https://github.com/Zhishe-Wang/ICAFusion .