Visible images provide abundant texture details and environmental information, while infrared images benefit from night-time visibility and suppression of highly dynamic regions; it is a meaningful task to fuse these two types of features from different sensors to generate an informative image. In this paper, we propose an unsupervised end-to-end learning framework for infrared and visible image fusion. We first construct enough benchmark training datasets using the visible and infrared frames, which can address the limitation of the training dataset. Additionally, due to the lack of labeled datasets, our architecture is derived from a robust mixed loss function that consists of the modified structural similarity (M-SSIM) metric and the total variation (TV) by designing an unsupervised learning process that can adaptively fuse thermal radiation and texture details and suppress noise interference. In addition, our method is an end to end model, which avoids setting hand-crafted fusion rules and reducing computational cost. Furthermore, extensive experimental results demonstrate that the proposed architecture performs better than state-of-the-art methods in both subjective and objective evaluations.

We study the problem of multi-focus image fusion, where the key challenge is detecting the focused regions accurately among multiple partially focused source images. Inspired by the conditional generative adversarial network (cGAN) to image-to-image task, we propose a novel FuseGAN to fulfill the images-to-image for multi-focus image fusion. To satisfy the requirement of dual input-to-one output, the encoder of the generator in FuseGAN is designed as Siamese network. The least square GAN objective is employed to enhance the training stability of FuseGAN, resulted in an accurate confidence map for focus region detection. Also, we exploit the convolutional conditional random fields (ConvCRFs) technique on the confidence map to reach a refined final decision map for better focus region detection. Moreover, due to lacking of large-scale standard dataset, we synthesize a large enough multi-focus image dataset based on a public natural image dataset PASCAL VOC 2012, where we utilize a normalized disk point spread function (PSF) to simulate the defocus and separate the background and foreground in the synthesis for each image. We conduct extensive experiments on two public datasets to verify the effectiveness of the proposed method. Results demonstrate that the proposed method presents accurate decision maps for focus regions in multi-focus images, such that the fused images are superior to eleven recently the start-of-the-art algorithms, not only in visual perception, but also in quantitative analysis in terms of five metrics.

As the optical lenses for cameras always have limited depth of field, the captured images with the same scene are not all in focus. Multifocus image fusion is an efficient technology that can synthesize an all-in-focus image using several partially focused images. Previous methods have accomplished the fusion task in spatial or transform domains. However, fusion rules are always a problem in most methods. In this letter, from the aspect of focus region detection, we propose a novel multifocus image fusion method based on a fully convolutional network (FCN) learned from synthesized multifocus images. The primary novelty of this method is that the pixel-wise focus regions are detected through a learning FCN, and the entire image, not just the image patches, are exploited to train the FCN. First, we synthesize 4500 pairs of multifocus images by repeatedly using a gaussian filter for each image from PASCAL VOC 2012, to train the FCN. After that, a pair of source images is fed into the trained FCN, and two score maps indicating the focus property are generated. Next, an inversed score map is averaged with another score map to produce an aggregative score map, which take full advantage of focus probabilities in two score maps. We implement the fully connected conditional random field (CRF) on the aggregative score map to accomplish and refine a binary decision map for the fusion task. Finally, we exploit the weighted strategy based on the refined decision map to produce the fused image. To demonstrate the performance of the proposed method, we compare its fused results with several start-of-the-art methods not only on a gray data set but also on a color data set. Experimental results show that the proposed method can achieve superior fusion performance in both human visual quality and objective assessment.

Multimodal medical image fusion (MMIF) is to merge multiple images for better imaging quality with preserving different specific features, which could be more informative for efficient clinical diagnosis. In this paper, a novel fusion framework is proposed for multimodal medical images based on multi-source information exchange encoding (MIEE) by using Pulse Coupled Neural Network (PCNN). We construct an MIEE model by using two types of PCNN, such that the information of an image can be exchanged and encoded to another image. Then the fusion contributions for each pixel are estimated qualitatively according to a logical comparison of exchanged information. Further， the exchanged information is nonlinearly transformed using an exponential function with a functional parameter. Finally, quantitative fusion contributions are produced through a reverse-proportional operator to the exchanged information. Also, particle swarm optimization-based derivative-free optimization and a total vibration-based derivative optimization are used to optimize the PCNN and functional transform parameters, respectively. Experiments demonstrate that our method gives the best results than other state-of-the-art fusion approaches.

RGBT tracking has received extensive attention as a research hotspot, benefiting from the complementarity of visible (RGB) and thermal (T) modalities, which can better handle tracking tasks in harsh environments. Existing studies typically deploy the same structure to extract features of heterogeneous modalities with different properties, which may be difficult to simultaneously search for mining strategies suitable for each modality, resulting in insufficient feature representation. And, with the success of Transformer, how to jointly consider the heterogeneity and global mining to facilitate better integration of heterogeneous modalities is also a crucial issue worth exploring. To solve these problems, we propose a unique differential reinforcement and global collaboration network for RGBT tracking, which includes a parallel backbone, modal differential reinforcement (MDR), and global mining collaboration module (GMCM). Specifically, MDR introduces raw data to control adaptive differential reinforcement for both modalities via a modality-sharing augmentation unit and a modality-aware gating unit. The data-driven design makes it easier for the two modalities to search for the respective appropriate mining strategies, further effectively enhancing the respective feature representations. GMCM first mines intra-modal and cross-modal global information separately, and then performs an adaptive ensemble operation to balance the two. This operation learns intra-modal and cross-modal global weights in RGB and T modalities to better integrate heterogeneous modalities based on heterogeneity and global cues. Extensive experiments on three datasets demonstrate that the performance of the proposed network is comparable to state-of-the-art RGBT trackers.

A single infrared image or visible image for the same scene is usually insufficient to simultaneously reveal the infrared objects and the scene details. Thus, image fusion techniques play an important role in producing a single image from the images captured by infrared and visible sensors. In this paper, we propose a novel total variation (TV)-based fusion for infrared and visible images. In our model, a weighted fidelity term is employed to fuse both the infrared objects in the infrared image and the salient scenes in the visible image. To this end, a weight estimation method is developed based on the global luminance contrast-based saliency. Also, to overcome the over-fitting, two constraints are further introduced to merge more details from the visible image and prevent the luminance degradation for the fused result, respectively. Moreover, joint norms are exploited to produce a better result. ${{\boldsymbol{l}}_{2,1,{\boldsymbol{rc}}}}$ provides the structural group sparseness for the fidelity term, whereas ${{\boldsymbol{l}}_{1/2}}$ presents the better gradient sparse for the detail preserving term and ${{\boldsymbol{l}}_2}$ is utilized for the luminance degradation preventing term. Experimental results indicate that the proposed method can give state-of-the-art performances both in visual perception and quantitative scores than other methods.