Visible images provide abundant texture details and environmental information, while infrared images benefit from night-time visibility and suppression of highly dynamic regions; it is a meaningful task to fuse these two types of features from different sensors to generate an informative image. In this paper, we propose an unsupervised end-to-end learning framework for infrared and visible image fusion. We first construct enough benchmark training datasets using the visible and infrared frames, which can address the limitation of the training dataset. Additionally, due to the lack of labeled datasets, our architecture is derived from a robust mixed loss function that consists of the modified structural similarity (M-SSIM) metric and the total variation (TV) by designing an unsupervised learning process that can adaptively fuse thermal radiation and texture details and suppress noise interference. In addition, our method is an end to end model, which avoids setting hand-crafted fusion rules and reducing computational cost. Furthermore, extensive experimental results demonstrate that the proposed architecture performs better than state-of-the-art methods in both subjective and objective evaluations.

We study the problem of multi-focus image fusion, where the key challenge is detecting the focused regions accurately among multiple partially focused source images. Inspired by the conditional generative adversarial network (cGAN) to image-to-image task, we propose a novel FuseGAN to fulfill the images-to-image for multi-focus image fusion. To satisfy the requirement of dual input-to-one output, the encoder of the generator in FuseGAN is designed as Siamese network. The least square GAN objective is employed to enhance the training stability of FuseGAN, resulted in an accurate confidence map for focus region detection. Also, we exploit the convolutional conditional random fields (ConvCRFs) technique on the confidence map to reach a refined final decision map for better focus region detection. Moreover, due to lacking of large-scale standard dataset, we synthesize a large enough multi-focus image dataset based on a public natural image dataset PASCAL VOC 2012, where we utilize a normalized disk point spread function (PSF) to simulate the defocus and separate the background and foreground in the synthesis for each image. We conduct extensive experiments on two public datasets to verify the effectiveness of the proposed method. Results demonstrate that the proposed method presents accurate decision maps for focus regions in multi-focus images, such that the fused images are superior to eleven recently the start-of-the-art algorithms, not only in visual perception, but also in quantitative analysis in terms of five metrics.

Images captured in low-light environments suffer from serious degradation due to insufficient light, leading to the performance decline of industrial and civilian devices. To address the problems of noise, chromatic aberration, and detail distortion for enhancing low-light images using existing enhancement methods, this paper proposes an integrated learning approach (LightingNet) for low-light image enhancement. The LightingNet consists of two core components: 1) the complementary learning sub-network and 2) the vision transformer (VIT) low-light enhancement sub-network. VIT low-light enhancement sub-network is designed to learn and fit the current data to provide local high-level features through a full-scale architecture, and the complementary learning sub-network is utilized to provide global fine-tuned features through learning transfer. Extensive experiments confirm the effectiveness of the proposed LightingNet.

Images and videos shot in low light are often accompanied by severe image degradation, such as color noise, chromatic aberrations and loss of details. Most existing convolutional neural network (CNN)-based low-light enhancement methods focus on decomposing the image into illumination and reflection parts via the Retinex model, but these methods often fail to adequately consider controlling noise during enhancement and perform poorly in the face of complex lighting environments. In this letter, we propose a powerful Vision Transformer-based Generative Adversarial Network (Transformer-GAN) for enhancing low-light images. Transformer-GAN consists of two subnets as follows: (1) the feature extraction is achieved by an iterative multi-branch network in the feature extraction subnet, and (2) the enhancement is completed in the image reconstruction subnet. The innovative core works are multi-head multi-covariance self-attention (MHMCA) and Light feature-forward module structures (LFFM) in Transformer-GAN. Experiments demonstrate that our method outperforms state-of-the-art low-light enhancement methods on popular low-light datasets.