A good result of infrared and visible image fusion should not only maintain significant contrast for distinguishing targets from the backgrounds, but also contain rich scene textures to cater for human visual perception. However, previous fusion methods usually do not fully utilize the information, and hence their fused results sacrifice either the salience of thermal targets or the sharpness of textures. To address this challenge, we propose a novel Generative Adversarial Network with Full-scale skip connection and dual Markovian discriminators (GAN-FM) to fully preserve effective information in infrared and visible images. First, a full-scale skip connected generator is designed to extract and fuse deep features of different scales, which can promote the direct transmission of shallow high-contrast features to the deep level, preserving the thermal radiation targets from the semantic level. As a result, the fused image can maintain significant contrast. Second, we propose two Markovian discriminators to establish adversarial games with the generator, so as to estimate probability distributions of infrared and visible modalities at the same time. Unlike conventional global discriminator, the Markovian discriminators try to distinguish each patch of input images, thus the attention of network is restricted to local regions and the fused results are forced to contain more details. In addition, we propose an effective joint gradient loss to ensure the harmonious coexistence of contrast and texture, which prevents the background texture pollution caused by the edge diffusion of the high-contrast target regions. Extensive qualitative and quantitative experiments demonstrate that our GAN-FM has advantages over the state-of-the-art methods in preserving significant contrast and rich textures. Moreover, we apply the fused image generated by our method to object detection and image segmentation, which can effectively improve the performance.

Pansharpening aims to fuse a multispectral (MS) image with low spatial resolution and a panchromatic (PAN) image with a high-spatial resolution to produce an image with both high spectral and high spatial resolution. In this study, we propose a variational pansharpening method by exploiting cartoon-texture similarities. After decomposition of the PAN image, the cartoon component always contains the global structure information, while the texture component includes the locally patterned information. This enables that the fused high-spatial resolution MS image can preserve the global and local spatial details (e.g., high-order information) well after leveraging the similarities of cartoon and texture components from PAN and MS images. To explore such cartoon-texture similarities, we describe cartoon similarity as gradient sparsity, formulated as a reweighted total variation term. Meanwhile, we use group low-rank constraint for texture similarity that is presented as repetitive texture patterns. By incorporating a data fidelity term for preserving the spectral information on the basis that the down-sampled fused MS image is consistent with the MS image, we further formulate pansharpening as an optimization problem and solve it efficiently using the alternative direction multiplier method. Extensive experiments have been conducted on a series of satellite data sets, and we also carry out a simulated vegetation coverage change experiment to verify the efficiency of the proposed method in remote sensing. The qualitative and quantitative results demonstrate that our method outperforms the state-of-the-art pansharpening methods in terms of both visual effect and objective metrics.

Robust small target detection is one of the key techniques in IR search and tracking systems for self-defense or attacks. In this paper we present a robust solution for small target detection in a single IR image. The key ideas of the proposed method are to use the directional support value of Gaussian transform (DSVoGT) to enhance the targets, and use the multiscale representation provided by DSVoGT to reduce the false alarm rate. The original image is decomposed into sub-bands in different orientations by convolving the image with the directional support value filters, which are deduced from the weighted mapped least-squares–support vector machines (LS–SVMs). Based on the sub-band images, a support value of Gaussian matrix is constructed, and the trace of this matrix is then defined as the target measure. The corresponding multiscale correlations of the target measures are computed for enhancing target signal while suppressing the background clutter. We demonstrate the advantages of the proposed method on real IR images and compare the results against those obtained from standard detection approaches, including the top-hat filter, max–mean filter, max–median filter, min–local–Laplacian of Gaussian (LoG) filter, as well as LS–SVM. The experimental results on various cluttered background images show that the proposed method outperforms other detectors.

In this paper, a novel self-supervised mask-optimization model, termed as SMFuse, is proposed for multi-focus image fusion. In our model, given two source images, a fully end-to-end Mask-Generator is trained to directly generate the binary mask without requiring any patch operation or postprocessing through self-supervised learning. On the one hand, based on the principle of repeated blur, we design a Guided-Block with guided filter to obtain an initial binary mask from source images, narrowing the solution domain and speeding up the convergence of the binary mask generation, which is constrained by a map loss. On the other hand, as the focused regions in source images show richer texture details than the defocused ones, i.e., larger gradients, we also design a max-gradient loss between the fused image and source images as a follow-up optimization operation to ensure the fused image to be all-in-focus, forcing our model to learn a more accurate binary mask. Extensive experimental results conducted on two publicly available datasets substantiate the effectiveness and superiority of our SMFuse compared with the current state-of-the-art. Our code is publicly available online. ^{1 1} [Online]. Available: https://github.com/jiayi-ma/SMFuse.