In this paper, we propose a novel end-to-end model for fusing medical images characterizing structural information, i.e., IS, and images characterizing functional information, i.e., IF, of different resolutions, by using a multi-generator multi-discriminator conditional generative adversarial network (MGMDcGAN). In the first cGAN, the generator aims to generate a real-like fused image based on a specifically designed content loss to fool two discriminators, while the discriminators aim to distinguish the structure differences between the fused image and source images. On this basis, we employ the second cGAN with a mask to enhance the information of dense structure in the final fused image, while preventing the functional information from being weakened. Consequently, the final fused image is forced to concurrently keep the structural information in IS and the functional information in IF. In addition, as a unified method, MGMDcGAN can be applied to different kinds of medical image fusion, i.e., MRI-PET, MRI-SPECT, and CT-SPECT, where MRI and CT are two kinds of IS of high resolution, PET and SPECT are typical kinds of IF of low resolution. Qualitative and quantitative experiments on publicly available datasets demonstrate the superiority of our MGMDcGAN over the state-of-the-art.

In this paper, a novel self-supervised mask-optimization model, termed as SMFuse, is proposed for multi-focus image fusion. In our model, given two source images, a fully end-to-end Mask-Generator is trained to directly generate the binary mask without requiring any patch operation or postprocessing through self-supervised learning. On the one hand, based on the principle of repeated blur, we design a Guided-Block with guided filter to obtain an initial binary mask from source images, narrowing the solution domain and speeding up the convergence of the binary mask generation, which is constrained by a map loss. On the other hand, as the focused regions in source images show richer texture details than the defocused ones, i.e., larger gradients, we also design a max-gradient loss between the fused image and source images as a follow-up optimization operation to ensure the fused image to be all-in-focus, forcing our model to learn a more accurate binary mask. Extensive experimental results conducted on two publicly available datasets substantiate the effectiveness and superiority of our SMFuse compared with the current state-of-the-art. Our code is publicly available online. ^{1 1} [Online]. Available: https://github.com/jiayi-ma/SMFuse.

In this paper, we propose a novel unsupervised model for multi-focus image fusion based on gradients and connected regions, termed as GCF. To overcome the stumbling block of vanishing gradients in applying deep networks for multi-focus image fusion, we design the Mask-Net which can directly generate a binary mask. Thus, there is no need for hand-crafted feature extraction or fusion rules. Based on the fact that objects within the depth-of-field (DOF) have shaper appearance, i.e., larger gradients, we use the gradient relation map obtained from source images to narrow the solution domain and speed up convergence. Then, the constraint of connected region numbers is conductive to finding the more accurate binary mask. With the consistency verification strategy, the final mask can be obtained by adapting the initial binary mask to generate the fused result. Therefore, the proposed method is an unsupervised model without the need of the ground-truth data. Both qualitative and quantitative experiments are conducted on the publicly available Lytro dataset. The results show that GCF can outperform the state-of-the-art in both visual perception and objective metrics.

In this paper, we propose the medical Wasserstein generative adversarial networks (MWGAN), an end-to-end model, for fusing magnetic resonance imaging (MRI) and positron emission tomography (PET) medical images. Our method establishes two adversarial games between a generator and two discriminators to generate a fused image with the details of soft tissue structures in organs from MRI images and the functional and metabolic information from PET images. Different information from source images can be effectively adjusted with a specifically designed loss function. In addition, we use WGAN instead of the traditional generative adversarial networks to make the training process more stable and allow our architecture to deal with source images of different resolutions. Qualitative and quantitative comparisons on publicly available datasets demonstrate the superiority of MWGAN over the state-of-the-art networks. Furthermore, our MWGAN is applied to the fusion of MRI and computed tomography images of different resolutions, achieving a satisfactory performance.