We study the problem of multi-focus image fusion, where the key challenge is detecting the focused regions accurately among multiple partially focused source images. Inspired by the conditional generative adversarial network (cGAN) to image-to-image task, we propose a novel FuseGAN to fulfill the images-to-image for multi-focus image fusion. To satisfy the requirement of dual input-to-one output, the encoder of the generator in FuseGAN is designed as Siamese network. The least square GAN objective is employed to enhance the training stability of FuseGAN, resulted in an accurate confidence map for focus region detection. Also, we exploit the convolutional conditional random fields (ConvCRFs) technique on the confidence map to reach a refined final decision map for better focus region detection. Moreover, due to lacking of large-scale standard dataset, we synthesize a large enough multi-focus image dataset based on a public natural image dataset PASCAL VOC 2012, where we utilize a normalized disk point spread function (PSF) to simulate the defocus and separate the background and foreground in the synthesis for each image. We conduct extensive experiments on two public datasets to verify the effectiveness of the proposed method. Results demonstrate that the proposed method presents accurate decision maps for focus regions in multi-focus images, such that the fused images are superior to eleven recently the start-of-the-art algorithms, not only in visual perception, but also in quantitative analysis in terms of five metrics.