Medical image fusion technology is an essential component of computer-aided diagnosis, which aims to extract useful cross-modality cues from raw signals to generate high-quality fused images. Many advanced methods focus on designing fusion rules, but there is still room for improvement in cross-modal information extraction. To this end, we propose a novel encoder-decoder architecture with three technical novelties. First, we divide the medical images into two attributes, namely pixel intensity distribution attributes and texture attributes, and thus design two self-reconstruction tasks to mine as many specific features as possible. Second, we propose a hybrid network combining a CNN and a transformer module to model both long-range and short-range dependencies. Moreover, we construct a self-adaptive weight fusion rule that automatically measures salient features. Extensive experiments on a public medical image dataset and other multimodal datasets show that the proposed method achieves satisfactory performance.

Traditional and deep learning-based fusion methods generated the intermediate decision map to obtain the fusion image through a series of postprocessing procedures. However, the fusion results generated by these methods are easy to lose some source image details or results in artifacts. Inspired by the image reconstruction techniques based on deep learning, we propose a multifocus image fusion network framework without any postprocessing to solve these problems in the end-to-end and supervised learning ways. To sufficiently train the fusion model, we have generated a large-scale multifocus image data set with ground-truth fusion images. What is more, to obtain a more informative fusion image, we further designed a novel fusion strategy based on unity fusion attention, which is composed of a channel attention module and a spatial attention module. Specifically, the proposed fusion approach mainly comprises three key components: feature extraction, feature fusion, and image reconstruction. We first utilize seven convolutional blocks to extract the image features from source images. Then, the extracted convolutional features are fused by the proposed fusion strategy in the feature fusion layer. Finally, the fused image features are reconstructed by four convolutional blocks. Experimental results demonstrate that the proposed approach for multifocus image fusion achieves remarkable fusion performance and superior time efficiency compared to 19 state-of-the-art fusion methods.

Due to the limitations of equipment and technology, few images captured by a single sensor can describe all the information in a scene. Multi-focus image fusion techniques provide an effective means of integrating images acquired from different sensors to produce a single fused image that is richer in scene information. To obtain better fusion results, we propose a novel deep learning structure for multi-focus image fusion to achieve end-to-end network structure without post-processing procedures. Specifically, an innovative Siamese multi-scale feature extraction module was introduced to extract multi-scale image features. Besides, aiming to enhance the quality of the fused image, we further designed a new adaptive fusion strategy by using the attention mechanism instead of following the previous manual design. Extensive experimental results confirm that the proposed algorithm is more effective, which surpasses the existing state-of-the-art multi-focus image fusion methods both quantitatively and qualitatively. Furthermore, additional expanded experiment conducted on the infrared and visible image dataset demonstrates that the proposed fusion framework is also capable of performing other fusion tasks with excellent fusion performance. The code of the proposed fusion method is available at https://github.com/govenda/ MSPA-Fuse.

Traditional and deep learning-based fusion methods generated the intermediate decision map to obtain the fusion image through a series of post-processing procedures. However, the fusion results generated by these methods are easy to lose some source image details or results in artifacts. Inspired by the image reconstruction techniques based on deep learning, we propose a multi-focus image fusion network framework without any post-processing to solve these problems in the end-to-end and supervised learning way. To sufficiently train the fusion model, we have generated a large-scale multi-focus image dataset with ground-truth fusion images. What's more, to obtain a more informative fusion image, we further designed a novel fusion strategy based on unity fusion attention, which is composed of a channel attention module and a spatial attention module. Specifically, the proposed fusion approach mainly comprises three key components: feature extraction, feature fusion and image reconstruction. We firstly utilize seven convolutional blocks to extract the image features from source images. Then, the extracted convolutional features are fused by the proposed fusion strategy in the feature fusion layer. Finally, the fused image features are reconstructed by four convolutional blocks. Experimental results demonstrate that the proposed approach for multi-focus image fusion achieves remarkable fusion performance compared to 19 state-of-the-art fusion methods.