Multi-modal fake news detection aims to identify fake information through text and corresponding images. The current methods purely combine images and text scenarios by a vanilla attention module but there exists a semantic gap between different scenarios. To address this issue, we introduce an image caption-based method to enhance the model’s ability to capture semantic information from images. Formally, we integrate image description information into the text to bridge the semantic gap between text and images. Moreover, to optimize image utilization and enhance the semantic interaction between images and text, we combine global and object features from the images for the final representation. Finally, we leverage a transformer to fuse the above multi-modal content. We carried out extensive experiments on two publicly available datasets, and the results show that our proposed method significantly improves performance compared to other existing methods.
8 November 2023 in IEEE Transactions on Neural Networks and Learning Systems
Infrared and visible image fusion (IVIF) aims to obtain an image that contains complementary information about the source images. However, it is challenging to define complementary information between source images in the lack of ground truth and without borrowing prior knowledge. Therefore, we propose a semisupervised transfer learning-based method for IVIF, termed STFuse, which aims to transfer knowledge from an informative source domain to a target domain, thus breaking the above limitations. The critical aspect of our method is to borrow supervised knowledge from the multifocus image fusion (MFIF) task and to filter out task-specific attribute knowledge by using a guidance loss $L_g$ , which motivates its cross-task use in IVIF tasks. Using this cross-task knowledge effectively alleviates the limitation of the lack of ground truth on fusion performance, and the complementary expression ability under the constraint of supervised knowledge is more instructive than prior knowledge. Moreover, we designed a cross-feature enhancement module (CEM) that utilizes self-attention and mutual-attention features to guide each branch to refine features and then facilitate the integration of cross-modal complementary features. Extensive experiments demonstrate that our method has good advantages in terms of visual quality and statistical metrics, as well as the docking of high-level vision tasks, compared with other state-of-the-art methods.
18 September 2024 in IEEE Journal of Selected Topics in Signal Processing
Infrared and visible image fusion (IVIF) aims to integrate complementary information between sensors and generate information-rich high-quality images. However, current methods mainly concentrate on the fusion of the source features from the sensors, ignoring the feature information mismatch caused by the property of the sensors, which results in redundant or even invalid information. To tackle the above challenges, this paper developed an end-to-end model based on the Retinex Decomposition Model (RDM), called RDMFuse, which utilizes a hierarchical feature process to alleviate the fusion performance degradation caused by the feature-level mismatch. Specifically, as infrared images only provide an overview of the intrinsic properties of the scene, we first use RDM to decouple visible images into a reflectance component containing intrinsic properties and an illumination component containing illumination information. Then, the contrast texture module (CTM) and the intrinsic fusion function are designed for the property of the intrinsic feature, which complements each other to aggregate the intrinsic information of the source images at a smaller cost and brings the fused image more comprehensive scene information. Besides, the illumination-adaptive module implements illumination component optimization in a self-supervised way to make the fused image with an appropriate intensity distribution. It is worth noting that this mechanism implicitly improves the entropy quality of the image to improve the image degradation problem caused by environmental factors, especially in the case of a dark environment. Numerous experiments have demonstrated the effectiveness and robustness of the RDMFuse and the superiority of generalization in high-level vision tasks due to the improved discriminability of the fused image to the captured scene. The code is available at https://github.com/wang-x-1997/RDMFuse.
This study proposes an infrared and visible image fusion method based on contrast saliency information guided, termed CSFusion. On the one hand, the gradual recovery network is devised to achieve self-refining processing and cross-stage integration of encoded features. Specifically, the network establishes one-to-one feature reuse between the encoder and the decoder, which can effectively facilitating the decoder integration and reconstruction of encoded features. Moreover, the introduction of the self-refine attention module (SRAM) effectively refines the encoded information of each branch and reduces the impact of redundant information on image reconstruction. On the other hand, the salient guided module (SGM) built with the edge-preserving filter can effectively output contrast saliency maps of the source images. Its synergistic operation with the objective function drives the fused image generated by the network to exhibit a better background texture while highlighting the content information of the source images. It is worth noting that the SGM as the auxiliary network is utilized only in the training stage. Compared to state-of-the-art methods, extensive experiments of qualitative and quantitative results prove the superiority and robustness of our method. Also, the performance on the object detection task further reveals its potential on high-level vision tasks.