Multi-modal fake news detection aims to identify fake information through text and corresponding images. The current methods purely combine images and text scenarios by a vanilla attention module but there exists a semantic gap between different scenarios. To address this issue, we introduce an image caption-based method to enhance the model’s ability to capture semantic information from images. Formally, we integrate image description information into the text to bridge the semantic gap between text and images. Moreover, to optimize image utilization and enhance the semantic interaction between images and text, we combine global and object features from the images for the final representation. Finally, we leverage a transformer to fuse the above multi-modal content. We carried out extensive experiments on two publicly available datasets, and the results show that our proposed method significantly improves performance compared to other existing methods.
31 December 2024 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Deep learning (DL)-based pan-sharpening methods have become mainstream due to their exceptional performance. However, the lack of ground truth for supervised learning forces most DL-based methods to use pseudo-ground truth multispectral images, limiting learning potential and constraining the model's solution space. Unsupervised methods often neglect mutual learning across modalities, leading to insufficient spatial details in pansharpened images. To address these issues, this study proposes a fusion-decomposition pan-sharpening model based on interactive learning of representation graphs. This model considers both the compression process from source images to fused results and the decomposition process back to the source images. It aims to leveraging feature consistency between these processes to enhance the spatial and spectral consistency learned by the fusion network in a data-driven manner. Specifically, the fusion network incorporates the meticulously designed Representational Graph Interaction Module and the Graph Interaction Fusion Module. These modules construct a representational graph structure for crossmodal feature communication, generating a global representation that guides the cross-modal semantic aggregation of multispectral and panchromatic data. In the decomposition network, the Spatial Structure Perception Module and the Spectral Feature Extraction Module, designed based on the attributes of the source image features, enable the network to better perceive and reconstruct multispectral and panchromatic data from the fused result. This, in turn, enhances the fusion network's perception of spectral information and spatial structure. Qualitative and quantitative results on the IKONOS, GaoFen-2, WorldView-2 and WorldView-3 datasets validate the effectiveness of the proposed method in comparison to other SOTA methods.
1 August 2024 in IEEE Transactions on Computational Social Systems
Multimodal sentiment analysis refers to predicting the emotions conveyed by the joint input of multiple modalities. The primary challenge of this task is to effectively integrate emotional features across different modalities to improve the precision of emotion recognition. For visual–textual sentiment analysis, existing methods typically employ two encoders to extract text and image features separately and enhance analysis performance through cross-modal fusion. However, previous methods often overlook the fact that due to the significant noise in social media data, the correlation between text and images may be weak, and forced alignment could lead to semantic distortion. To address the problem, we propose a context-based alignment method that employs a global modality-aware attention mechanism. This mechanism allows each image or text to reference relevant information from other samples in the entire training set during the alignment process. In addition, to further enhance global alignment, we integrate self-supervised contrastive learning to refine feature representations. By leveraging global semantic information, our approach enhances the robustness of local alignment, effectively reducing alignment bias caused by data noise and improving multimodal fusion performance. Experimental results demonstrate that the proposed approach better understands and integrates information from different modalities in multimodal sentiment analysis, leading to improved accuracy and robustness in multimodal sentiment analysis.