10 October 2024 in IEEE Transactions on Knowledge and Data Engineering
With the rapid expansion of multimodal content in online social media, automatic detection of multimodal fake news has received much attention. Multimodal joint training commonly used in existing methods is expected to benefit from thoroughly leveraging cross-modal features, yet these methods still suffer from insufficient learning of uni-modal features. Due to the heterogeneity of multimodal networks, optimizing a single objective will inevitably make the models prone to rely on specific modality while leaving other modalities under-optimized. On the other hand, simply expecting each modality to play a significant role in identifying all the rumors is also not appropriate as the multimodal fake news often involves tampering in only one modality. Therefore, how to find the genuine tampering on the per-sample basis becomes the key point to unlock the full power of each modality in a good collaborative manner. To address these issues, we propose a Uni-modal Event-agnostic Knowledge Distillation framework framework (UEKD), which aims to transfer knowledge contained in the fine-grained prediction from uni-modal teachers to the multimodal student model through modality-specific distillation. Specifically, we find that the uni-modal teachers simply trained on the whole training set are easy to memorize the event-specific noise information to make a correct but biased prediction, failing to reflect the genuine degree of tampering in each modality. To tackle this problem, we propose to train and validate the teacher models on different domains in training dataset through a cross-validation manner, as the predictions from the out-of-domain teachers can be regarded as event-agnostic knowledge without spurious connections with event-specific information. Finally, to balance the convergence speeds across modalities, we dynamically monitor the involvement of each modality during training, through which we could identify the more under-optimized modalities and re-weight the distillation loss accordingly. Our method could be served as a plug-and-play module for existing multimodal fake news detection backbones. Extensive experiments on three public datasets and four state-of-the-art fake news detection backbones show that our proposed method can improve the performance by a large margin.
1 January 2026 in IEEE Transactions on Multimedia
Existing multimodal fake news detection methods based on traditional small models are prone to learn superficial features while struggling to perform deep reasoning and truly perceive fine-grained image-text consistency. Recently, fueled by Large Language Models and large-scale multimodal pretraining data, Large Vision-Language Models (LVLMs) have seen significant progress in these aspects. In our pilot study, domain-specific fine-tuned LVLM (FT-LVLM) Qwen2-VL-2B has achieved superior performance for multimodal fake news detection. However, we still find two weaknesses within it: (1) The model tends to rely only on textual modality while learning little from visual modality. (2) Directly outputting predictions hinders the model from exploiting the powerful reasoning capability of the LVLM. To address the above issues, we further propose a novel framework of Mixture-of-Experts Multi-View Large Visual-Language Models (MMVM) to explicitly disentangle multi-view features within the LVLM and adaptively fuse them for fake news detection. Specifically, we first extract representations of three distinct views, i.e., main branch, visual branch and reasoning branch, from the LVLM respectively. Then, considering that a given piece of fake news may not contain clues of equal importance in all the views, View-Aware Mixture-of-Experts networks (VAMoE) are proposed to dynamically reweight single-view representations and route the most informative features for the final prediction. Extensive experiments on four public datasets demonstrate the significant effectiveness and generalization ability of our proposed framework.