4 October 2025 in Information Processing & Management
21 April 2026 in SSRN Electronic Journal
Out-Of-Context (OOC) misinformation detection aims to identify whether a genuine image is repurposed in a misleading textual content. Such misinformation arises when genuine images are detached from their original contexts and paired with semantically related but false texts, making detection based solely on image-text pairs challenging. Existing methods primarily exploit internal knowledge from image-text pairs or external knowledge from retrieved evidence, yet they often fail to establish deep alignment between these sources due to limited interaction. To address these limitations, we propose SDLE, a novel Shallow-Deep Learning framework that enhances Multimodal Large Language Models (MLLMs) for OOC misinformation detection. Specifically, SDLE first retrieves external evidence from various sources to gather contextual information related to image-text pairs. To build a shallow relationship between internal and external knowledge, SDLE designs a prompt learning method to reduce noise and capture key evidence. Subsequently, to build a deep relationship between them, SDLE aligns MLLMs for evidence-based rationale generation. Furthermore, we adopt instruction tuning techniques to empower the model with both judgments and explanations in the realm of OOC misinformation detection. Comprehensive experiments on two widely used benchmark datasets, NewsCLIPpings and VERITE, show that SDLE consistently outperforms existing methods, achieving F1-score improvements of 1.4% and 2.0%, respectively.
6 April 2022 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Supervised statistical models rely on large-scale high-quality labeled data, which is important for model training but expensive to construct. Therefore, instead of constructing new dataset, researchers have attempted to make full use of various existing heterogeneous datasets to boost model performance, considering it is ubiquitous that the same task may have multiple annotated data following different and incompatible annotation guidelines. Representative methods include the guide-feature method which use the knowledge projected from the source-side to the target-side as extra features for target model guidance, and the multi-task learning (MTL) method which simultaneously train on multiple heterogeneous annotations with shared parameters to gain resource-share knowledge. Though effective, the guide-feature method fails to directly use the source-side data as training data, and the MTL method ignores the implicit mappings between heterogeneous datasets. Compared with the above methods, directly converting the heterogeneous datasets into homogeneous datasets for target model training is a more straightforward and effective way to fully exploit heterogeneous resources. In this work, we propose a neural coupled sequence labeling model for heterogeneous annotation conversion. First, for each token, we map a given one-side tag into a set of bundled tags by concatenating the tag with all the possible tags at the other side. Then, we build a neural coupled model over the bundled tag space. Finally, we convert heterogeneous annotations into homogeneous annotations by performing constraint decoding on the coupled model. We also propose a pruning strategy to address the oversize issue of the bundled tag space, which improves efficiency without hurting model performance. Experiments for part-of-speech (POS) tagging, word segmentation (WS), and WS\&POS tagging tasks show that our proposed neural coupled model consistently outperforms several benchmark models for all the three tasks by large margin.
22 April 2026 in SSRN Electronic Journal
Out-Of-Context (OOC) misinformation detection aims to identify whether a genuine image is repurposed in a misleading textual content. Such misinformation arises when genuine images are detached from their original contexts and paired with semantically related but false texts, making detection based solely on image-text pairs challenging. Existing methods primarily exploit internal knowledge from image-text pairs or external knowledge from retrieved evidence, yet they often fail to establish deep alignment between these sources due to limited interaction. To address these limitations, we propose SDLE, a novel Shallow-Deep Learning framework that enhances Multimodal Large Language Models (MLLMs) for OOC misinformation detection. Specifically, SDLE first retrieves external evidence from various sources to gather contextual information related to image-text pairs. To build a shallow relationship between internal and external knowledge, SDLE designs a prompt learning method to reduce noise and capture key evidence. Subsequently, to build a deep relationship between them, SDLE aligns MLLMs for evidence-based rationale generation. Furthermore, we adopt instruction tuning techniques to empower the model with both judgments and explanations in the realm of OOC misinformation detection. Comprehensive experiments on two widely used benchmark datasets, NewsCLIPpings and VERITE, show that SDLE consistently outperforms existing methods, achieving F1-score improvements of 1.4% and 2.0%, respectively.
31 May 2025 in Expert Systems with Applications