Multimedia content has become ubiquitous on social media platforms, leading to the rise of multimodal misinformation (MM) and the urgent need for effective strategies to detect and prevent its spread. In recent years, the challenge of multimodal misinformation detection (MMD) has garnered significant attention by researchers and has mainly involved the creation of annotated, weakly annotated, or synthetically generated training datasets, along with the development of various deep learning MMD models. However, the problem of unimodal bias has been overlooked, where specific patterns and biases in MMD benchmarks can result in biased or unimodal models outperforming their multimodal counterparts on an inherently multimodal task, making it difficult to assess progress. In this study, we systematically investigate and identify the presence of unimodal bias in widely used MMD benchmarks, namely VMU-Twitter and COSMOS. To address this issue, we introduce the “VERification of Image-TExt pairs” (VERITE) benchmark for MMD which incorporates real-world data, excludes “asymmetric multimodal misinformation” and utilizes “modality balancing”. We conduct an extensive comparative study with a transformer-based architecture that shows the ability of VERITE to effectively address unimodal bias, rendering it a robust evaluation framework for MMD. Furthermore, we introduce a new method—termed Crossmodal HArd Synthetic MisAlignment (CHASMA)—for generating realistic synthetic training data that preserve crossmodal relations between legitimate images and false human-written captions. By leveraging CHASMA in the training process, we observe consistent and notable improvements in predictive performance on VERITE; with a 9.2% increase in accuracy. We release our code at: https://github.com/stevejpapad/image-text-verification

With the expansion of social media and the increasing dissemination of multimedia content, the spread of misinformation has become a major concern. This necessitates effective strategies for multimodal misinformation detection (MMD) that detect whether the combination of an image and its accompanying text could mislead or misinform. Due to the data-intensive nature of deep neural networks and the labor-intensive process of manual annotation, researchers have been exploring various methods for automatically generating synthetic multimodal misinformation - which we refer to as Synthetic Misinformers - in order to train MMD models. However, limited evaluation on real-world misinformation and a lack of comparisons with other Synthetic Misinformers makes difficult to assess progress in the field. To address this, we perform a comparative study on existing and new Synthetic Misinformers that involves (1) out-of-context (OOC) image-caption pairs, (2) cross-modal named entity inconsistency (NEI) as well as (3) hybrid approaches and we evaluate them against real-world misinformation; using the COSMOS benchmark. The comparative study showed that our proposed CLIP-based Named Entity Swapping can lead to MMD models that surpass other OOC and NEI Misinformers in terms of multimodal accuracy and that hybrid approaches can lead to even higher detection accuracy. Nevertheless, after alleviating information leakage from the COSMOS evaluation protocol, low Sensitivity scores indicate that the task is significantly more challenging than previous studies suggested. Finally, our findings showed that NEI-based Synthetic Misinformers tend to suffer from a unimodal bias, where text-only models can outperform multimodal ones.

Stress has a severe impact on individuals irrespective of age, sex, work, or background. The reliable development of stress detection techniques enhances the social, educational, physical, economic, and professional quality of life, preventing chronic stress and proposing alleviation strategies. Research studies examine psychological, cognitive, behavioral, and physiological reactions to identify stress adequately. Deep Learning (DL) has received significant attention in recent years as it deals with high-dimensional, heterogeneous data and automatically learns representative features. This paper presents a survey on stress detection with recent DL approaches, leveraging data from all possible sources (physiological, speech, facial expressions, gestures, and social media content). The methodological outlines, the best results, and the main contributions of each study are discussed. We also describe publicly available datasets used by several of the presented works. Finally, we emphasize various open issues within the field of research and highlight key directions for future work

Near-infrared spectroscopy (NIRS)-based brain-computer interface (BCI) systems use feature extraction methods relying mainly on the slope characteristics and mean changes of the hemodynamic responses in respect to certain mental tasks. Nevertheless, spatial patterns across the measurement channels have been detected and should be considered during the feature vector extraction stage of the BCI realization. In this paper, a graph signal processing (GSP) approach for feature extraction is adopted in order to capture the aforementioned spatial information of the NIRS signals. The proposed GSP-based methodology for feature extraction in NIRS-based BCI systems, namely graph NIRS (GNIRS), is applied on a publicly available dataset of NIRS recordings during a mental arithmetic task. GNIRS exhibits higher classification rates (CRs), up to 92.52%, as compared to the CRs of two state-of-the-art feature extraction methodologies related to slope and mean values of hemodynamic response, i.e., 90.35% and 82.60%, respectively. In addition, GNIRS leads to the formation of feature vectors with reduced dimensionality in comparison with the baseline approaches. Moreover, it is shown to facilitate high CRs even from the first second after the onset of the mental task, paving the way for faster NIRS-based BCI systems.

Online misinformation is often multimodal in nature, i.e., caused by misleading associations between texts and accompanying images. To support the fact-checking process, researchers have been recently developing automatic multimodal methods that gather and analyze external information, evidence, related to the image–text pairs under examination. However, prior works incorrectly assumed that all external information collected from the Web is relevant. In this study, we introduce a “relevant evidence detection” (RED) module to discern whether each piece of evidence is relevant, to support or refute the claim. Specifically, we develop the “relevant evidence detection directed transformer” (RED-DOT) and explore multiple architectural variants (e.g., single or dual-stage) and mechanisms (e.g., “guided attention”). Extensive ablation and comparative experiments demonstrate that RED-DOT outperforms the state-of-the-art (SotA), achieving up to 33.7% accuracy improvement on the VERITE benchmark. Furthermore, our evidence reranking and element-wise modality fusion led to RED-DOT surpassing the SotA on NewsCLIPpings+ by up to 3% without the need for numerous evidence or multiple backbone encoders. We release our code at: https://github.com/stevejpapad/relevant-evidencedetection.

Automated fact-checking (AFC) is garnering increasing attention by researchers aiming to help fact-checkers combat the increasing spread of misinformation online. While many existing AFC methods incorporate external information from the Web to help examine the veracity of claims, they often overlook the importance of verifying the source and quality of collected “evidence”. One overlooked challenge involves the reliance on “leaked evidence”, information gathered directly from fact-checking websites and used to train AFC systems, resulting in an unrealistic setting for early misinformation detection. Similarly, the inclusion of information from unreliable sources can undermine the effectiveness of AFC systems. To address these challenges, we present a comprehensive approach to evidence verification and filtering. We create the “CREDible, Unreliable or LEaked” (CREDULE) dataset, which consists of 91,632 articles classified as Credible, Unreliable and Fact-checked (Leaked). Additionally, we introduce the EVidence VERification Network (EVVER-Net), trained on CREDULE to detect leaked and unreliable evidence in both short and long texts. EVVER-Net can be used to filter evidence collected from the Web, thus enhancing the robustness of end-to-end AFC systems. We experiment with various language models and show that EVVER-Net can demonstrate impressive performance of up to 91.5% and 94.4% accuracy, while leveraging domain credibility scores along with short or long texts, respectively. Finally, we assess the evidence provided by widely-used fact-checking datasets including LIAR-PLUS, MOCHEG, FACTIFY, NewsCLIPpings+ and VERITE, some of which exhibit concerning rates of leaked and unreliable evidence.