Automatic detection of multimodal fake news has gained a widespread attention recently. Many existing approaches seek to fuse unimodal features to produce multimodal news representations. However, the potential of powerful cross-modal contrastive learning methods for fake news detection has not been well exploited. Besides, how to aggregate features from different modalities to boost the performance of the decision-making process is still an open question. To address that, we propose COOLANT, a cross-modal contrastive learning framework for multimodal fake news detection, aiming to achieve more accurate image-text alignment. To further capture the fine-grained alignment between vision and language, we leverage an auxiliary task to soften the loss term of negative samples during the contrast process. A cross-modal fusion module is developed to learn the cross-modality correlations. An attention mechanism with an attention guidance module is implemented to help effectively and interpretably aggregate the aligned unimodal representations and the cross-modality correlations. Finally, we evaluate the COOLANT and conduct a comparative study on two widely used datasets, Twitter and Weibo. The experimental results demonstrate that our COOLANT outperforms previous approaches by a large margin and achieves new state-of-the-art results on the two datasets.

Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian-Guang Lou, Shuai Ma. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Background: A healthy lifestyle is beneficial to individuals’ health. However, little is known about the associations of lifestyle factors with mental health and well-being. This study examined the associations of lifestyle factors with mental health (i.e., depression, anxiety, loneliness, perceived pressure, and self-rated health status) and well-being in Chinese adults.Methods: A nationally representative survey was conducted in China from 20 June 2022 to 31 August 2022. Data from the survey were analyzed using multiple linear regression to determine the associations of lifestyle with mental health and well-being in Chinese adults. Standardized regression coefficients (β) and 95% confidence intervals (CIs) were estimated using multiple linear regression.Results: The survey included 28,138 Chinese adults. Multiple linear regression results showed that there were significant negative associations of lifestyle scores with scores of depression (β = −0.93, 95% CI: −0.98, −0.88), anxiety (β = −0.71, 95% CI: −0.76, −0.67), loneliness (β = −0.23, 95% CI: −0.24, −0.21), and perceived pressure (β = −0.19, 95% CI: −0.22, −0.16). Moreover, there were significant positive associations of lifestyle with self-rated health status (β = 1.99, 95% CI: 1.79, 2.20) and well-being (β = 0.96, 95% CI: 0.91, 1.02).Conclusion: This study provides insight into the associations of lifestyle factors with mental health and well-being and highlights the importance of improving and maintaining healthy lifestyle behaviors for favorable mental health and well-being.

Multi-modal entity alignment (MMEA) aims to identify equivalent entities between multi-modal knowledge graphs (MMKGs), where entities can be associated with related images. Most existing studies rely heavily on the automatically learned multi-modal fusion modules, which may allow redundant information such as misleading clues in the generated entity representations, impeding the feature consistency of equivalent entities. To this end, we propose a variational framework for MMEA via information bottleneck, termed as IBMEA, by emphasizing alignment-relevant information while suppressing alignment-irrelevant information in entity representations. Specifically, we first develop multi-modal variational encoders that represent modal-specific features as probability distributions. Then, we propose four modal-specific information bottleneck regularizers to limit the misleading clues in the modal-specific entity representations. Finally, we propose a modal-hybrid information contrastive regularizer to integrate modal-specific representations and ensure the similarity of equivalent entities between MMKGs to achieve MMEA. We conduct extensive experiments on 2 cross-KG and 3 bilingual MMEA datasets. Experimental results demonstrate that our model consistently outperforms previous state-of-the-art methods, and also shows promising and robust performance especially in the low-resource and high-noise data scenarios.