Identifying sarcastic clues from both textual and visual information has become an important research issue, called Multimodal Sarcasm Detection. In this paper, we investigate multimodal sarcasm detection from a novel perspective, where a multimodal graph contrastive learning strategy is proposed to fuse and distinguish the sarcastic clues for textual modality and visual modality. Specifically, we first utilize object detection to derive the crucial visual regions accompanied by their captions of the images, which allows better learning of the key visual regions of visual modality. In addition, to make full use of the semantic information of the visual modality, we employ optical character recognition to extract the textual content in the images. Then, based on image regions, the textual content of visual modality, and the context of the textual modality, we build a multimodal graph for each sample to model the intricate sarcastic relations between modalities. Furthermore, we devise a graph-oriented contrastive learning strategy to leverage the correlations in the same label and differences between different labels, so as to capture better multimodal representations for multimodal sarcasm detection. Extensive experiments show that our method outperforms the previous best baseline models (with a 2.47% improvement in Accuracy, a 1.99% improvement in F-score, and a 2.20% improvement in Macro F-score). The ablation study shows that both multimodal graph structure and graph-oriented contrastive learning are important to our framework. Further, the experiments of using different pre-trained methods show that the proposed multimodal graph contrastive learning framework can directly work with various pre-trained models and achieve outstanding performance in multimodal sarcasm detection.

Existing aspect-based/category sentiment analysis methods have shown great success in detecting sentiment polarity toward a given aspect in a sentence with supervised learning, where the training and inference stages share the same pre-defined set of aspects. However, in practice, the aspect categories are changing rather than keeping fixed over time. Dealing with unseen aspect categories is under-explored in existing methods. In this article, we formulate a new few-shot aspect category sentiment analysis (FSACSA) task, which aims to effectively predict the sentiment polarity of previously unseen aspect categories. To this end, we propose a novel Aspect-Focused Meta-Learning (AFML) framework that constructs aspect-aware and aspect-contrastive representations from external knowledge to match the target aspect with aspects in the training set. Concretely, we first construct two auxiliary contrastive sentences for a given sentence with the incorporation of external knowledge, enabling the learning of sentence representations with a better generalization. Then, we devise an aspect-focused induction network to leverage the contextual sentiment toward a given aspect to refine the label vectors. Furthermore, we employ the episode-based meta-learning algorithm to train the whole network, so as to learn to generalize to novel aspects. Extensive experiments on multiple real-life datasets show that our proposed AFML framework achieves the state-of-the-art results for the FSACSA task.

Ang Li, Bin Liang, Jingqian Zhao, Bowen Zhang, Min Yang, Ruifeng Xu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Emotion recognition in conversations (ERC) needs to detect the emotion of each utterance in conversations. However, it is difficult for machines to recognize the emotion of utterances like humans, partly because of the lack of commonsense knowledge. Despite existing efforts gradually incorporate knowledge in ERC, they can not adaptively adjust knowledge according to different utterances and their context. In this paper, we propose a knowledge selection framework SKSEC ( S elect K nowledge in light of S entiment E motion and C ontext). In the SKSEC framework, firstly, external knowledge is eliminated by three Knowledge Elimination (KE) modules. More concretely, In word-level KE, the concept knowledge different from the sentiment corresponding to the word in utterances is randomly eliminated. In utterance- or context-level KE, If the similarity between the knowledge representation and the emotion label representation of the current utterance or its context is less than the preset threshold, the knowledge will be eliminated. Then we refine the weight of knowledge using two Graph ATtention (GAT) mechanisms. Specifically, In Sentics GAT, we employ a dimensional emotion model to measure words in utterances and their corresponding knowledge and adjust the weight of knowledge according to their emotional similarity. In Semantics GAT, the weight of knowledge is adjusted according to the semantic similarity between context and incorporated knowledge. Finally, we feed the selected knowledge to the most advanced models to evaluate the quality of knowledge. The experimental results show that the SKSEC framework can effectively improve the performance of the model by eliminating and refining external knowledge in different size and domain datasets.

Video inpainting is a crucial task with diverse applications, including fine-grained video editing, video recovery, and video dewatermarking. However, most existing video inpainting methods primarily focus on visual content completion while neglecting text information. There are only a limited number of text-guided video inpainting techniques, and these techniques struggle with maintaining visual quality and exhibit poor semantic representation capabilities. In this paper, we introduce CoCoCo, a text-guided video inpainting diffusion framework. To address the aforementioned challenges, we enhance both the training data and model structure. Specifically, we devise an instance-aware region selection strategy for masked area sampling and develop a novel motion block that incorporates efficient 3D full attention and textual cross attention. Additionally, our CoCoCo framework can be seamlessly integrated with various personalized text-to-image diffusion models through a delicate training-free transfer mechanism. Comprehensive experiments demonstrate that CoCoCo can create high-quality visual content with enhanced temporal consistency, improved text controllability, and better compatibility with personalized image models.