19 February 2022 in Information Processing & Management
15 November 2023 in IEEE Transactions on Affective Computing
Multimodal aspect-based sentiment analysis (MABSA) aims to determine the sentiment polarity of each aspect mentioned in the text based on multimodal content. Various approaches have been proposed to model multimodal sentiment features for each aspect via modal interactions. However, most existing approaches have two shortcomings: (1) The representation gap between textual and visual modalities may increase the risk of misalignment in modal interactions; (2) In some examples where the image is not related to the text, the visual information may not enrich the textual modality when learning aspect-based sentiment features. In such cases, blindly leveraging information from visual modal may introduce noises in reasoning the aspect-based sentiment expressions. To tackle the shortcomings mentioned above, we propose an end-to-end MABSA framework with image conversion and noise filtration. Specifically, to bridge the representation gap in different modalities, we attempt to translate images into the input space of a pre-trained language model (PLM). To this end, we develop an image-to-text conversion module that can convert an image to an implicit sequence of token embedding. Moreover, an aspect-oriented filtration module is devised to alleviate the noise in the implicit token embeddings, which consists of two attention operations. The former aims to create an enhanced aspect embedding as a query, and the latter seeks to use this query to retrieve relevant auxiliary information from the implicit token embeddings to supplement the textual content. After filtering the noise, we leverage a PLM to encode the text, aspect, and image prompt derived from filtered implicit token embeddings as sentiment features to perform aspect-based sentiment prediction. Experimental results on two MABSA datasets show that our framework achieves state-of-the-art performance. Furthermore, extensive experimental analysis demonstrates the proposed framework has superior robustness and efficiency.
2 January 2024 in IEEE Transactions on Computational Social Systems
In this article, we present our work in social emotion prediction to online trending topics. While most prior works focus on emotion from writers or the readers’ emotions evoked by news articles, we investigate discussions from massive social media users and explore the public feelings to the online trending topic. We employ user-generated “#hashtags” to indicate online trending topics and construct a large-scale Chinese dataset for social emotion prediction (LCSEP) to trending topics collected from the Chinese microblog Sina Weibo. It contains more than 20 000 trending topics, each with social emotions voted in 24 fine-grained types, and gathers hashtags, posts, comments, and related metadata to give each trending topic a thorough context. We also propose a Hashtag- and Topic-Enhanced Attention Model (HTEAM) that combines a pretrained BERT model, a neural topic model, and an attention mechanism via joint training to understand social emotion. Experiments show that HTEAM outperforms baselines and achieves the state-of-the-art result.
22 July 2024 in IEEE Transactions on Affective Computing
Cross-domain sentiment analysis (CDSA) aims to predict the sentiment polarities of reviews in the target domain using a sentiment classifier learned from the source labeled domain. Most existing studies are dominant with adversarial learning methods and focus on learning domain-invariant sentiment representations in both the source and target domains. However, since sentiment-specific features are not explicitly decoupled, the model may confuse domain features with sentiment features, thus affecting its generalization ability on target domains. Unlike previous studies, in this paper, we tackle the CDSA task from the view of disentangled representation learning, which explicitly learns the disentangled representations of review, focusing in particular on sentiment and domain semantics. Specifically, we disentangle sentiment-specific and domain-specific features from the text representation of the review by two different linear transformations. Then, we introduce a straightforward disentangled loss to disallow the sentiment-specific feature to capture domain information. Moreover, we leverage target unlabeled data to improve the quality of the learned sentiment-specific features via prototypical learning. It indirectly encourages the sentiment-specific features of target samples having potentially different classes more discriminative. Extensive experiments on widely used CDSA datasets show that our method surpasses competitive baselines and achieves new state-of-the-art results, demonstrating its effectiveness and superiority.
28 June 2023 in International Journal of Machine Learning and Cybernetics