Jianfei Yu

Jianfei Yu

Auto-generated profile, based on publicly available information

Jianfei Yu

Nanjing University of Science and Technology, Nanjing, China

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
6 December 2019 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
JYJianfei Yu
Jianfei Yu
JJJing Jiang
Jing Jiang
RXRui Xia
Rui Xia
Entity-level (aka target-dependent) sentiment analysis of social media posts has recently attracted increasing attention, and its goal is to predict the sentiment orientations over individual target entities mentioned in users’ posts. Most existing approaches to this task primarily rely on the textual content, but fail to consider the other important data sources (e.g., images, videos, and user profiles), which can potentially enhance these text-based approaches. Motivated by the observation, we study entity-level multimodal sentiment classification in this article, and aim to explore the usefulness of images for entity-level sentiment detection in social media posts. Specifically, we propose an Entity-Sensitive Attention and Fusion Network (ESAFN) for this task. First, to capture the intra-modality dynamics, ESAFN leverages an effective attention mechanism to generate entity-sensitive textual representations, followed by aggregating them with a textual fusion layer. Next, ESAFN learns the entity-sensitive visual representation with an entity-oriented visual attention mechanism, followed by a gated mechanism to eliminate the noisy visual context. Moreover, to capture the inter-modality dynamics, ESAFN further fuses the textual and visual representations with a bilinear interaction layer. To evaluate the effectiveness of ESAFN, we manually annotate the sentiment orientation over each given entity based on two recently released multimodal NER datasets, and show that ESAFN can significantly outperform several highly competitive unimodal and multimodal methods.

28 April 2022 in IEEE Transactions on Affective Computing
Verified
Journal Article
Closed access
JYJianfei Yu
Jianfei Yu
KCKai Chen
Kai Chen
RXRui Xia
Rui Xia
Aspect-based multimodal sentiment analysis (ABMSA) aims to determine the sentiment polarities of each aspect or entity mentioned in a multimodal post or review. Previous studies to ABMSA can be summarized into two subtasks: aspect-term based multimodal sentiment classification (ATMSC) and aspect-category based multimodal sentiment classification (ACMSC). However, these existing studies have three shortcomings: (1) ignoring the object-level semantics in images; (2) primarily focusing on aspect-text and aspect-image interactions; (3) failing to consider the semantic gap between text and image representations. To tackle these issues, we propose a general Hierarchical Interactive Multimodal Transformer (HIMT) model for ABMSA. Specifically, we extract salient features with semantic concepts from images via an object detection method, and then propose a hierarchical interaction module to first model the aspect-text and aspect-image interactions, followed by capturing the text-image interactions. Moreover, an auxiliary reconstruction module is devised to largely eliminate the semantic gap between text and image representations. Experimental results show that our HIMT model significantly outperforms the state-of-the-art methods on two benchmarks for ATMSC and one benchmark for ACMSC.

5 December 2022 in IEEE Transactions on Affective Computing
Verified
Journal Article
Closed access
FWFanfan Wang
Fanfan Wang
ZDZixiang Ding
Zixiang Ding
RXRui Xia
Rui Xia
ZLZhaoyu Li
Zhaoyu Li
JYJianfei Yu
Jianfei Yu
Conversation is an important form of human communication and contains a large number of emotions. It is interesting to discover emotions and their causes in conversations. Conversation in its natural form is multimodal. Many studies have been carried out on multimodal emotion recognition in conversations, yet there is still a lack of work on multimodal emotion cause analysis. In this work, we introduce a new task named Multimodal Emotion-Cause Pair Extraction in Conversations, aiming to jointly extract emotions and the corresponding causes from conversations reflected in multiple modalities (i.e., text, audio and video). We accordingly construct a multimodal conversational emotion cause dataset, Emotion-Cause-in-Friends, which contains 9,794 multimodal emotion-cause pairs among 13,619 utterances in the Friends sitcom. We benchmark the task by establishing two baseline systems including a heuristic approach considering inherent patterns in the location of causes and emotions and a deep learning approach that incorporates multimodal features for emotion-cause pair extraction, and conduct the human performance test for comparison. Furthermore, we investigate the effect of multimodal information, explore the potential of incorporating commonsense knowledge, and perform the task under both Static and Real-time settings.

10 October 2018 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
JYJianfei Yu
Jianfei Yu
JJJing Jiang
Jing Jiang
RXRui Xia
Rui Xia
Extracting aspect terms and opinion terms are two fundamental tasks in opinion mining. The recent success of deep learning has inspired various neural network architectures, which have been shown to achieve highly competitive performance in these two tasks. However, most existing methods fail to explicitly consider the syntactic relations among aspect terms and opinion terms, which may lead to the inconsistencies between the model predictions and the syntactic constraints. To this end, we first apply a multi-task learning framework to implicitly capture the relations between the two tasks, and then propose a global inference method by explicitly modelling several syntactic constraints among aspect term extraction and opinion term extraction to uncover their intra-task and inter-task relationship, which seeks an optimal solution over the neural predictions for both tasks. Extensive evaluations on three benchmark datasets demonstrate that our global inference approach is able to bring consistent improvements over several base models in different scenarios.


1 January 2024
Verified
Conference Paper
Open access
PDF file
FWFanfan Wang
Fanfan Wang
HMHeqing Ma
Heqing Ma
RXRui Xia
Rui Xia
JYJianfei Yu
Jianfei Yu
Erik CambriaErik Cambria
Erik Cambria
Fanfan Wang, Heqing Ma, Rui Xia, Jianfei Yu, Erik Cambria. Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024). 2024.