Li Yuan

Li Yuan

Auto-generated profile, based on publicly available information

Li Yuan

School of Software Engineering South China University of Technology Guangzhou China

Research Focus & Metrics

Academic Career Length

15Years (2012-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
26 June 2023 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
LYLi Yuan
Li Yuan
YCYi Cai
Yi Cai
JWJin Wang
Jin Wang
QLQing Li
Qing Li
Multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) are two fundamental subtasks in the multimodal knowledge graph construction task. However, the existing methods usually handle two tasks independently, which ignores the bidirectional interaction between them. This paper is the first to propose jointly performing MNER and MRE as a joint multimodal entity-relation extraction (JMERE) task . Besides, the current MNER and MRE models only consider aligning the visual objects with textual entities in visual and textual graphs but ignore the entity-entity relationships and object-object relationships. To address the above challenges, we propose an edge-enhanced graph alignment network and a word-pair relation tagging (EEGA) for the JMERE task. Specifically, we first design a word-pair relation tagging to exploit the bidirectional interaction between MNER and MRE and avoid error propagation. Then, we propose an edge-enhanced graph alignment network to enhance the JMERE task by aligning nodes and edges in the cross-graph. Compared with previous methods, the proposed method can leverage the edge information to auxiliary alignment between objects and entities and find the correlations between entity-entity relationships and object-object relationships. Experiments are conducted to show the effectiveness of our model.

7 July 2023 in IEEE Transactions on Affective Computing
Verified
Journal Article
Open access
LYLi Yuan
Li Yuan
JWJin Wang
Jin Wang
LYLiang-Chih Yu
Liang-Chih Yu
XZXuejie Zhang
Xuejie Zhang
Aspect-based sentiment triplet extraction (ASTE) aims to extract triplets consisting of aspect terms and their associated opinion terms and sentiment polarities from sentences, a relatively new and challenging subtask of aspect-based sentiment analysis (ABSA). Previous studies have used either pipeline models or unified tagging schema models. These models ignore the syntactic relationships between the aspect and its corresponding opinion words, which leads them to mistakenly focus on syntactically unrelated words. One feasible option is to use a graph convolution network (GCN) to exploit syntactic information by propagating the representation from the opinion words to the aspect. However, such a method considers all syntactic dependencies to be of the same type and thus may still incorrectly associate unrelated words to the target aspect through the iterations of graph convolutional propagation. Herein, a syntax-aware transformer (SA-Transformer) is proposed to extend the GCN strategy by fully exploiting the dependency types of edges to block inappropriate propagation. The proposed approach can obtain different representations and weights even for edges with the same dependency type according to their adjacent dependency type of edges. Instead of using a GCN layer, we used an L -layer SA transformer to encode syntactic information in the word-pair representation to improve performance. Experimental results on four benchmark datasets show that the proposed model outperforms various previous models for ASTE.

8 December 2022 in IEEE Transactions on Artificial Intelligence
Verified
Journal Article
Closed access
LYLi Yuan
Li Yuan
JWJin Wang
Jin Wang
LYLiang-Chih Yu
Liang-Chih Yu
XZXuejie Zhang
Xuejie Zhang
Aspect-level sentiment classification (ASC) is designed to identify the sentiment orientation of given aspect terms in a sentence. Previous neural networks have used attention mechanisms to align context words with the appropriate aspect terms. Without considering syntactic dependencies, these models may erroneously focus on context words that are not related to the aspect terms. To address this issue, the graph convolution network (GCN) and the graph attention network (GAT) are proposed to build a graph based on the dependency parse tree, allowing the representations of context words to be propagated to the aspect terms according to their syntactic dependencies. However, these models consider all syntactic dependencies to be of the same type, and thus may result in inappropriate propagation of word representations in the graph. To further distinguish between the syntactic dependencies, this study proposes a syntactic graph attention network (SGAN) to incorporate the knowledge of dependency types into the graph attention network. The dependency types are modeled as edge embeddings to learn the attention weight of each edge according to its dependency type. By considering different dependency types and their weights, the proposed method can block inappropriate propagation to better associate the context words to aspect terms. To increase training process stability and enrich the diversity of graph representations, a weighted multihead attention is applied to compose the graph representations generated by different heads. The experimental results on five benchmark datasets show that the SGAN yields more accurate results than existing methods.

25 October 2024 in IEEE Transactions on Knowledge and Data Engineering
Verified
Journal Article
Closed access
LYLi Yuan
Li Yuan
YCYi Cai
Yi Cai
JXJingyu Xu
Jingyu Xu
QLQing Li
Qing Li
TWTao Wang
Tao Wang
Joint multimodal entity-relation extraction (JMERE) is a challenging task that involves two joint subtasks, i.e., named entity recognition and relation extraction, from multimodal data such as text sentences with associated images. Previous JMERE methods have primarily employed 1) pipeline models, which apply pre-trained unimodal models separately and ignore the interaction between tasks, or 2) word-pair relation tagging methods, which neglect neighboring word pairs. To address these limitations, we propose a fine-grained network for JMERE. Specifically, we introduce a fine-grained alignment module that utilizes a phrase-patch to establish connections between text phrases and visual objects. This module can learn consistent multimodal representations from multimodal data. Furthermore, we address the task-irrelevant image information issue by proposing a gate fusion module, which mitigates the impact of image noise and ensures a balanced representation between image objects and text representations. Furthermore, we design a multi-word decoder that enables ensemble prediction of tags for each word pair. This approach leverages the predicted results of neighboring word pairs, improving the ability to extract multi-word entities. Evaluation results from a series of experiments demonstrate the superiority of our proposed model over state-of-the-art models in JMERE.

8 August 2022 in Information Processing & Management
Verified
Journal Article
Closed access
LYLi Yuan
Li Yuan
JWJin Wang
Jin Wang
LYLiang-Chih Yu
Liang-Chih Yu
XZXuejie Zhang
Xuejie Zhang

28 October 2024
Verified
Conference Paper
Open access
LYLi Yuan
Li Yuan
YCYi Cai
Yi Cai
JHJunsheng Huang
Junsheng Huang
Joint Multimodal Entity-Relation Extraction (JMERE) is a challenging task that aims to extract entities and their relations from textimage pairs in social media posts. Existing methods for JMERE require large amounts of labeled data. However, gathering and annotating fine-grained multimodal data for JMERE poses significant challenges. Initially, we construct diverse and comprehensive multimodal few-shot datasets fitted to the original data distribution. To address the insufficient information in the few-shot setting, we introduce the Knowledge-Enhanced Cross-modal Prompt Model (KECPM) for JMERE. This method can effectively address the problem of insufficient information in the few-shot setting by guiding a large language model to generate supplementary background knowledge. Our proposed method comprises two stages: (1) a knowledge ingestion stage that dynamically formulates prompts based on semantic similarity guide ChatGPT generating relevant knowledge and employs self-reflection to refine the knowledge; (2) a knowledge-enhanced language model stage that merges the auxiliary knowledge with the original input and utilizes a transformerbased model to align with JMERE's required output format. We extensively evaluate our approach on a few-shot dataset derived from the JMERE dataset, demonstrating its superiority over strong baselines in terms of both micro and macro F1 scores. Additionally, we present qualitative analyses and case studies to elucidate the effectiveness of our model. Code and Data are released at https://github.com/YuanLi95/KECPM.