26 June 2023 in Proceedings of the AAAI Conference on Artificial Intelligence
Multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) are two fundamental subtasks in the multimodal knowledge graph construction task. However, the existing methods usually handle two tasks independently, which ignores the bidirectional interaction between them. This paper is the first to propose jointly performing MNER and MRE as a joint multimodal entity-relation extraction (JMERE) task . Besides, the current MNER and MRE models only consider aligning the visual objects with textual entities in visual and textual graphs but ignore the entity-entity relationships and object-object relationships. To address the above challenges, we propose an edge-enhanced graph alignment network and a word-pair relation tagging (EEGA) for the JMERE task. Specifically, we first design a word-pair relation tagging to exploit the bidirectional interaction between MNER and MRE and avoid error propagation. Then, we propose an edge-enhanced graph alignment network to enhance the JMERE task by aligning nodes and edges in the cross-graph. Compared with previous methods, the proposed method can leverage the edge information to auxiliary alignment between objects and entities and find the correlations between entity-entity relationships and object-object relationships. Experiments are conducted to show the effectiveness of our model.
7 July 2023 in IEEE Transactions on Affective Computing
Aspect-based sentiment triplet extraction (ASTE) aims to extract triplets consisting of aspect terms and their associated opinion terms and sentiment polarities from sentences, a relatively new and challenging subtask of aspect-based sentiment analysis (ABSA). Previous studies have used either pipeline models or unified tagging schema models. These models ignore the syntactic relationships between the aspect and its corresponding opinion words, which leads them to mistakenly focus on syntactically unrelated words. One feasible option is to use a graph convolution network (GCN) to exploit syntactic information by propagating the representation from the opinion words to the aspect. However, such a method considers all syntactic dependencies to be of the same type and thus may still incorrectly associate unrelated words to the target aspect through the iterations of graph convolutional propagation. Herein, a syntax-aware transformer (SA-Transformer) is proposed to extend the GCN strategy by fully exploiting the dependency types of edges to block inappropriate propagation. The proposed approach can obtain different representations and weights even for edges with the same dependency type according to their adjacent dependency type of edges. Instead of using a GCN layer, we used an L -layer SA transformer to encode syntactic information in the word-pair representation to improve performance. Experimental results on four benchmark datasets show that the proposed model outperforms various previous models for ASTE.
8 December 2022 in IEEE Transactions on Artificial Intelligence
Aspect-level sentiment classification (ASC) is designed to identify the sentiment orientation of given aspect terms in a sentence. Previous neural networks have used attention mechanisms to align context words with the appropriate aspect terms. Without considering syntactic dependencies, these models may erroneously focus on context words that are not related to the aspect terms. To address this issue, the graph convolution network (GCN) and the graph attention network (GAT) are proposed to build a graph based on the dependency parse tree, allowing the representations of context words to be propagated to the aspect terms according to their syntactic dependencies. However, these models consider all syntactic dependencies to be of the same type, and thus may result in inappropriate propagation of word representations in the graph. To further distinguish between the syntactic dependencies, this study proposes a syntactic graph attention network (SGAN) to incorporate the knowledge of dependency types into the graph attention network. The dependency types are modeled as edge embeddings to learn the attention weight of each edge according to its dependency type. By considering different dependency types and their weights, the proposed method can block inappropriate propagation to better associate the context words to aspect terms. To increase training process stability and enrich the diversity of graph representations, a weighted multihead attention is applied to compose the graph representations generated by different heads. The experimental results on five benchmark datasets show that the SGAN yields more accurate results than existing methods.
25 October 2024 in IEEE Transactions on Knowledge and Data Engineering
Joint multimodal entity-relation extraction (JMERE) is a challenging task that involves two joint subtasks, i.e., named entity recognition and relation extraction, from multimodal data such as text sentences with associated images. Previous JMERE methods have primarily employed 1) pipeline models, which apply pre-trained unimodal models separately and ignore the interaction between tasks, or 2) word-pair relation tagging methods, which neglect neighboring word pairs. To address these limitations, we propose a fine-grained network for JMERE. Specifically, we introduce a fine-grained alignment module that utilizes a phrase-patch to establish connections between text phrases and visual objects. This module can learn consistent multimodal representations from multimodal data. Furthermore, we address the task-irrelevant image information issue by proposing a gate fusion module, which mitigates the impact of image noise and ensures a balanced representation between image objects and text representations. Furthermore, we design a multi-word decoder that enables ensemble prediction of tags for each word pair. This approach leverages the predicted results of neighboring word pairs, improving the ability to extract multi-word entities. Evaluation results from a series of experiments demonstrate the superiority of our proposed model over state-of-the-art models in JMERE.
8 August 2022 in Information Processing & Management