Jiang Li

Jiang Li

Auto-generated profile, based on publicly available information

Jiang Li

School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
22 March 2023 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
JLJiang Li
Jiang Li
XWXiaoping Wang
Xiaoping Wang
GLGuoqing Lv
Guoqing Lv
ZZZhigang Zeng
Zhigang Zeng
Emotion Recognition in Conversation (ERC) plays a significant part in Human-Computer Interaction (HCI) systems since it can provide empathetic services. Multimodal ERC can mitigate the drawbacks of uni-modal approaches. Recently, Graph Neural Networks (GNNs) have been widely used in a variety of fields due to their superior performance in relation modeling. In multimodal ERC, GNNs are capable of extracting both long-distance contextual information and inter-modal interactive information. Unfortunately, since existing methods such as MMGCN directly fuse multiple modalities, redundant information may be generated and diverse information may be lost. In this work, we present a directed Graph based Cross-modal Feature Complementation (GraphCFC) module that can efficiently model contextual and interactive information. GraphCFC alleviates the problem of heterogeneity gap in multimodal fusion by utilizing multiple subspace extractors and Pair-wise Cross-modal Complementary (PairCC) strategy. We extract various types of edges from the constructed graph for encoding, thus enabling GNNs to extract crucial contextual and interactive information more accurately when performing message passing. Furthermore, we design a GNN structure called GAT-MLP, which can provide a new unified network framework for multimodal learning. The experimental results on two benchmark datasets show that our GraphCFC outperforms the state-of-the-art (SOTA) approaches.

24 March 2023 in IEEE Transactions on Affective Computing
Verified
Journal Article
Closed access
JLJiang Li
Jiang Li
XWXiaoping Wang
Xiaoping Wang
GLGuoqing Lv
Guoqing Lv
ZZZhigang Zeng
Zhigang Zeng
Multimodal Emotion Recognition in Conversation (ERC) plays an influential role in the field of human-computer interaction and conversational robotics since it can motivate machines to provide empathetic services. Multimodal data modeling is an up-and-coming research area in recent years, which is inspired by human capability to integrate multiple senses. Several graph-based approaches claim to capture interactive information between modalities, but the heterogeneity of multimodal data makes these methods prohibit optimal solutions. In this work, we introduce a multimodal fusion approach named Graph and Attention based Two-stage Multi-source Information Fusion (GA2MIF) for emotion detection in conversation. Our proposed method circumvents the problem of taking heterogeneous graph as input to the model while eliminating complex redundant connections in the construction of graph. GA2MIF focuses on contextual modeling and cross-modal modeling through leveraging Multi-head Directed Graph ATtention networks (MDGATs) and Multi-head Pairwise Cross-modal ATtention networks (MPCATs), respectively. Extensive experiments on two public datasets (i.e., IEMOCAP and MELD) demonstrate that the proposed GA2MIF has the capacity to validly capture intra-modal long-range contextual information and inter-modal complementary information, as well as outperforms the prevalent State-Of-The-Art (SOTA) models by a remarkable margin.

16 April 2024 in IEEE Transactions on Affective Computing
Verified
Journal Article
Closed access
JLJiang Li
Jiang Li
XWXiaoping Wang
Xiaoping Wang
YLYingjian Liu
Yingjian Liu
ZZZhigang Zeng
Zhigang Zeng
Multimodal emotion recognition in conversation (ERC) has garnered growing attention from research communities in various fields. In this paper, we propose a Crossmodal Fusion Network with Emotion-Shift Awareness (CFNESA) for ERC. Extant approaches employ each modality equally without distinguishing the amount of emotional information in these modalities, rendering it hard to adequately extract complementary information from multimodal data. To cope with this problem, in CFN-ESA, we treat textual modality as the primary source of emotional information, while visual and acoustic modalities are taken as the secondary sources. Besides, most multimodal ERC models ignore emotion-shift information and overfocus on contextual information, leading to the failure of emotion recognition under emotion-shift scenario. We elaborate an emotion-shift module to address this challenge. CFNESA mainly consists of unimodal encoder (RUME), cross-modal encoder (ACME), and emotion-shift module (LESM). RUME is applied to extract conversation-level contextual emotional cues while pulling together data distributions between modalities; ACME is utilized to perform multimodal interaction centered on textual modality; LESM is used to model emotion shift and capture emotion-shift information, thereby guiding the learning of the main task. Experimental results demonstrate that CFN-ESA can effectively promote performance for ERC and remarkably outperform state-of-the-art models.

1 January 2024 in Cyborg and Bionic Systems
Verified
Journal Article
Open access
JLJunnan Li
Junnan Li
JLJiang Li
Jiang Li
XWXiaoping Wang
Xiaoping Wang
XZXin Zhan
Xin Zhan
ZZZhigang Zeng
Zhigang Zeng
Emotion recognition from physiological signals (ERPS) has drawn tremendous attention and can be potentially applied to numerous fields. Since physiological signals are nonstationary time series with high sampling frequency, it is challenging to directly extract features from them. Additionally, there are 2 major challenges in ERPS: (a) how to adequately capture the correlations between physiological signals at different times and between different types of physiological signals and (b) how to effectively minimize the negative effect caused by temporal covariate shift (TCS). To tackle these problems, we propose a domain generalization and residual network-based approach for emotion recognition from physiological signals (DGR-ERPS). We first pre-extract time- and frequency-domain features from the original time series to compose a new time series. Then, in order to fully extract the correlation information of different physiological signals, these time series are converted into 3D image data to serve as input for a residual-based feature encoder (RBFE). In addition, we introduce a domain generalization-based technique to mitigate the issue posed by TCS. We have conducted extensive experiments on 2 real-world datasets, and the results indicate that our DGR-ERPS achieves superior performance under both TCS and non-TCS scenarios.

19 June 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
JLJiang Li
Jiang Li
XWXiaoping Wang
Xiaoping Wang
ZZZhigang Zeng
Zhigang Zeng
Multimodal emotion recognition in conversation (MERC) has garnered substantial research attention recently. Existing MERC methods face several challenges: (1) they fail to fully harness direct inter-modal cues, possibly leading to less-than-thorough cross-modal modeling; (2) they concurrently extract information from the same and different modalities at each network layer, potentially triggering conflicts from the fusion of multi-source data; (3) they lack the agility required to detect dynamic sentimental changes, perhaps resulting in inaccurate classification of utterances with abrupt sentiment shifts. To address these issues, a novel approach named GraphSmile is proposed for tracking intricate emotional cues in multimodal dialogues. GraphSmile comprises two key components, i.e., GSF and SDP modules. GSF ingeniously leverages graph structures to alternately assimilate inter-modal and intra-modal emotional dependencies layer by layer, adequately capturing cross-modal cues while effectively circumventing fusion conflicts. SDP is an auxiliary task to explicitly delineate the sentiment dynamics between utterances, promoting the model's ability to distinguish sentimental discrepancies. GraphSmile is effortlessly applied to multimodal sentiment analysis in conversation (MSAC), thus enabling simultaneous execution of MERC and MSAC tasks. Empirical results on multiple benchmarks demonstrate that GraphSmile can handle complex emotional and sentimental patterns, significantly outperforming baseline models.

25 July 2024 in Science China Information Sciences
Verified
Journal Article
Closed access
PDF file
YLYingjian Liu
Yingjian Liu
JLJiang Li
Jiang Li
XWXiaoping Wang
Xiaoping Wang
ZZZhigang Zeng
Zhigang Zeng