16 August 2022 in IEEE Journal of Biomedical and Health Informatics
The dynamic uncertain relationship among each brain region is a necessary factor that limits EEG-based emotion recognition. It is a thought-provoking problem to availably employ time-varying spatial and temporal characteristics from multi-channel electroencephalogram (EEG) signals. Although deep learning has made remarkable achievements in emotion recognition, the biological topological information among brain regions does not fully exploit, which is vital for EEG-based emotion recognition. In response to this problem, we design a hybrid model called ST-GCLSTM, which comprises a spatial-graph convolutional network (SGCN) module and an attention-enhanced bi-directional Long Short-Term Memory (LSTM) module. The main advantage of ST-GCLSTM is that it can consider the biological topology information of each brain region to extract representative spatial-temporal features from multiple EEG channels. Specifically, we construct two layers SGCN by introducing adjacency matrices to adaptively learn the intrinsic connection among different EEG channels. Moreover, an attention-enhanced mechanism is placed into a bi-directional LSTM module to extract the crucial spatial-temporal features from sequential EEG data, and then these features serve as the input layer of the classifier to learn discriminative emotion-related features. Extensive experiments on the DEAP, SEED, and SEED-IV datasets demonstrate the effectiveness of the proposed ST-GCLSTM model, revealing that our model had an absolute performance improvement over state-of-the-art strategies.
13 October 2023 in IEEE Transactions on Neural Networks and Learning Systems
The electroencephalogram (EEG) signal has become a highly effective decoding target for emotion recognition and has garnered significant attention from researchers. Its spatial topological and time-dependent characteristics make it crucial to explore both spatial information and temporal information for accurate emotion recognition. However, existing studies often focus on either spatial or temporal aspects of EEG signals, neglecting the joint consideration of both perspectives. To this end, this article proposes a hybrid network consisting of a dynamic graph convolution (DGC) module and temporal self-attention representation (TSAR) module, which concurrently incorporates the representative knowledge of spatial topology and temporal context into the EEG emotion recognition task. Specifically, the DGC module is designed to capture the spatial functional relationships within the brain by dynamically updating the adjacency matrix during the model training process. Simultaneously, the TSAR module is introduced to emphasize more valuable time segments and extract global temporal features from EEG signals. To fully exploit the interactivity between spatial and temporal information, the hierarchical cross-attention fusion (H-CAF) module is incorporated to fuse the complementary information from spatial and temporal features. Extensive experimental results on the DEAP, SEED, and SEED-IV datasets demonstrate that the proposed method outperforms other state-of-the-art methods.
6 January 2024 in Neural Networks
13 August 2024 in Neural Networks
4 July 2024 in IEEE Transactions on Computational Social Systems
Multimodal emotion recognition (MER) has attracted much attention since it can leverage consistency and complementary relationships across multiple modalities. However, previous studies mostly focused on the complementary information of multimodal signals, neglecting the consistency information of multimodal signals and the topological structure of each modality. To this end, we propose a dense graph convolution network (DGC) equipped with a joint cross attention (JCA), named DG-JCA, for MER. The main advantage of the DG-JCA model is that it simultaneously integrates the spatial topology, consistency, and complementarity of multimodal data into a unified network framework. Meanwhile, DG-JCA extends the graph convolution network (GCN) via a dense connection strategy and introduces cross attention to joint model well-learned features from multiple modalities. Specifically, we first build a topology graph for each modality and then extract neighborhood features of different modalities using DGC driven by dense connections with multiple layers. Next, JCA performs cross-attention fusion in intra- and intermodality based on each modality’s characteristics while balancing the contributions of various modalities’ features. Finally, subject-dependent and subject-independent experiments on the DEAP and SEED-IV datasets are conducted to evaluate the proposed method. Abundant experimental results show that the proposed model can effectively extract and fuse multimodal features and achieve outstanding performance in comparison with some state-of-the-art approaches.
29 January 2025 in IEEE Transactions on Multimedia
Multi-modal Emotion Recognition (MER) has demonstrated competitive performance in affective computing, owing to synthesizing information from diverse modalities. However, many existing approaches still face unresolved challenges, such as: (i) how to learn compact yet representative features from multi-modal data simultaneously, and (ii) how to address differences among subjects and enhance the generalization of the emotion recognition model, given the diverse nature of individual biological signals. To this end, we propose a Dynamic Interactive Network with Self-Distillation (DISD-Net) for cross-subject MER. The DISD-Net incorporates a dynamin interactive module to capture the intra- and inter-modal interactions from multi-modal data. Additionally, to enhance compactness in modal representations, we leverage the soft labels generated by the DISD-Net model as supplemental training guidance. This involves incorporating self-distillation, aiming to transfer the knowledge that the DISD-Net model contains hard and soft labels to each modality. Finally, domain adaptation (DA) is seamlessly integrated into the dynamic interactive and self-distillation components, forming a unified framework to extract subject-invariant multi-modal emotional features. Experimental results indicate that the proposed model achieves a mean accuracy of 75.00% with a standard deviation of 7.68% for the DEAP dataset and a mean accuracy of 65.65% with a standard deviation of 5.08% for the SEED-IV dataset.