Due to neurovascular coupling, an inherent emotional complementarity is observed between electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS); however, these modalities exhibit significant cross-modal spatiotemporal and individual differences in emotional expression. Furthermore, there is an inherent negative coupling between oxygenated hemoglobin and deoxygenated hemoglobin in fNIRS, which can lead to inconsistent results when models perform emotion prediction based on both. To address these challenges, this paper proposes the EEG-fNIRS spatiotemporal alignment network (ENSTAN), which comprises a dynamic multigraph attention mechanism, temporally aligned cross-attention, and a subspace source-target emotion alignment module. Given the limited availability of publicly accessible EEG-fNIRS emotion datasets, we validated the proposed ENSTAN on the self-constructed ENTER dataset and compared it with various baseline methods. The results demonstrate that ENSTAN achieved average accuracies of 91.97% in random cross-validation and 88.96% in trialunit cross-validation for subject-dependent experiments, with a 73.24% average accuracy in subject-independent experiments, confirming its effectiveness. In addition, the proposed ENSTAN elucidates the salient interaction channels and response latencies between EEG and fNIRS.
Background/Objectives: Driven by the neurovascular coupling mechanism, electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) capture shared underlying neural activity from electrophysiological and hemodynamic perspectives and thus complement each other. Emotional processing relies on functional synergy across brain regions recorded via EEG and fNIRS channels. Functionally similar channels are co-activated and adjacent in functional space. Nevertheless, existing EEG–fNIRS emotion recognition methods characterize inter-channel synergy using physical distances or graph structures. They fail to quantify channel-relative positions in functional space and capture synergies between functionally similar channels. Furthermore, these methods overlook dynamic modality contributions during fusion, rendering predictions susceptible to low-quality signals. Methods: To tackle these limitations, we propose the Geometric Positional Encoding and Dynamic Weight Fusion Network (GDWF-Net). The EEG branch adopts functional-space geometric distances as positional cues and embeds them into attention via rotary positional encoding to model inter-channel functional synergy. The fNIRS branch extracts spatiotemporal features using lightweight modules. The fusion module computes dynamic weights from modality contributions, suppresses noisy low-quality modalities via negative learning, and models cross-modal interactions within a mixed-curvature space. Conclusions: Evaluated on the ENTER dataset, GDWF-Net achieves accuracies of 81.63% and 65.71% in subject-dependent and cross-subject settings, surpassing state-of-the-art baselines. Ablation experiments and visualizations verify the effectiveness and interpretability of each proposed module.
Background/Objectives: Hybrid EEG-fNIRS emotion recognition aims to accurately identify an individual’s emotional state by analyzing neurophysiological signals and constitutes an important research direction in affective brain–computer interfaces and human–computer interaction. In recent years, EEG-fNIRS emotion recognition has advanced from handcrafted feature extraction and shallow fusion to deep learning and graph-based modeling. However, most existing methods rely on predefined fixed frequency-band partitioning and second-order graph structures that only support pairwise connections, making it difficult to accommodate inter-subject frequency variability and to characterize high-order brain network relationships such as multi-channel synergistic activation within a frequency band and cross-frequency coupling. Methods: To address these issues, this paper proposes an EEG-fNIRS emotion recognition framework based on a Spatial-Frequency Hypergraph Neural Network (SF-HGNN). First, a Dynamic Frequency Band Decomposition module is designed to achieve adaptive optimization of the EEG and fNIRS frequency bands; second, a Multi-scale Temporal Convolution module extracts temporal features at different time scales; third, a Spatial-Frequency Adaptive Hypergraph Convolution module is constructed to model intra-band cross-channel spatial synergy and channel-wise cross-frequency coupling; and finally, a Cross-Modal Attention Fusion mechanism achieves high-order interaction between the complementary information of the two modalities. The proposed method was validated on the public ENTER dataset comprising 50 participants and four emotion categories (sadness, happiness, fear, and calm). Results: Experimental results show that SF-HGNN achieves accuracies of 82.94% and 68.85% in subject-dependent and subject-independent experiments, respectively; ablation studies and visualization analyses further verify the effectiveness of each module and the interpretability of the model. Conclusions: Future work will focus on validation with larger-scale data and improving cross-subject domain generalization.