Detecting driver fatigue is critical for ensuring traffic safety. Electroencephalography (EEG) is the golden standard for brain activity measurement and is considered a good indicator of detecting driver fatigue. However, the current driver fatigue detection algorithm has limitations in mining and fusing the spatiotemporal characteristics of EEG signals. In this paper, we propose a multi-branch deep learning network named spatio-temporal fusion network with brain region partitioning strategy (STFN-BRPS) to improve the accuracy and robustness of driver fatigue recognition. Initially, we develop a recurrent multi-scale convolution module (RMSCM) comprising a multi-scale convolution sub-module, a CNN-Bi-LSTM sub-module, and a residual structure branch. RMSCM effectively extracts highly discriminative long short-term temporal feature information. Secondly, we propose a dynamic graph convolution module and a spatial graph edges’ importance weight assignment method based on brain region partitioning strategy, which can acquire intrinsic spatial feature information between electrodes. Thirdly, we design a feature fusion module (FFM) that utilizes channel attention to fuse long short-term temporal and spatial features. FFM learns and prioritizes the significance and relevance of each channel in the fused features. Finally, the fused spatio-temporal features are passed into the classification module to obtain the predicted driver fatigue state. Extensive comparison and ablation studies are conducted on EEG signals collected from real-world driving scenarios. The results demonstrate that the proposed STFN-BRPS model delivers superior classification performance compared to the mainstream methods. This study establishes a benchmark for EEG-based driver fatigue detection and related deep-learning modeling work.

Electroencephalography (EEG)-based emotion recognition is essential for medical assistance and human-computer interaction. Although deep learning-based emotion recognition methods have demonstrated high performance, several challenges remain: 1) How to effectively utilize the complex dynamic-static spatial patterns inherent in emotion-related EEG signals. 2) How to hierarchically learn the latent correlations among multi-domain features. To address these challenges, a spatio-temporal representation fusion learning network (STRFLNet) is proposed to improve both the accuracy and robustness of emotion recognition using EEG signals. Specifically, dynamic-static graph topologies are constructed to capture comprehensive brain functional connectivity states, and a continuous dynamic-static graph ordinary differential equations is introduce to reveal continuous spatial patterns within EEG signals. Additionally, a hierarchical transformer fusion module is developed to fully leverage the latent correlations among multi-domain features to obtain the fused spatio-temporal representation. The experimental results on the SEED, SEED-IV, and DREAMER public EEG emotion datasets, under both subject-independent and subject-dependent settings, demonstrate that STRFLNet outperforms state-of-the-art methods in emotion recognition tasks. We further validate the effectiveness of the proposed model through interpretability analysis, which reveals the associations between the activated brain regions and corresponding emotional states. Our work highlights the significance of continuous spatial pattern learning and spatio-temporal feature fusion in emotion recognition, providing new insights for EEG-based emotion modeling.

Effectively integrating the time-space-frequency information of multi-modal signals from armband sensor, including surface electromyogram (sEMG) and accelerometer data, is critical for accurate gesture recognition. Existing approaches often neglect the abundant spatial relationships inherent in multi-channel sEMG signals obtained via armband sensors and face challenges in harnessing the correlations across multiple feature domains. To address this issue, we propose a novel multi-feature fusion network with spatial partitioning strategy and cross-attention (MFN-SPSCA) to improve the accuracy and robustness of gesture recognition. Specifically, a spatiotemporal graph convolution module with a spatial partitioning strategy is designed to capture potential spatial feature of multi-channel sEMG signals. Additionally, we design a cross-attention fusion module to learn and prioritize the importance and correlation of multi-feature domain. Extensive experiment demonstrate that the MFN-SPSCA method outperforms other state-of-the-art methods on self-collected dataset and the Ninapro DB5 dataset. Our work addresses the challenge of recognizing gestures from the multi-modal data collected by armband sensor, emphasizing the importance of integrating time-space-frequency information. Codes are available at https://github.com/ZJUTofBrainIntelligence/MFN-SPSCA.

Objective. Brain-computer interface (BCI) technology based on motor imagery (MI) control has become a research hotspot but continues to encounter numerous challenges. BCI can assist in the recovery of stroke patients and serve as a key technology in robot control. Current research on MI almost exclusively focuses on the hands, feet, and tongue. Therefore, the purpose of this paper is to establish a four-class MI BCI system, in which the four types are the four articulations within the right upper limbs, involving the shoulder, elbow, wrist, and hand. Approach. Ten subjects were chosen to perform nine upper-limb analytic movements, after which the differences were compared in P300, movement-related potentials(MRPS), and event-related desynchronization/event-related synchronization under voluntary MI (V-MI) and involuntary MI (INV-MI). Next, the cross-frequency coupling (CFC) coefficient based on mutual information was extracted from the electrodes and frequency bands with interest. Combined with the image Fourier transform and twin bounded support vector machine classifier, four kinds of electroencephalography data were classified, and the classifier’s parameters were optimized using a genetic algorithm. Main results. The results were shown to be encouraging, with an average accuracy of 93.2% and 92.2% for V-MI and INV-MI, respectively, and over 95% for any three classes and any two classes. In most cases, the accuracy of feature extraction using the proximal articulations as the basis was found to be relatively high and had better performance. Significance. This paper discussed four types of MI according to three aspects under two modes and classed them by combining graph Fourier transform and CFC. Accordingly, the theoretical discussion and classification methods may provide a fundamental theoretical basis for BCI interface applications.