Electroencephalography (EEG) signals have been proven to be one of the most predictive and reliable indicators for estimating driving fatigue state. However, how to make full use of EEG data for driving fatigue detection remains a challenge. Many existing methods include a time-consuming manual process or tedious parameter tunings for feature extraction, which is inconvenient to train and implement. On the other hand, most models ignore or manually determine EEG connectivity features between different channels, thus failing to thoroughly exploit the intrinsic interchannel relations for classification. In this article, we introduce a new attention-based multiscale convolutional neural network-dynamical graph convolutional network (AMCNN-DGCN) model, aiming to conquer these two issues in a unified end-to-end model. AMCNN-DGCN starts with attention-based multiscale temporal convolutions to automatically learn frequency filters to extract the salient pattern from raw EEG data. Subsequently, AMCNN-DGCN uses dynamical graph convolutional networks (DGCNs) to learn spatial filters, in which the adjacency matrix is adaptively determined in a data-driven way to exploit the intrinsic relationship between channels effectively. With the temporal–spatial structure, AMCNN-DGCN can capture highly discriminative features. To verify the effectiveness of AMCNN-DGCN, we conduct a simulated fatigue driving environment to collect EEG signals from 29 healthy subjects (male/female = 17/12 and age = 23.28±2.70 years) through a remote wireless cap with 24 channels. The results demonstrate that our proposed model outperforms six widely used competitive EEG models with high accuracy of 95.65%. Finally, the critical brain regions and connections for driving fatigue detection were investigated through the dynamically learned adjacency matrix.

Deep neural networks have recently been successfully extended to EEG-based driving fatigue detection. Nevertheless, most existing models fail to reveal the intrinsic inter-channel relations that are known to be beneficial for EEG-based classification. Additionally, these models require substantial data for training, which is often impractical due to the high cost of data collection. To simultaneously address these two issues, we propose a Self-Attentive Channel-Connectivity Capsule Network (SACC-CapsNet) for EEG-based driving fatigue detection in this paper. SACC-CapsNet starts with a temporal-channel attention module to investigate the critical temporal information and important channels for driving fatigue detection, refining the input EEG signals. Subsequently, the refined EEG data are transformed into a channel covariance matrix to capture the inter-channel relations, followed by selective kernel attention to extract the highly discriminative channel-connectivity features. Finally, a capsule neural network is employed to effectively learn the relationships between connectivity features, which is more suitable for limited data. To confirm the effectiveness of SACC-CapsNet, we collected 24-channel EEG data from 31 subjects (mean age=23.13±2.68 years, male/female=18/13) in a simulated fatigue driving environment. Extensive experiments were conducted with the acquired data, and the comparison results show that our proposed model outperforms state-of-the-art methods. Additionally, the channel covariance matrix learned from SACC-CapsNet reveals that the frontal pole is most informative for detecting driving fatigue, followed by the parietal and central regions. Intriguingly, the temporal-channel attention module can enhance the significance of these critical regions, and the reconstructed channel covariance matrix generated by the decoder network of SACC-CapsNet can effectively preserve valuable information about them.

A brain-computer interface (BCI), which provides an advanced direct human-machine interaction, has gained substantial research interest in the last decade for its great potential in various applications including rehabilitation and communication. Among them, the P300-based BCI speller is a typical application that is capable of identifying the expected stimulated characters. However, the applicability of the P300 speller is hampered for the low recognition rate partially attributed to the complex spatio-temporal characteristics of the EEG signals. Here, we developed a deep-learning analysis framework named ST-CapsNet to overcome the challenges regarding better P300 detection using a capsule network with both spatial and temporal attention modules. Specifically, we first employed spatial and temporal attention modules to obtain refined EEG signals by capturing event-related information. Then the obtained signals were fed into the capsule network for discriminative feature extraction and P300 detection. In order to quantitatively assess the performance of the proposed ST-CapsNet, two publicly-available datasets (i.e., Dataset IIb of BCI Competition 2003 and Dataset II of BCI Competition III) were applied. A new metric of averaged symbols under repetitions (ASUR) was adopted to evaluate the cumulative effect of symbol recognition under different repetitions. In comparison with several widely-used methods (i.e., LDA, ERP-CapsNet, CNN, MCNN, SWFP, and MsCNN-TL-ESVM), the proposed ST-CapsNet framework significantly outperformed the state-of-the-art methods in terms of ASUR. More interestingly, the absolute values of the spatial filters learned by ST-CapsNet are higher in the parietal lobe and occipital region, which is consistent with the generation mechanism of P300.

Frequency-domain (FD) features reveal the activated patterns of individual local brain regions responding to different emotions, whereas brain connectivity (BC) features involve the coordination of multiple brain regions for generating emotional responses; these two types of features are complementary to each other. To date, the fusion of these two types of features for electroencephalography (EEG)-based cross-subject emotion recognition remains to be fully investigated due to the intersubject variability in EEG signals. In this article, we first attempt to investigate these fused features for cross-subject emotion recognition from multiple perspectives, including critical frequency bands, complementary characteristics for each emotional state, critical channels, and crucial connections, using a fast and robust approximate empirical kernel map-fusion-based support vector machine (AEKM-Fusion-SVM) method. The experimental results on the SJTU emotion EEG dataset (SEED), BCI2020-A, and BCI2020-B datasets reveal that: 1) the AEKM-fusion method improves the effectiveness and efficiency of the fusion of features of different dimensions; 2) the recognition accuracy of the fused features outperforms each individual feature, and this outperformance is more significant in the high-frequency bands (i.e., the beta and gamma bands); 3) the fused features significantly enhance the classification performance for negative emotion; and 4) the fused features built with 27 selected channels achieve comparable performance to that of the fused features built with the full number of channels (i.e., 62 channels), allowing for easier establishment of brain–computer interface (BCI) systems in real-world scenarios. Our study enriches the research of emotion-related brain mechanisms and provides new insight into affective computing.

Brain-computer interface (BCI) is a novel human-computer interaction model, which does not depend on the conventional output pathway (peripheral nerve and muscle tissue). In the past three decades, it has attracted the interest of researchers and gradually become a research hotspot. As a typical BCI application, the brain-controlled wheelchair (BCW) could provide a new communicating channel with the external environment for physically disabled people. However, the main challenge of BCW is how to decode multi-degree of freedom control instruction from electroencephalogram (EEG) as soon as possible. The research progress of BCW has been developed rapidly over the past fifteen years. In this review, we investigate the BCW from multiple perspectives, include the type of signal acquisition, the pattern of commands for the control system and the working mechanism of the control system. Furthermore, we summarize the development trend of BCW based on the previous investigation, and it is mainly manifested in three aspects: from a wet electrode to dry electrode, from single-mode to multi-mode, and from synchronous control to asynchronous control. With the continuous development of BCW, we also find new functions have been introduced into BCW to increase its stability and robustness. It is believed that BCW will be able to enter the real-life from the laboratory and will be widely used in rehabilitation medicine in the future.

Affective brain-computer interfaces (aBCIs) have garnered widespread applications, with remarkable advancements in utilizing electroencephalogram (EEG) technology for emotion recognition. However, the time-consuming process of annotating EEG data, inherent individual differences, non-stationary characteristics of EEG data, and noise artifacts in EEG data collection pose formidable challenges in developing subject-specific cross-session emotion recognition models. To simultaneously address these challenges, we propose a unified pre-training framework based on multi-scale masked autoencoders (MSMAE), which utilizes large-scale unlabeled EEG signals from multiple subjects and sessions to extract noise-robust, subject-invariant, and temporal-invariant features. We subsequently fine-tune the obtained generalized features with only a small amount of labeled data from a specific subject for personalization and enable cross-session emotion recognition. Our framework emphasizes: 1) multi-scale representation to capture diverse aspects of EEG signals, obtaining comprehensive information; 2) an improved masking mechanism for robust channel-level representation learning, addressing missing channel issues while preserving inter-channel relationships; and 3) invariance learning for regional correlations in spatial-level representation, minimizing inter-subject and inter-session variances. Under these elaborate designs, the proposed MSMAE exhibits a remarkable ability to decode emotional states from a different session of EEG data during the testing phase. Extensive experiments conducted on the two publicly available datasets, i.e., SEED and SEED-IV, demonstrate that the proposed MSMAE consistently achieves stable results and outperforms competitive baseline methods in cross-session emotion recognition.