With the continuous development of portable noninvasive human sensor technologies such as brain–computer interfaces (BCI), multimodal emotion recognition has attracted increasing attention in the area of affective computing. This paper primarily discusses the progress of research into multimodal emotion recognition based on BCI and reviews three types of multimodal affective BCI (aBCI): aBCI based on a combination of behavior and brain signals, aBCI based on various hybrid neurophysiology modalities and aBCI based on heterogeneous sensory stimuli. For each type of aBCI, we further review several representative multimodal aBCI systems, including their design principles, paradigms, algorithms, experimental results and corresponding advantages. Finally, we identify several important issues and research directions for multimodal emotion recognition based on BCI.

In addition to helping develop products that aid the disabled, brain–computer interface (BCI) technology can also become a modality of entertainment for all people. However, most BCI games cannot be widely promoted due to the poor control performance or because they easily cause fatigue. In this paper, we propose a P300 brain–computer-interface game (MindGomoku) to explore a feasible and natural way to play games by using electroencephalogram (EEG) signals in a practical environment. The novelty of this research is reflected in integrating the characteristics of game rules and the BCI system when designing BCI games and paradigms. Moreover, a simplified Bayesian convolutional neural network (SBCNN) algorithm is introduced to achieve high accuracy on limited training samples. To prove the reliability of the proposed algorithm and system control, 10 subjects were selected to participate in two online control experiments. The experimental results showed that all subjects successfully completed the game control with an average accuracy of 90.7% and played the MindGomoku an average of more than 11 min. These findings fully demonstrate the stability and effectiveness of the proposed system. This BCI system not only provides a form of entertainment for users, particularly the disabled, but also provides more possibilities for games.

Fatigue is a state commonly caused by overworked, which seriously affects daily work and life. How to detect mental fatigue has always been a hot spot for researchers to explore. Electroencephalogram (EEG) is considered one of the most accurate and objective indicators. This article investigated the development of classification algorithms applied in EEG-based fatigue detection in recent years. According to the different source of the data, we can divide these classification algorithms into two categories, intra-subject (within the same subject) and cross-subject (across different subjects). In most studies, traditional machine learning algorithms with artificial feature extraction methods were commonly used for fatigue detection as intra-subject algorithms. Besides, deep learning algorithms have been applied to fatigue detection and could achieve effective result based on large-scale dataset. However, it is difficult to perform long-term calibration training on the subjects in practical applications. With the lack of large samples, transfer learning algorithms as a cross-subject algorithm could promote the practical application of fatigue detection methods. We found that the research based on deep learning and transfer learning has gradually increased in recent years. But as a field with increasing requirements, researchers still need to continue to explore efficient decoding algorithms, design effective experimental paradigms, and collect and accumulate valid standard data, to achieve fast and accurate fatigue detection methods or systems to further widely apply.

As an effective brain signal recording technique for neuroscience, magnetoencephalography (MEG) is widely used in cognitive research. However, due to the low signal-to-noise ratio and the structural or functional variabilities of MEG signals between different subjects, conventional methods perform poorly in decoding human brain responds. Inspired by deep recurrent network for processing sequential data, we applied the gated recurrent units for MEG signals processing. In the paper, we proposed a hybrid gated recurrent network (HGRN) for inter-subject visual MEG decoding. Without the need of any information from test subjects, the HGRN effectively distinguished MEG signals evoked by different visual stimulations, face and scrambled face. In the leave-one-out cross-validation experiments on sixteen subjects, our method achieved better performance than many existing methods. For more in-depth analysis, HGRN can be utilized to extract spatial features and temporal features of MEG signals. These features conformed to the previous cognitive studies which demonstrated the practicality of our method for MEG signal processing. Consequently, the proposed model can be considered as a new tool for decoding and analyzing brain MEG signal, which is significant for visual cognitive research in neuroscience.

Background: Brain computer interface (BCI) technology may provide a new way of communication for some patients with disorder of consciousness (DOC), which can directly connect the brain and external devices. However, the DOC patients’ EEG differ significantly from that of the normal person and are difficult to collected, the decoding algorithm currently only is trained based on a small amount of the patient’s own data and performs poorly.Methods: In this study, a decoding algorithm called WD-ADSTCN based on domain adaptation is proposed to improve the DOC patients’ P300 signal detection. We used the Wasserstein distance to filter the normal population data to increase the training data. Furthermore, an adversarial approach is adopted to resolve the differences between the normal and patient data.Results: The results showed that in the cross-subject P300 detection of DOC patients, 7 of 11 patients achieved an average accuracy of over 70%. Furthermore, their clinical diagnosis changed and CRS-R scores improved three months after the experiment.Conclusion: These results demonstrated that the proposed method could be employed in the P300 BCI system for the DOC patients, which has important implications for the clinical diagnosis and prognosis of these patients.