Combing brain-computer interfaces (BCI) and virtual reality (VR) is a novel technique in the field of medical rehabilitation and game entertainment. However, the limitations of BCI such as a limited number of action commands and low accuracy hinder the widespread use of BCI-VR. Recent studies have used hybrid BCIs that combine multiple BCI paradigms and/or the multi-modal biosensors to alleviate these issues, which may become the mainstream of BCIs in the future. The main purpose of this review is to discuss the current status of multi-modal BCI-VR. This study first reviewed the development of the BCI-VR, and explored the advantages and disadvantages of incorporating eye tracking, motion capture, and myoelectric sensing into the BCI-VR system. Then, this study discussed the development trend of the multi-modal BCI-VR, hoping to provide a pathway for further research in this field.

The convolutional neural network (CNN) model is an active research topic in the field of EEG signals analysis. However, the classification effect of CNN on EEG signals of amnestic mild cognitive impairment (aMCI) with type 2 diabetes mellitus (T2DM) is not ideal. Even if EEG signals are transformed into multispectral images that are more closely matched with the model, the best classification performance can not be achieved. Therefore, to improve the performance of CNN toward EEG multispectral image classification, a multi-view convolutional neural network (MVCNN) classification model based on inceptionV1 is designed in this study. This model mainly improves and optimizes the convolutional layers and stochastic gradient descent (SGD) in the convolutional architecture model. Firstly, based on the discreteness of EEG multispectral image features, the multi-view convolutional layer structure was proposed. Then the learning rate change function of the SGD was optimized to increase the classification performance. The multi-view convolutional nerve was used in an EEG multispectral classification task involving 19 aMCI with T2DM and 20 normal controls. The results showed that compared with the traditional classification models, MVCNN had a better stability and accuracy. Therefore, MVCNN could be used as an effective feature classification method for aMCI with T2DM.

The Study of Evaluation and Rehabilitation of Patients With Different Cognitive Impairment Phases Based on Virtual Reality and EEG

In this study, a multi-scale high-density convolutional neural network (MHCNN) classification method for spatial cognitive ability assessment was proposed, aiming at achieving the binary classification of task-state EEG signals before and after spatial cognitive training. Besides, the multi-dimensional conditional mutual information method was used to extract the frequency band features of the EEG data. And the coupling features under the combination of multi-frequency bands were transformed into multi-spectral images. At the same time, the idea of Densenet was introduced to improve the multi-scale convolutional neural network. Firstly, according to the discreteness of multispectral EEG image features, two-scale convolution kernels were used to calculate and learn useful channel and frequency band feature information in multispectral image data. Secondly, to enhance feature propagation and reduce the number of parameters, the dense network was connected after the multi-scale convolutional network, and the learning rate change function of the stochastic gradient descent algorithm was optimized to objectively evaluate the training effect. The experimental results showed that compared with the classical convolution neural network (CNN) and multi-scale convolution neural network, the proposed MHCNN had better classification performance in the six frequency band combinations with the highest accuracy of 98%: Theta-Alpha2-Gamma, Alpha2-Beta2-Gamma, Beta1-Beta2-Gamma, Theta-Beta2-Gamma, Theta- Alpha1-Gamma, and Alpha1-Alpha2-Gamma. By comparing the classification results of six frequency band combinations, it was found that the combination of the Theta-Beta2-Gamma band had the best classification effect. The MHCNN classification method proposed in this research could be used as an effective biological indicator of spatial cognitive training effect and could be extended to other brain function evaluations.