One of the challenges in motor imagery (MI) classification tasks is finding an easy-handled electroencephalogram (EEG) representation method which can preserve not only temporal features but also spatial ones. To fully utilize the features on various dimensions of EEG, a novel MI classification framework is first introduced in this paper, including a new 3D representation of EEG, a multi-branch 3D convolutional neural network (3D CNN) and the corresponding classification strategy. The 3D representation is generated by transforming EEG signals into a sequence of 2D array which preserves spatial distribution of sampling electrodes. The multibranch 3D CNN and classification strategy are designed accordingly for the 3D representation. Experimental evaluation reveals that the proposed framework reaches state-of-the-art classification kappa value level and significantly outperforms other algorithms by 50% decrease in standard deviation of different subjects, which shows good performance and excellent robustness on different subjects. The framework also shows great performance with only nine sampling electrodes, which can significantly enhance its practicality. Moreover, the multi-branch structure exhibits its low latency and a strong ability in mitigating overfitting issues which often occur in MI classification because of the small training dataset.

Throat cancer treatment involves surgical removal of the tumor, leaving patients with facial disfigurement as well as temporary or permanent loss of voice. Surface electromyography (sEMG) generated from the jaw contains lots of voice information. However, it is difficult to record because of not only the weakness of the signals but also the steep skin curvature. This paper demonstrates the design of an imperceptible, flexible epidermal sEMG tattoo-like patch with the thickness of less than 10 μm and peeling strength of larger than 1 N cm⁻¹ that exhibits large adhesiveness to complex biological surfaces and is thus capable of sEMG recording for silent speech recognition. When a tester speaks silently, the patch shows excellent performance in recording the sEMG signals from three muscle channels and recognizing those frequently used instructions with high accuracy by using the wavelet decomposition and pattern recognization. The average accuracy of action instructions can reach up to 89.04%, and the average accuracy of emotion instructions is as high as 92.33%. To demonstrate the functionality of tattoo-like patches as a new human–machine interface (HMI) for patients with loss of voice, the intelligent silent speech recognition, voice synthesis, and virtual interaction have been implemented, which are of great importance in helping these patients communicate with people and make life more enjoyable.

The internal availability of silent speech serves as a translator for people with aphasia and keeps human–machine/human interactions working under various disturbances. This paper develops a silent speech strategy to achieve all-weather, natural interactions. The strategy requires few usage specialized skills like sign language but accurately transfers high-capacity information in complicated and changeable daily environments. In the strategy, the tattoo-like electronics imperceptibly attached on facial skin record high-quality bio-data of various silent speech, and the machine-learning algorithm deployed on the cloud recognizes accurately the silent speech and reduces the weight of the wireless acquisition module. A series of experiments show that the silent speech recognition system (SSRS) can enduringly comply with large deformation (~45%) of faces by virtue of the electricity-preferred tattoo-like electrodes and recognize up to 110 words covering daily vocabularies with a high average accuracy of 92.64% simply by use of small-sample machine learning. We successfully apply the SSRS to 1-day routine life, including daily greeting, running, dining, manipulating industrial robots in deafening noise, and expressing in darkness, which shows great promotion in real-world applications.

While deep learning algorithms significantly improves the decoding performance of brain-computer interface (BCI) based on electroencephalogram (EEG) signals, the performance relies on a large number of high-resolution data for training. However, collecting sufficient usable EEG data is difficult due to the heavy burden on the subjects and the high experimental cost. To overcome this data insufficiency, a novel auxiliary synthesis framework is first introduced in this paper, which composes of a pre-trained auxiliary decoding model and a generative model. The framework learns the latent feature distributions of real data and uses Gaussian noise to synthesize artificial data. The experimental evaluation reveals that the proposed method effectively preserves the time-frequency-spatial features of the real data and enhances the classification performance of the model using limited training data and is easy to implement, which outperforms the common data augmentation methods. The average accuracy of the decoding model designed in this work is improved by (4.72±0.98)% on the BCI competition IV 2a dataset. Furthermore, the framework is applicable to other deep learning-based decoders. The finding provides a novel way to generate artificial signals for enhancing classification performance when there are insufficient data, thus reducing data acquisition consuming in the BCI field.