Taste sensation recognition is a keystone for taste-related Brain-Computer Interface (BCI). A commonly used measurement of brain activity in response to specific stimulation is through electroencephalography (EEG) signals. However, it remains challenging to develop accurate and generalizable EEG-based measurement for human taste sensation. This paper proposes EEG-MSRNet, a novel fully convolutional neural network for EEG-based classification of basic taste sensations (blank, sour, sweet, bitter, salty, umami). Firstly, a multi-scale temporal convolution operation with residual learning is designed to extract features in different frequencies from the down-sampled EEG signals. Subsequently, a multi-scale spatial convolution operation represents the features in a cross-channel manner. Finally, a convolutional layer and global average pooling (GAP) layer are introduced to make predictions with the feature representation instead of the commonly used fully connected layers for classification. An experimental procedure is developed to acquire the EEG signals under taste stimulation. Comparison experiments and ablation studies have proved the stable and generalizable recognition performance of EEG-MSRNet on our self-collected EEG dataset. The results suggest that our EEG-based system with EEG-MSRNet is effective and generalizable for taste sensation recognition, which provides a powerful measurement for taste-related BCI such as taste disorder diagnosis and virtual taste.

Silent speech decoding (SSD), based on articulatory neuromuscular activities, has become a prevalent task of brain–computer interfaces (BCIs) in recent years. Many works have been devoted to decoding surface electromyography (sEMG) from articulatory neuromuscular activities. However, restoring silent speech in tonal languages such as Mandarin Chinese is still difficult. This paper proposes an optimized sequence-to-sequence (Seq2Seq) approach to synthesize voice from the sEMG-based silent speech. We extract duration information to regulate the sEMG-based silent speech using the audio length. Then, we provide a deep-learning model with an encoder–decoder structure and a state-of-the-art vocoder to generate the audio waveform. Experiments based on six Mandarin Chinese speakers demonstrate that the proposed model can successfully decode silent speech in Mandarin Chinese and achieve a character error rate (CER) of 6.41% on average with human evaluation.

Based on surface electromyography (sEMG), a novel recognition method to distinguish six types of human primary taste sensations was developed, and the recognition accuracy was 74.46%. The sEMG signals were acquired under the stimuli of no taste substance, distilled vinegar, white granulated sugar, instant coffee powder, refined salt, and Ajinomoto. Then, signals were preprocessed with the following steps: sample augments, removal of trend items, high-pass filter, and adaptive power frequency notch. Signals were classified with random forest and the classifier gave a five-fold cross-validation accuracy of 74.46%, which manifested the feasibility of the recognition task. To further improve the model performance, we explored the impact of feature dimension, electrode distribution, and subject diversity. Accordingly, we provided an optimized feature combination that reduced the number of feature types from 21 to 4, a preferable selection of electrode positions that reduced the number of channels from 6 to 4, and an analysis of the relation between subject diversity and model performance. This study provides guidance for further research on taste sensation recognition with sEMG.

Human taste sensation can be qualitatively described with surface electromyography. However, the pattern recognition models trained on one subject (the source domain) do not generalize well on other subjects (the target domain). To improve the generalizability and transferability of taste sensation models developed with sEMG data, two methods were innovatively applied in this study: domain regularized component analysis (DRCA) and conformal prediction with shrunken centroids (CPSC). The effectiveness of these two methods was investigated independently in an unlabeled data augmentation process with the unlabeled data from the target domain, and the same crossuser adaptation pipeline was conducted on six subjects. The results show that DRCA improved the classification accuracy on six subjects (p < 0.05), compared with the baseline models trained only with the source domain data, while CPSC did not guarantee the accuracy improvement. Furthermore, the combination of DRCA and CPSC presented statistically significant improvement (p < 0.05) in classification accuracy on six subjects. The proposed strategy of combining DRCA and CPSC showed its effectiveness in addressing the cross-user data distribution drift in sEMGbased taste sensation recognition application. It also shows the potential for more cross-user adaptation applications.

Silent Speech Interface (SSI) based on neuromuscular signals has become popular in Human-Computer Interaction. Articulatory neuromuscular activity can be captured with surface electromyography (sEMG). However, in the absence of acoustic sounds, it remains challenging to convert the articulatory neuromuscular signals of silent speakers into corresponding audio. This paper proposes a Mandarin-based SSI, a novel converter between articulatory neuromuscular from silent speakers and audio signals, by providing an auxiliary set of sEMG and audio pairs from vocal speakers. The proposed method is based on a hybrid framework combining convolutional neural networks and Transformer to utilize both global and local information. We experimentally validate the feasibility of the proposed method by obtaining an average objective character error rate (CER) of 10.69% using an automatic speech recognition (ASR) evaluation tool with four silent speakers. The results show that our Mandarin-based SSI with vocal speaker assistance facilitates the conversion from sEMG into audio in the cross-subject scenario.