Silent speech decoding is a novel application of the Brain–Computer Interface (BCI) based on articulatory neuromuscular activities, reducing difficulties in data acquirement and processing. In this paper, spatial features and decoders that can be used to recognize the neuromuscular signals are investigated. Surface electromyography (sEMG) data are recorded from human subjects in mimed speech situations. Specifically, we propose to utilize transfer learning and deep learning methods by transforming the sEMG data into spectrograms that contain abundant information in time and frequency domains and are regarded as channel-interactive. For transfer learning, a pre-trained model of Xception on the large image dataset is used for feature generation. Three deep learning methods, Multi-Layer Perception, Convolutional Neural Network and bidirectional Long Short-Term Memory, are then trained using the extracted features and evaluated for recognizing the articulatory muscles’ movements in our word set. The proposed decoders successfully recognized the silent speech and bidirectional Long Short-Term Memory achieved the best accuracy of 90%, outperforming the other two algorithms. Experimental results demonstrate the validity of spectrogram features and deep learning algorithms.

Taste is a key sense that helps identify different food types and most of this work was carried out on primary tastes rather than generating different flavors. In this work, we proposed a plan to create other flavors rather than primary tastes, adjusted the electrical (40–180 µA) and thermal stimulation (20–38 °C and 38–20 °C), and revealed the digital coding for multi-flavors. Our results showed that different combinations of digital coding could generate different flavors and that tastes related to different stimuli are easy to develop. The novelty of this work is to design other types of flavors and primary tastes. The experimental results demonstrated that the novel method proposed for digital taste coding could realize primary tastes (sweet, sour, salty, spicy, and mint) and mixed flavors. Furthermore, some innovative sensations have been realized, which are sprite, soda water, sweet-sour, salty-sweet, and salty-mint sensations. We presume that this innovation could digitally enhance various flavors.

Taste sensation recognition is a keystone for taste-related Brain-Computer Interface (BCI). A commonly used measurement of brain activity in response to specific stimulation is through electroencephalography (EEG) signals. However, it remains challenging to develop accurate and generalizable EEG-based measurement for human taste sensation. This paper proposes EEG-MSRNet, a novel fully convolutional neural network for EEG-based classification of basic taste sensations (blank, sour, sweet, bitter, salty, umami). Firstly, a multi-scale temporal convolution operation with residual learning is designed to extract features in different frequencies from the down-sampled EEG signals. Subsequently, a multi-scale spatial convolution operation represents the features in a cross-channel manner. Finally, a convolutional layer and global average pooling (GAP) layer are introduced to make predictions with the feature representation instead of the commonly used fully connected layers for classification. An experimental procedure is developed to acquire the EEG signals under taste stimulation. Comparison experiments and ablation studies have proved the stable and generalizable recognition performance of EEG-MSRNet on our self-collected EEG dataset. The results suggest that our EEG-based system with EEG-MSRNet is effective and generalizable for taste sensation recognition, which provides a powerful measurement for taste-related BCI such as taste disorder diagnosis and virtual taste.

Silent speech decoding (SSD), based on articulatory neuromuscular activities, has become a prevalent task of brain–computer interfaces (BCIs) in recent years. Many works have been devoted to decoding surface electromyography (sEMG) from articulatory neuromuscular activities. However, restoring silent speech in tonal languages such as Mandarin Chinese is still difficult. This paper proposes an optimized sequence-to-sequence (Seq2Seq) approach to synthesize voice from the sEMG-based silent speech. We extract duration information to regulate the sEMG-based silent speech using the audio length. Then, we provide a deep-learning model with an encoder–decoder structure and a state-of-the-art vocoder to generate the audio waveform. Experiments based on six Mandarin Chinese speakers demonstrate that the proposed model can successfully decode silent speech in Mandarin Chinese and achieve a character error rate (CER) of 6.41% on average with human evaluation.

Based on surface electromyography (sEMG), a novel recognition method to distinguish six types of human primary taste sensations was developed, and the recognition accuracy was 74.46%. The sEMG signals were acquired under the stimuli of no taste substance, distilled vinegar, white granulated sugar, instant coffee powder, refined salt, and Ajinomoto. Then, signals were preprocessed with the following steps: sample augments, removal of trend items, high-pass filter, and adaptive power frequency notch. Signals were classified with random forest and the classifier gave a five-fold cross-validation accuracy of 74.46%, which manifested the feasibility of the recognition task. To further improve the model performance, we explored the impact of feature dimension, electrode distribution, and subject diversity. Accordingly, we provided an optimized feature combination that reduced the number of feature types from 21 to 4, a preferable selection of electrode positions that reduced the number of channels from 6 to 4, and an analysis of the relation between subject diversity and model performance. This study provides guidance for further research on taste sensation recognition with sEMG.