Non-invasive, electroencephalography (EEG)-based brain-computer interfaces (BCIs) on motor imagery movements translate the subject’s motor intention into control signals through classifying the EEG patterns caused by different imagination tasks, e.g., hand movements. This type of BCI has been widely studied and used as an alternative mode of communication and environmental control for disabled patients, such as those suffering from a brainstem stroke or a spinal cord injury (SCI). Notwithstanding the success of traditional machine learning methods in classifying EEG signals, these methods still rely on hand-crafted features. The extraction of such features is a difficult task due to the high non-stationarity of EEG signals, which is a major cause by the stagnating progress in classification performance. Remarkable advances in deep learning methods allow end-to-end learning without any feature engineering, which could benefit BCI motor imagery applications. We developed three deep learning models: (1) A long short-term memory (LSTM); (2) a spectrogram-based convolutional neural network model (CNN); and (3) a recurrent convolutional neural network (RCNN), for decoding motor imagery movements directly from raw EEG signals without (any manual) feature engineering. Results were evaluated on our own publicly available, EEG data collected from 20 subjects and on an existing dataset known as 2b EEG dataset from “BCI Competition IV”. Overall, better classification performance was achieved with deep learning models compared to state-of-the art machine learning techniques, which could chart a route ahead for developing new robust techniques for EEG signal decoding. We underpin this point by demonstrating the successful real-time control of a robotic arm using our CNN based BCI.

Objective . The objective of this work is to use the capability of spiking neural networks to capture the spatio-temporal information encoded in time-series signals and decode them without the use of hand-crafted features and vector-based learning and the realization of the spiking model on low-power neuromorphic hardware. Approach . The NeuCube spiking model was used to classify different grasp movements directly from raw surface electromyography signals (sEMG), the estimations of the applied finger forces as well as the classification of two motor imagery movements from raw electroencephalography (EEG). In a parallel investigation, the designed spiking decoder was implemented on SpiNNaker neuromorphic hardware, which allows low-energy real-time processing. Main results . Experimental results reveal a better classification accuracy using the NeuCube model compared to traditional machine learning methods. For sEMG classification, we reached a training accuracy of 85% and a test accuracy of 84.8%, as well as less than 19% of relative root mean square error (rRMSE) when estimating finger forces from six subjects. For the EEG classification, a mean accuracy of 75% was obtained when tested on raw EEG data from nine subjects from the existing 2b dataset from ‘BCI competition IV’. Significance . This work provides a proof of concept for a successful implementation of the NeuCube spiking model on the SpiNNaker neuromorphic platform for raw sEMG and EEG decoding, which could chart a route ahead for a new generation of portable closed-loop and low-power neuroprostheses.

Objective . The objective of this work is to present gumpy , a new free and open source Python toolbox designed for hybrid brain–computer interface (BCI). Approach . Gumpy provides state-of-the-art algorithms and includes a rich selection of signal processing methods that have been employed by the BCI community over the last 20 years. In addition, a wide range of classification methods that span from classical machine learning algorithms to deep neural network models are provided. Gumpy can be used for both EEG and EMG biosignal analysis, visualization, real-time streaming and decoding. Results . The usage of the toolbox was demonstrated through two different offline example studies, namely movement prediction from EEG motor imagery, and the decoding of natural grasp movements with the applied finger forces from surface EMG (sEMG) signals. Additionally, gumpy was used for real-time control of a robot arm using steady-state visually evoked potentials (SSVEP) as well as for real-time prosthetic hand control using sEMG. Overall, obtained results with the gumpy toolbox are comparable or better than previously reported results on the same datasets. Significance . Gumpy is a free and open source software, which allows end-users to perform online hybrid BCIs and provides different techniques for processing and decoding of EEG and EMG signals. More importantly, the achieved results reveal that gumpy ’s deep learning toolbox can match or outperform the state-of-the-art in terms of accuracy. This can therefore enable BCI researchers to develop more robust decoding algorithms using novel techniques and hence chart a route ahead for new BCI improvements.

In recent times, we have witnessed a push towards restoring sensory perception to upper-limb amputees, which includes the whole spectrum from gentle touch to noxious stimuli. These are essential components for body protection as well as for restoring the sense of embodiment. Notwithstanding the considerable advances that have been made in designing suitable sensors and restoring tactile perceptions, pain perception dynamics and its decoding using effective bio-markers, are still not fully understood. Here, using electroencephalography (EEG) recordings, we identified and validated a spatio-temporal signature of brain activity during innocuous, moderately more intense, and noxious stimulation of an amputee’s phantom limb using transcutaneous nerve stimulation (TENS). Based on the spatio-temporal EEG features, we developed a system for detecting pain perception and reaction in the brain, which successfully classified three different stimulation conditions with a test accuracy of 94.66%, and we investigated the cortical activity in response to sensory stimuli in these conditions. Our findings suggest that the noxious stimulation activates the pre-motor cortex with the highest activation shown in the central cortex (Cz electrode) between 450 ms and 750 ms post-stimulation, whereas the highest activation for the moderately intense stimulation was found in the parietal lobe (P2, P4, and P6 electrodes). Further, we localized the cortical sources and observed early strong activation of the anterior cingulate cortex (ACC) corresponding to the noxious stimulus condition. Moreover, activation of the posterior cingulate cortex (PCC) was observed during the noxious sensation. Overall, although this is a single case study, this work presents a novel approach and a first attempt to analyze and classify neural activity when restoring sensory perception to amputees, which could chart a route ahead for designing a real-time pain reaction system in upper-limb prostheses.

Objective. A major challenge for controlling a prosthetic arm is communication between the device and the user’s phantom limb. We show the ability to enhance phantom limb perception and improve movement decoding through targeted transcutaneous electrical nerve stimulation in individuals with an arm amputation. Approach. Transcutaneous nerve stimulation experiments were performed with four participants with arm amputation to map phantom limb perception. We measured myoelectric signals during phantom hand movements before and after participants received sensory stimulation. Using electroencephalogram (EEG) monitoring, we measured the neural activity in sensorimotor regions during phantom movements and stimulation. In one participant, we also tracked sensory mapping over 2 years and movement decoding performance over 1 year. Main results. Results show improvements in the participants’ ability to perceive and move the phantom hand as a result of sensory stimulation, which leads to improved movement decoding. In the extended study with one participant, we found that sensory mapping remains stable over 2 years. Sensory stimulation improves within-day movement decoding while performance remains stable over 1 year. From the EEG, we observed cortical correlates of sensorimotor integration and increased motor-related neural activity as a result of enhanced phantom limb perception. Significance. This work demonstrates that phantom limb perception influences prosthesis control and can benefit from targeted nerve stimulation. These findings have implications for improving prosthesis usability and function due to a heightened sense of the phantom hand.

Understanding the human brain’s perception of different thermal sensations has sparked the interest of many neuroscientists. The identification of distinct brain patterns when processing thermal stimuli has several clinical applications, such as phantom-limb pain prediction, as well as increasing the sense of embodiment when interacting with neurorehabilitation devices. Notwithstanding the remarkable number of studies that have touched upon this research topic, understanding how the human brain processes different thermal stimuli has remained elusive. More importantly, very intense thermal stimuli perception dynamics, their related cortical activations, as well as their decoding using effective features are still not fully understood. In this study, using electroencephalography (EEG) recorded from three healthy human subjects, we identified spatial, temporal, and spectral patterns of brain responses to different thermal stimulations ranging from extremely cold and hot stimuli (very intense), moderately cold and hot stimuli (intense), to a warm stimulus (innocuous). Our results show that very intense thermal stimuli elicit a decrease in alpha power compared to intense and innocuous stimulations. Spatio-temporal analysis reveals that in the first 400 ms post-stimulus, brain activity increases in the prefrontal and central brain areas for very intense stimulations, whereas for intense stimulation, high activity of the parietal area was observed post-500 ms. Based on these identified EEG patterns, we successfully classified the different thermal stimulations with an average test accuracy of 84% across all subjects. En route to understanding the underlying cortical activity, we source localized the EEG signal for each of the five thermal stimuli conditions. Our findings reveal that very intense stimuli were anticipated and induced early activation (before 400 ms) of the anterior cingulate cortex (ACC). Moreover, activation of the pre-frontal cortex, somatosensory, central, and parietal areas, was observed in the first 400 ms post-stimulation for very intense conditions and starting 500 ms post-stimuli for intense conditions. Overall, despite the small sample size, this work presents novel findings and a first comprehensive approach to explore, analyze, and classify EEG-brain activity changes evoked by five different thermal stimuli, which could lead to a better understanding of thermal stimuli processing in the brain and could, therefore, pave the way for developing a real-time withdrawal reaction system when interacting with prosthetic limbs. We underpin this last point by benchmarking our EEG results with a demonstration of a real-time withdrawal reaction of a robotic prosthesis using a human-like artificial skin.