Advances in machine learning technologies in recent years have facilitated developments in autonomous robotic systems. Designing these autonomous systems typically requires manually specified models of the robotic system and world when using classical control-based strategies, or time consuming and computationally expensive data-driven training when using learning-based strategies. Combination of classical control and learning-based strategies may mitigate both requirements. However, the performance of the combined control system is not obvious given that there are two separate controllers. This paper focuses on one such combination, which uses gravity-compensation together with reinforcement learning (RL). We present a study of the effects of gravity compensation on the performance of two reinforcement learning algorithms when solving reaching tasks using a simulated seven-degree-of-freedom robotic arm. The results of our study demonstrate that gravity compensation coupled with RL can reduce the training required in reaching tasks involving elevated target locations, but not all target locations.

Creating flexible and robust brain machine interfaces (BMIs) is currently a popular topic of research that has been explored for decades in medicine, engineering, commercial, and machine-learning communities. In particular, the use of techniques using reinforcement learning (RL) has demonstrated impressive results but is under-represented in the BMI community. To shine more light on this promising relationship, this article aims to provide an exhaustive review of RL’s applications to BMIs. Our primary focus in this review is to provide a technical summary of various algorithms used in RL-based BMIs to decode neural intention, without emphasizing preprocessing techniques on the neural signals and reward modeling for RL. We first organize the literature based on the type of RL methods used for neural decoding, and then each algorithm’s learning strategy is explained along with its application in BMIs. A comparative analysis highlighting the similarities and uniqueness among neural decoders is provided. Finally, we end this review with a discussion about the current stage of RLBMIs including their limitations and promising directions for future research.

Brain machine interfaces (BMIs) offer great potential to improve the quality of life for individuals with neurological disorders or severe motor impairments. Among various neural recording modalities, electroencephalogram (EEG) is particularly favorable for BMIs due to its noninvasive nature, portability, and high temporal resolution. Existing EEG datasets for BMIs are often limited to experimental settings that fail to address subjects’ freewill in decision making. We present a large EEG dataset, containing a total of 6808 trials, recorded from 23 healthy young adults (eight females and 15 males with an age range from 18 to 24 years) while performing reaching and grasping tasks, where the target object is freely chosen at their desired pace according to their own will. This EEG dataset provides a realistic representation of reaching and grasping movement, making it useful for developing practical BMIs.

Decoding movement related intentions is a key step to implement BMIs. Decoding EEG has been challenging due to its low spatial resolution and signal to noise ratio. Metric learning allows finding a representation of data in a way that captures a desired notion of similarity between data points. In this study, we investigate how metric learning can help finding a representation of the data to efficiently classify EEG movement and pre-movement intentions. We evaluate the effectiveness of the obtained representation by comparing classification the performance of a Support Vector Machine (SVM) as a classifier when trained on the original representation, called Euclidean, and representations obtained with three different metric learning algorithms, including Conditional Entropy Metric Learning (CEML), Neighborhood Component Analysis (NCA), and the Entropy Gap Metric Learning (EGML) algorithms. We examine different types of features, such as time and frequency components, which input to the metric learning algorithm, and both linear and non-linear SVM are applied to compare the classification accuracies on a publicly available EEG data set for two subjects (Subject B and C). Although metric learning algorithms do not increase the classification accuracies, their interpretability using an importance measure we define here, helps understanding data organization and how much each EEG channel contributes to the classification. In addition, among the metric learning algorithms we investigated, EGML shows the most robust performance due to its ability to compensate for differences in scale and correlations among variables. Furthermore, from the observed variations of the importance maps on the scalp and the classification accuracy, selecting an appropriate feature such as clipping the frequency range has a significant effect on the outcome of metric learning and subsequent classification. In our case, reducing the range of the frequency components to 0–5 Hz shows the best interpretability in both Subject B and C and classification accuracy for Subject C. Our experiments support potential benefits of using metric learning algorithms by providing visual explanation of the data projections that explain the inter class separations, using importance. This visualizes the contribution of features that can be related to brain function.

Simultaneous electroencephalogram and functional magnetic resonance imaging (EEG-fMRI) is a unique combined technique that provides synergy in the understanding and localization of seizure onset in epilepsy. However, reported experimental protocols for EEG-fMRI recordings fail to address details about conducting such procedures on epilepsy patients. In addition, these protocols are limited solely to research settings. To fill the gap between patient monitoring in an epilepsy monitoring unit (EMU) and conducting research with an epilepsy patient, we introduce a unique EEG-fMRI recording protocol of epilepsy during the interictal period. The use of an MR conditional electrode set, which can also be used in the EMU for a simultaneous scalp EEG and video recording, allows an easy transition of EEG recordings from the EMU to the scanning room for concurrent EEG-fMRI recordings. Details on the recording procedures using this specific MR conditional electrode set are provided. In addition, the study explains step-by-step EEG processing procedures to remove the imaging artifacts, which can then be used for clinical review. This experimental protocol promotes an amendment to the conventional EEG-fMRI recording for enhanced applicability in both clinical (i.e., EMU) and research settings. Furthermore, this protocol provides the potential to expand this modality to postictal EEG-fMRI recordings in the clinical setting.