Legged robotic technologies have moved out of the lab to operate in real environments, characterized by a wide variety of unpredictable irregularities and disturbances, all this in close proximity with humans. Demonstrating the ability of current robots to move robustly and reliably in these conditions is becoming essential to prove their safe operation. Here, we report an in-depth literature review aimed at verifying the existence of common or agreed protocols and metrics to test the performance of legged system in realistic environments. We primarily focused on three types of robotic technologies, i.e., hexapods, quadrupeds and bipeds. We also included a comprehensive overview on human locomotion studies, being it often considered the gold standard for performance, and one of the most important sources of bioinspiration for legged machines. We discovered that very few papers have rigorously studied robotic locomotion under irregular terrain conditions. On the contrary, numerous studies have addressed this problem on human gait, being nonetheless of highly heterogeneous nature in terms of experimental design. This lack of agreed methodology makes it challenging for the community to properly assess, compare and predict the performance of existing legged systems in real environments. On the one hand, this work provides a library of methods, metrics and experimental protocols, with a critical analysis on the limitations of the current approaches and future promising directions. On the other hand, it demonstrates the existence of an important lack of benchmarks in the literature, and the possibility of bridging different disciplines, e.g., the human and robotic, towards the definition of standardized procedures that will boost not only the scientific development of better bioinspired solutions, but also their market uptake.

One important point in the development of a brain-machine Interface (BMI) commanding an exoskeleton is the assessment of the cognitive engagement of the subject during the motor imagery tasks conducted. However, there are not many databases that provide electroencephalography (EEG) data during the use of a lower-limb exoskeleton. The current paper presents a database designed with an experimental protocol aiming to assess not only motor imagery during the control of the device, but also the attention to gait on flat and inclined surfaces. The research was conducted as an EUROBENCH subproject in the facilities sited in Hospital Los Madroños, Brunete (Madrid). The data validation reaches accuracies over 70% in the assessment of motor imagery and attention to gait, which marks the present database as a valuable resource for researches interested on developing and testing new EEG-based BMIs.

Background: In neurorehabilitation, we are witnessing a growing awareness of the importance of standardized quantitative assessment of limb functions. Detailed assessments of the sensorimotor deficits following neurological disorders are crucial. So far, this assessment has relied mainly on clinical scales, which showed several drawbacks. Different technologies could provide more objective and repeatable measurements. However, the current literature lacks practical guidelines for this purpose. Nowadays, the integration of available metrics, protocols, and algorithms into one harmonized benchmarking ecosystem for clinical and research practice is necessary.Methods: This work presents a benchmarking framework for upper limb capacity. The scheme resulted from a multidisciplinary and iterative discussion among several partners with previous experience in benchmarking methodology, robotics, and clinical neurorehabilitation. We merged previous knowledge in benchmarking methodologies for human locomotion and direct clinical and engineering experience in upper limb rehabilitation. The scheme was designed to enable an instrumented evaluation of arm capacity and to assess the effectiveness of rehabilitative interventions with high reproducibility and resolution. It includes four elements: (1) a taxonomy for motor skills and abilities, (2) a list of performance indicators, (3) a list of required sensor modalities, and (4) a set of reproducible experimental protocols.Results: We proposed sixmotor primitivesas building blocks of most upper-limb daily-life activities and combined them into a set of functionalmotor skills. We identified the main aspects to be considered during clinical evaluation, and grouped them into tenmotor abilitiescategories.For each ability, we proposed a set ofperformance indicatorsto quantify the proposed ability on a quantitative and high-resolution scale. Finally, we defined the procedures to be followed to perform the benchmarking assessment in a reproducible and reliable way, including the definition of the kinematic models and the target muscles.Conclusions: This work represents the first unified scheme for the benchmarking of upper limb capacity. To reach a consensus, this scheme should be validated with real experiments across clinical conditions and motor skills. This validation phase is expected to create a shared database of human performance, necessary to have realistic comparisons of treatments and drive the development of new personalized technologies.

Objective: The objective of this study is to verify the reliability and the concurrent and discriminant validity of the measurements of spasticity offered by the robotic device, quantifying the (1) test–retest reliability, (2) correlation with the clinical evaluation using the Modified Ashworth Scale (MAS), (3) inter-rater reliability between the two physiotherapists, and (4) ability to discriminate between healthy and stroke patients.Methods: A total of 20 stroke patients and 20 healthy volunteers participated in the study. Two physical therapists (PT1 and PT2) independently evaluated the hand spasticity of stroke subjects using the MAS. Spasticity was assessed, both in healthy and stroke patients, with the Amadeo device at three increasing velocities of passive movement for three consecutive repeated assessments, while raw data of force and position were collected through an external program.Data analysis: The intraclass correlation coefficient (ICC) and the weighted kappa were computed to estimate the reliability of the Amadeo device measurements, the inter-rater reliability of MAS, and the correlation between the MAS and Amadeo device measurements. The discriminant ability of the Amadeo device was assessed by comparing the stroke and healthy subjects' spasticity measurements with the percentage of agreements with 0 in MAS for healthy subjects.Results: The test–retest reliability of the Amadeo device was high with ICC at all three velocities (ICC = 0.908, 0.958, and 0.964, respectively) but lower if analyzed with weighted kappa correlation (0.584, 0.748, and 0.749, respectively) as mean values for each velocity. The correlation between Amadeo and the clinical scale for stroke patients with weighted kappa correlation was poor (0.280 ± 0.212 for PT1 and 0.290 ± 0.155 for PT2). The inter-rater reliability of the clinical MAS was high (ICC = 0.911).Conclusion: Both MAS and Amadeo spasticity scores showed good reliability. The Amadeo scores did not show a strong clinical correlation with the MAS in stroke patients. Hitherto, Amadeo evaluation shows trends that are consistent with the characteristics of spasticity, such as an increase in spasticity as the speed of muscle stretching increases. The ability of the device to discriminate between stroke patients and healthy controls is low. Future studies adopting an instrumental gold standard for spasticity may provide further insight into the validity of these measurements.

The increasing application of TMS in research and therapy has spawned an ever-growing number of commercial and non-commercial TMS devices and technology development. New CE-marked devices appear at a rate of approximately one every two years, with new FDA-approved application of TMS occurring at a similar rate. With the resulting complex landscape of TMS devices and their application, accessible information about the technological characteristics of the TMS devices, such as the type of their circuitry, their pulse characteristics, or permitted protocols would be beneficial. We here present an overview and open access database summarizing key features and applications of available commercial and non-commercial TMS devices (http://www.tmsbase.info). This may guide comparison and decision making about the use of these devices. A bibliometric analysis was performed by identifying commercial and non-commercial TMS devices from which a comprehensive database was created summarizing their publicly available characteristics, both from a technical and clinical point of view. In this document, we introduce both the commercial devices and prototypes found in the literature. The technical specifications that unify these devices are briefly analysed in two separate tables: power electronics, waveform, protocols, and coil types. In the prototype TMS systems, the proposed innovations are focused on improving the treatment regarding the patient: noise cancellation, controllable parameters, and multiple stimulation. This analysis shows that the landscape of TMS is becoming increasingly fragmented, with new devices appearing ever more frequently. The review provided here can support development of benchmarking frameworks and comparison between TMS systems, inform the choice of TMS platforms for specific research and therapeutic applications, and guide future technology development for neuromodulation devices. This standardisation strategy will allow a better end-user choice, with an impact on the TMS manufacturing industry and a homogenisation of patient samples in multi-centre clinical studies. As an open access repository, we envisage the database to grow along with the dynamic development of TMS devices and applications through community-lead curation.