It is crucial to increase the independence of severely disabled individuals. Assistive robotics can aid in the desired activities of daily living, including tasks requiring remote performance e.g. grasping remote objects, turning switches on/off, and opening/closing doors. The robot control is compromised by the lack of efficient interfaces for individuals with disabilities and the lack of depth perception. This paper addresses these challenges by presenting the development and evaluation of efficient tongue-based robot interfaces and low-level robot control schemes targeting tele-robotic control through a 2D display. Ten able-bodied participants were successful in completing ten rounds of controlling a JACO robot to perform a pouring water task, using five different control methods, under 2D or 3D visual feedback. The tool-frame based tongue interface layout, L2_TF (with emulated joystick, mode switch button and a “GO” button) improved the 2D visual guided control of the JACO robot compared with the other tongue control methods. The mean trajectory length of completing the task using L2_TF was 3% longer compared with the standard joystick when controlling through 2D. The trajectory length for reaching and grabbing a bottle was shortest for L2_TF compared with all other control methods, including the joystick. The iTongue control layouts performed well in gripping time, showing no significant difference between 2D and 3D. The transition from 2D to 3D resulted in a mean decrease of 27.7% for task completion time across all interfaces. L2_TF and the joystick had the strongest and most similar robustness to the transition between 3D and 2D.

Increasing the independence of individuals with tetraplegia is a challenging task. One potential solution is to allow for use of an assistive robotic manipulator (ARM), when solving tasks in personal and remote space. There is a lack in available control interfaces that are suitable for severely disabled individuals. The aim of this paper is twofold: to allow for remote tongue-based control of an ARM and, to study the effect of semiautomation when compared with full manual control of an ARM. Ten able-bodied individuals participated in a two-day experiment where they were asked to drive a wheelchair mounted ARM away from the participant and out of sight. Thereafter, they should either pick up a strawberry or a bottle from a table. All the participants successfully finished three trials for three different control methods: 1) manual control (MA), 2) adaptive level semiautomation (SA), and 3) fixed level semi-automation (FA). The data was analyzed using repeated measures analysis of variance. When grasping the strawberry, there was a significant decrease in the gripping time (60.16±9.13 vs. 90.62±10.06, p = 0.012) and number of used commands (0.73±0.07 vs. 0.86±0.09, p = 0.03) when using FA compared to MA. When grasping the bottle, the SA showed a significant reduction in gripping time (32.30±3.09 vs. 66.15±9.77, p = 0.022) and number of used commands (0.63±0.05 vs. 0.85±0.09, p<0.001) compared with MA. This paper is a step in the direction of providing severely paralyzed individuals with a way to increase their independence and overall quality of life

Stable grasping is essential for assistive robots aiding individuals with severe motor–sensory disabilities in their everyday lives. Slip detection can prevent unstably grasped objects from falling out of the gripper and causing accidents. Recent research on slip detection focuses on tactile sensing; however, not every robot arm can be equipped with such sensors. In this paper, we propose a slip detection method solely based on data collected by a RealSense D435 Red Green Blue-Depth (RGBd) camera. By utilizing Farneback optical flow (OF) to estimate the motion field of the grasped object relative to the gripper, while also removing potential background noise, the algorithm can perform in a multitude of environments. The algorithm was evaluated on a dataset of 28 daily objects that were lifted 30 times each, resulting in a total of 840 frame sequences. Our proposed slip detection method achieves an accuracy of up to 82.38% and a recall of up to 87.14%, which is comparable to state-of-the-art approaches when only using camera data. When excluding objects for which movements are challenging for vision-based methods to detect, such as untextured or transparent objects, the proposed method performs even better, with an accuracy of up to 87.19% and a recall of up to 95.09%.

Preprint submitted to IOP Journal of Neural Engineering Objective. Individuals with Amyotrophic lateral sclerosis (ALS) progressively lose muscle functionality and therefore experience both an increased need for assistive robot technologies and a reduced ability to control such robots. While these individuals may use high-performing control systems, such as tongue control, at the beginning of their disease progression, they will eventually be restricted to a lower-performing control system, such as brain control. However, an adaptive multimodal control interface framework consisting of combinations of tongue control and noninvasive brain control can utilize the residual tongue functionality to optimize the control performance throughout the disease progression. Approach. To investigate this concept, a new adaptive tongue-brain multimodal control framework for manual and continuous control of a 7-degree-of-freedom robot arm is developed, based on a prior validation study. The new framework focuses on improved visual feedback, as individuals with ALS specifically requested this in a previous validation study, and consists of four subsystems: the first uses full tongue control; the second and third use hybrid tongue and noninvasive brain control, with a decreasing need for tongue functionality; and the fourth uses noninvasive brain control only. The framework was evaluated with three participants with ALS. Main results. All participants were succesful with all subsystems. One user could no longer efficiently use the full tongue control interface but achieved good results with the third and fourth subsystems. The second participant achieved significantly better results with the subsystems that included some tongue control, thereby showing the advantage of including some tongue control. The third participant performed well with all subsystems showing the ideal performance progression between each subsystem. Moreover, all participants, including the two with good tongue control, chose a multimodal control interface as their favorite. Significance. The results indicate that individuals with ALS prefer interfaces that combine multiple control modalities.

Increasing the independence of individuals with tetraplegia is a challenging task. One potential solution is to allow for use of assistive robotics, specifically an assistive robotic manipulator (ARM), when solving varying tasks in personal and remote space. Allowing for remote performances has a positive influence on the independence of the user, allowing the user to be self-sufficient even when lying in bed. The control interfaces that are suitable for severely disabled individuals is lacking. The aim of this paper is twofold: to allow for remote tongue-based control of an ARM and second, to compare the effect of semi-automation on the control of a robotic ARM. Ten able-bodied individuals participated in a two-day experiment where they were asked to drive a wheelchair mounted ARM away from the participant and out of sight. Thereafter, they should either pick up a strawberry or a bottle from a table. All the participants successfully finished three trials for three different control methods: 1) manual control (MA), 2) adaptive level semi-automation (SA), and 3) fixed level semi-automation (FA). When grasping the strawberry, there was a significant decrease in the gripping time and number of used commands when using FA compared to MA. When grasping the bottle, the SA showed a significant reduction in gripping time and number of used commands compared with MA. This paper is a step in the direction of providing severely paralyzed individuals with a way to increase their independence and overall quality of life.

The objective of this paper was to for the first time demonstrate and evaluate a remotely tongue-controlled wheelchair mounted assistive robotic manipulator (ARM) with three individuals with cervical spinal cord injury. On three days they remotely tonguecontrolled the wheelchair and the ARM (WMARM) to complete two activities of daily living (ADL): Driving the wheelchair and ARM to a remote setting to 1) pick up a bottle and 2) pick up a ball. The participants controlled the system using full manual control by tongue and through semi-automation. Finally, the participants answered a NASA Task Load Index (TLX) questionnaire and a semistructured interview. Results: All participants were able to remotely control the WMARM by tongue. Semi-automation resulted in shorter task completion time, gripping time and fewer commands as compared with manual control. Moreover, semi-automation decreased the measured mental load in the NASA TLX questionnaire by an average of 57%. The participants rated satisfaction with the system high. All the participants thought that the biggest advantage to the system was the potential for performing ADLs on their own and thereby increasing their self-supportiveness. They agreed that automation made the control easier, and that the system would benefit from increased environmental information, e.g. by adding a rear-view camera. Conclusion: It was possible for the users with tetraplegia to control the wheelchair with the ARM using their tongue to perform ADL in Wi-Fi based remote setting. This proposed system has the potential to increase independence and social interaction of individuals with tetraplegia.