Manipulation skill learning and generalisation have gained increasing attention due to the wide applications of robot manipulators and the spurt of robot learning techniques. Especially, the learning from demonstration method has been exploited widely and successfully in the robotic community, and it is regarded as a promising direction to realise the manipulation skill learning and generalisation. In addition to the learning techniques, the immersive teleoperation enables the human to operate a remote robot with an intuitive interface and achieve the telepresence. Thus, it is a promising way to transfer manipulation skills from humans to robots by combining the learning methods and teleoperation, and adapting the learned skills to different tasks in new situations. This review, therefore, aims to provide an overview of immersive teleoperation for skill learning and generalisation to deal with complex manipulation tasks. To this end, the key technologies, for example, manipulation skill learning, multimodal interfacing for teleoperation and telerobotic control, are introduced. Then, an overview is given in terms of the most important applications of immersive teleoperation platform for robot skill learning. Finally, this survey discusses the remaining open challenges and promising research topics.

Tele-echography has emerged as a promising and effective solution, leveraging the expertise of sonographers and the autonomy of robots to perform ultrasound scanning for patients residing in remote areas, without the need for in-person visits by the sonographer. Designing effective and natural human-robot interfaces for tele-echography remains challenging, with patient safety being a critical concern. In this article, we develop a teleoperation system for robot-assisted sonography with two different interfaces, a haptic device-based interface and a low-cost 3D Mouse-based interface, which can achieve continuous and intuitive telemanipulation by a leader device with a small workspace. To achieve compliant interaction with patients, we design impedance controllers in Cartesian space to track the desired position and orientation for these two teleoperation interfaces. We also propose comprehensive evaluation metrics of robot-assisted sonography, including subjective and objective evaluation, to evaluate tele-echography interfaces and control performance. We evaluate the ergonomic performance based on the estimated muscle fatigue and the acquired ultrasound image quality. We conduct user studies based on the NASA Task Load Index to evaluate the performance of these two human-robot interfaces. The tracking performance and the quantitative comparison of these two teleoperation interfaces are conducted by the Franka Emika Panda robot. The results and findings provide guidance on human-robot collaboration design and implementation for robot-assisted sonography. Note to Practitioners —Robot-assisted sonography has demonstrated efficacy in medical diagnosis during clinical trials. However, deploying fully autonomous robots for ultrasound scanning remains challenging due to various constraints in practice, such as patient safety, dynamic tasks, and environmental uncertainties. Semi-autonomous or teleoperation-based robot sonography represents a promising approach for practical deployment. Previous work has produced various expensive teleoperation interfaces but lacks user studies to guide teleoperation interface selection. In this article, we present two typical teleoperation interfaces and implement a continuous and intuitive teleoperation control system. We also propose a comprehensive evaluation metric for assessing their performance. Our findings show that the haptic device outperforms the 3D Mouse, based on operators’ feedback and acquired image quality. However, the haptic device requires more learning time and effort in the training stage. Furthermore, the developed teleoperation system offers a solution for shared control and human-robot skill transfer. Our results provide valuable guidance for designing and implementing human-robot interfaces for robot-assisted sonography in practice.

Learning a stable autonomous dynamic system (ADS) encoding human motion rules has been shown as an effective way for demonstration learning. However, the stability guarantee may sacrifice the demonstration learning accuracy. This article solves the issue by learning a stability certificate, represented by a neural energy function, on the demonstration set. We propose a polarlike space analysis approach to derive parameter constraints to guarantee the unique-minimum property of the neural energy function, which is essential for it to be a cogent stability certificate. Then, the neural energy function is learned to capture the demonstration preferences via constrained optimization algorithms. With the learned neural energy function, a globally asymptotically stable ADS with predefined position constraint is further formulated. We also quantitatively analyze the generalization ability of the learned ADS by utilizing the substantial flexibility of the neural energy function. The effectiveness of the proposed approach is validated on the LASA dataset and two representative robotic experiments.

Human-robot interaction (HRI) is a crucial component in the field of robotics, and enabling faster response, higher accuracy, as well as smaller human effort, is essential to improve the efficiency, robustness, and applicability of HRI-driven tasks. In this article, we develop a novel neuroadaptive admittance control with human motion intention (HMI) estimation and output error constraint for natural and stable interaction. First, the interaction force information of the robot is utilized to predict the HMI and the stiffness in the admittance model is dynamically updated based on surface electromyography (sEMG) signals of the human upper limb to achieve human-like compliance. Then, based on the designed error transformation mechanism, an innovative prescribed performance control (PPC) is proposed that allows the trajectory error to converge to the given constraint range within a predefined time for any bounded initial conditions, thus enabling the robot to maintain a comprehensive performance of moving in the desired direction as guided by the human. Also, an adaptive neural network (NN) is employed to compensate for the uncertainty of robotics systems to improve the tracking accuracy further. According to the Lyapunov stability analysis criterion, our approach ensures that all states of the closed-loop system remain globally uniformly ultimately bounded. Finally, a series of real-world robot experiments demonstrate the effectiveness of the proposed framework.

This article presents a novel shared-control teleoperation framework that integrates imitation learning and bilateral control to achieve system stability based on a new dynamic movement primitives (DMPs) observer. First, a DMPs-based observer is first created to capture human operational skills through offline human demonstrations. The learning results are then used to predict human action intention in teleoperation. Compared with other observers, the DMPs-based observer incorporates human operational features and can predict long-term actions with minor errors. A high-gain observer is established to monitor the robot’s status in real time on the leader side. Subsequently, two controllers on both the follower and leader sides are constructed based on the outputs of the observers. The follower controller shares control authorities to address accidents in real-time and correct prediction errors of the observation using delayed leader commands. The leader controller minimizes position-tracking errors through force feedback. The convergence of the predictions of the DMPs-based observer under the time delays and teleoperation system stability are proved by building two Lyapunov functions. Finally, two groups of comparative experiments are conducted to verify the advantages over other methods and the effectiveness of the proposed framework in motion prediction with time delays and obstacle avoidance.

In this paper, a distributed observer-based prescribed performance control method is proposed for using a multi-robot teleoperation system to manipulate a common deformable object. To achieve a stable position-tracking effect and realize the desired cooperative operational performance, we first define a new hybrid error matrix for both the relative distances and absolute positions of robots and then decompose the matrix into two new error terms for cooperative and independent robot control. Then, we improve the Kelvin-Voigt (K-V) contact model based on the new error terms. Because the center position and deformation of the object cannot be measured, the object dynamics are then expressed by the relative distances of robots and an equivalent impedance term. Each robot incorporates an observer to estimate contact force and object dynamics based on its own measurements. To address the position errors caused by biases in force estimation and realize the position-tracking effect of each robot, we improve the barrier Lyapunov functions (BLFs) by incorporating the errors into system control. which allows us to achieve a predefined position-tracking effect. We conduct an experiment to verify the proposed controller’s ability in a dual-telerobot cooperative manipulation task, even when the object is subjected to unknown disturbances. Note to Practitioners —This article is inspired by the limitations of multi-telerobot manipulation with a deformable object, where the deformation of the object cannot be measured directly. Meanwhile, force sensors, especially 6-axis force sensors, are very expensive. To realize the purpose that objects manipulated by multiple robots match the same state as operated on the leader side, we propose an object-centric teleoperation framework based on the estimates of contact forces and object dynamics and the improved barrier Lyapunov functions (BLFs). This framework contributes to two aspects in practice: 1) propose a control diagram for deformable object co-teleoperation of multi-robots for unmeasurable object’s centre position and deformation; 2) propose an improved BLFs controller based on the estimation of contact force and robot dynamics. The estimation errors are considered and transferred using an equivalent impedance to be integrated into the Lyapunov function to minimize both force and motion-tracking errors. The experimental results verify the effectiveness of the proposed method. The developed framework can be used in industrial applications with a similar scenario.