Robots are expanding from industrial applications to daily life, in areas such as medical robotics, rehabilitative robotics, social robotics, and mobile/aerial robotics systems. In recent years, augmented reality (AR) has been integrated into many robotic applications, including medical, industrial, human–robot interactions, and collaboration scenarios. In this work, AR for both medical and industrial robot applications is reviewed and summarized. For medical robot applications, we investigated the integration of AR in (1) preoperative and surgical task planning; (2) image-guided robotic surgery; (3) surgical training and simulation; and (4) telesurgery. AR for industrial scenarios is reviewed in (1) human–robot interactions and collaborations; (2) path planning and task allocation; (3) training and simulation; and (4) teleoperation control/assistance. In addition, the limitations and challenges are discussed. Overall, this article serves as a valuable resource for working in the field of AR and robotic research, offering insights into the recent state of the art and prospects for improvement.
23 December 2020 in IEEE Transactions on Industrial Electronics
Mobile medical robots have been widely used in various structured scenarios, such as hospital drug delivery, public area disinfection, and medical examinations. Considering the challenge of environment modeling and controller design, how to achieve the information from the human demonstration in a structured environment directly arouse our interests. Learning skills is a powerful way that can reduce the complexity of algorithm in searching space. This is especially true when naturally acquiring new skills, as mobile medical robot must learn from the interaction with a human being or the environment with limited programming effort. In this article, a learning scheme with nonlinear model predictive control (NMPC) is proposed for mobile robot path tracking. The learning-by-imitation system consists of two levels of hierarchy: in the first level, a multivirtual spring-dampers system is presented for imitation of the mobile robot's trajectories; and in the second level, the NMPC method is used in the motion control system. The NMPC strategy utilizes a varying-parameter one-layer projection neural network to solve an online quadratic programming optimization via iteration over a limited receding horizon. The proposed algorithm is evaluated on a mobile medical robot with an emulated trajectory in simulation and three scenarios used in the experiment.
1 March 2024 in IEEE Transactions on Instrumentation and Measurement
This work presents an optimization-based variable impedance control strategy for controlling a robotic manipulator in medical contact tasks. Specifically, the optimal robot stiffness for performing the medical contact task is obtained using online Quadratic Programming (QP). In the meantime, an energy tank approach is incorporated into the control loop to regulate the system’s passivity. To verify the performance of the proposed strategy, experiments are conducted on both “ static ” and “ scanning ” medical contact tasks, utilizing materials with different properties, different magnitudes of contact forces, as well as uneven conditions with a human torso phantom model and slope surface. The maximum Root Mean Square Error (RMSE) of force tracking with the proposed method in the “ static ” and “ scanning ” tasks , across all setups, is 0.88 N and 0.5 N, respectively. The experiment results demonstrate the superiority of the proposed control strategy compared with traditional manual contact and constant stiffness impedance control-based ones. The proposed control framework is promising to be integrated into robot-assisted medical contact tasks, for example, the palpation and Ultrasound imaging scenarios.
This work presents a compliant and passive shared control framework for teleoperated robot-assisted tasks. Inspired by the human operator's capability of continuously regulating the arm impedance to perform contact-rich tasks, a novel control schema, exploiting the variable impedance control framework for force tracking is proposed. Moreover, bilateral teleoperation and shared control strategies are implemented to alleviate the human operator's workload. Furthermore, a global energy tank-based approach is integrated to enforce the system's passivity. The proposed framework is first evaluated to assess the force-tracking capability when the robot autonomously performs contact-rich tasks, e.g., in an ultrasound scanning scenario. Then, a validation experiment is conducted utilizing the proposed shared control framework. Finally, the system's usability is investigated with 12 users. The experiment results in system assessment revealed a maximum median error of 0.25 N across all the force-tracking experiment setups, i.e., constant and time-varying ones. Then, the validation experiment demonstrated significant improvements regarding the force tracking tasks compared to conventional control methods, and the system passivity was preserved during the task execution. Finally, the usability experiment shows that the human operator workload is significantly reduced by 54.6% compared to the other two control modalities. The proposed framework holds significant potential for the execution of remote robot-assisted medical procedures, such as palpation and ultrasound scanning, particularly in addressing deformation challenges while ensuring safety, compliance, and system passivity.
16 February 2024 in IEEE Robotics & Automation Magazine
During percutaneous nephrolithotomy (PCNL) operations, the surgeon is required to define the incision point on the patient’s back, align the needle to a preplanned path, and perform puncture operations afterward. The procedure is currently performed manually using ultrasound or fluoroscopy imaging for needle orientation, which, however, implies limited accuracy and low reproducibility. This work incorporates augmented reality (AR) visualization with an optical see-through head-mounted display (OST-HMD) and human–robot collaboration (HRC) framework to empower the surgeon’s task completion performance. In detail, eye-to-hand calibration, system registration, and hologram model registration are performed to realize visual guidance. A Cartesian impedance controller is used to guide the operator during the needle-puncture task execution. Experiments are conducted to verify the system performance compared with conventional manual puncture procedures and a 2D monitor-based visualization interface. The results showed that the proposed framework achieves the lowest median and standard deviation error of 3.17 ± 1.36 mm in translation and 1.2 ± .9° in orientation across all of the experimental groups, respectively. Furthermore, the National Aeronautics and Space Administration task load index (NASA-TLX) user evaluation results indicate that the proposed framework requires the lowest workload score of 42.5 ± 13.7 for task completion compared to other experimental setups. The proposed framework exhibits significant potential for clinical application in the PCNL task, as it enhances the surgeon’s perception capability, facilitates collision-free needle insertion path planning, and minimizes errors in task completion.
Bionic hands have been employed in a wide range of applications, including prosthetics, robotic grasping, and human–robot interaction. However, considering the underactuated and nonlinear characteristics, as well as the mechanical structure’s backlash, achieving natural and intuitive teleoperation control of an underactuated bionic hand remains a critical issue. In this paper, the teleoperation control of an underactuated bionic hand using wearable and vision-tracking system-based methods is investigated. Firstly, the nonlinear behaviour of the bionic hand is observed and the kinematics model is formulated. Then, the wearable-glove-based and the vision-tracking-based teleoperation control frameworks are implemented, respectively. Furthermore, experiments are conducted to demonstrate the feasibility and performance of these two methods in terms of accuracy in both static and dynamic scenarios. Finally, a user study and demonstration experiments are conducted to verify the performance of these two approaches in grasp tasks. Both developed systems proved to be exploitable in both powered and precise grasp tasks using the underactuated bionic hand, with a success rate of and , respectively. The glove-based method turned out to be more accurate and better performing than the vision-based one, but also less comfortable, requiring greater effort by the user. By further incorporating a robot manipulator, the system can be utilised to perform grasp, delivery, or handover tasks in daily, risky, and infectious scenarios.