Surgical robots capable of autonomously performing various tasks could enhance efficiency and augment human productivity in addressing clinical needs. Although current solutions have automated specific actions within defined contexts, they are challenging to generalize across diverse environments in general surgery. Embodied intelligence enables general-purpose robot learning with applications for daily tasks, yet its application in the medical domain remains limited. We introduced an open-source surgical embodied intelligence simulator for an interactive environment to develop reinforcement learning methods for minimally invasive surgical robots. Using such embodied artificial intelligence, this study further addresses surgical task automation, enabling zero-shot transfer of simulation-trained policies to real-world scenarios. The proposed method encompasses visual parsing, a perceptual regressor, policy learning, and a visual servoing controller, forming a paradigm that combines the advantages of data-driven policy and classic controller. The visual parsing uses stereo depth estimation and image segmentation with a visual foundation model to handle complex scenes. Experiments demonstrated autonomy in seven game-based skill training tasks on the da Vinci Research Kit, with a proof-of-concept study on haptic-assisted skill training as a practical application. Moreover, we conducted automation of five surgical assistive tasks with the Sentire surgical system on ex vivo animal tissues with various scenes, object sizes, instrument types, and illuminations. The learned policies were also validated in a live-animal trial for three tasks in dynamic in vivo surgical environments. We hope this open-source infrastructure, coupled with a general-purpose learning paradigm, will inspire and facilitate future research on embodied intelligence toward autonomous surgical robots.
6 February 2024 in IEEE Robotics and Automation Letters
Deformable linear object (DLO) manipulation in constrained environments with obstacles has received limited investigations due to DLO's complex intrinsic deformation. In this study, we focus on addressing constrained DLO manipulation problems, especially in the context of avoiding cluttered environment obstacles. Unlike sampling-based planners, which struggle with the high-dimensional state space or require modifications to ensure DLO's kinematic feasibility, we propose a novel obstacle avoidance approach by combining a learning-based predictive control method and an efficient control-theoretic technique. Specifically, we utilize a learning-based model predictive control (MPC) strategy with an attention-based global deformation model to generate low-dimensional reference actions that inherently align with DLO's physics. The attention-based model outperforms multilayer perceptron and bi-directional long short-term memory models by capturing contextual relationships among feature points on DLOs. To mitigate the inevitable modeling errors, a safety-critical filter is designed based on the control barrier function (CBF) principle. An online local linear model is employed in the filter to steer clear of obstacles in close proximity. The proposed approach was validated with extensive simulations and physical experiments on constrained DLO manipulation tasks.
Microscale robotics has emerged as a transformative field, offering unparalleled opportunities for innovation and advancement in various fields. Owing to the distinctive benefits of wireless operation and a heightened level of safety, magnetic actuation has emerged as a widely adopted technique in the field of microrobotics. However, factors such as Brownian motion, fluid dynamic flows, and various nonlinear forces introduce uncertainties in the motion of micro/nanoscale robots, making it challenging to achieve precise navigational control in complex environments. This paper presents an extensive review encompassing the trajectory from theoretical foundations of the generation and modeling of magnetic fields as well as magnetic field-actuation modeling to motion control methods of magnetic microrobots. We introduce traditional control methods and the learning-based control approaches for robotic systems at the micro/nanoscale, and then these methods are compared. Unlike the conventional navigation methods based on precise mathematical models, the learning-based control and navigation approaches can directly learn control signals for the actuation systems from data and without relying on precise models. This endows the micro/nanorobots with high adaptability to dynamic and complex environments whose models are difficult/impossible to obtain. We hope that this review can provide insights and guidance for researchers interested in automated magnetic microrobot control.
15 July 2019 in IEEE Robotics and Automation Letters
In this letter, we propose a unified framework to address underactuated orientation control problems of the tailed robot in flight phase through a 2 DoFs actuated tail. We approach the underactuated control problem from the perspective of an inverse problem of an over-constrained Jacobian matrix. The resulting non-trivial null space can nullify the kinematics control by attracting the system into itself. For this reason, we propose a novel and intuitive null space avoidance control that can always drive the system to states that is perpendicular to its null space, while reaching the desired orientation. Unlike the classic feedforward trajectory tracking approach for nonholonomic systems, the null-space-avoidance-based framework is a real time, feedback control approach with disturbance rejection capability. We verify the generality of the framework in two typical maneuvers: Body-Only Reorientation and Body-Tail Stabilization . Both the results of the kinematics-level and dynamics-level simulation show that the proposed controllers can track the desired orientation and avoid the null space simultaneously in a free falling condition. Experiments of the body-tail stabilization demonstrate the effectiveness of the proposed framework on a dynamical platform.
31 October 2024 in IEEE Transactions on Instrumentation and Measurement
Ultrasound-guided percutaneous needle insertion is a standard procedure employed in both biopsy and ablation in clinical practices. However, due to the complex interaction between tissue and instrument, the needle may deviate from the in-plane view, resulting in a lack of close monitoring of the percutaneous needle. To address this challenge, we introduce a robot-assisted ultrasound (US) imaging system designed to seamlessly monitor the insertion process and autonomously restore the visibility of the inserted instrument when misalignment happens. To this end, the adversarial structure is presented to encourage the generation of segmentation masks that align consistently with the ground truth in high-order space. This study also systematically investigates the effects on segmentation performance by exploring various training loss functions and their combinations. When misalignment between the probe and the percutaneous needle is detected, the robot is triggered to perform transverse searching to optimize the positional and rotational adjustment to restore needle visibility. The experimental results on ex-vivo porcine samples demonstrate that the proposed method can precisely segment the percutaneous needle (with a tip error of 0.37 ± 0.29 mm and an angle error of 1.19 ± 0.29°). Furthermore, the needle appearance can be successfully restored under the repositioned probe pose in all 45 trials, with repositioning errors of 1.51 ± 0.95 mm and 1.25 ± 0.79°.
19 May 2022 in IEEE/ASME Transactions on Mechatronics
Animals and human often use their external appendages (such as tails or arms) to achieve spectacular maneuverability and various agile motions, which cannot be easily realized in the existing legged robots. Many animals can bend or straighten up their tails in about a tenth of a second to facilitate rapid adjustment in the moment of inertia and the center of mass. However, existing robotic tails have a fixed length, limiting their inertial adjustment capability for agile locomotion. Therefore, in this article, a monopod robot with a compact, 3-DoF morphable inertial tail, is proposed to explore the effect of tail’s inertial adjustment capability on robot agility, particularly on somersault motion. The proposed 3-DoF tail composes of tail orientation structure and scissor-lift parallel mechanism based morphable inertial structure, enabling effective inertial adjustment within 0.15 s. A novel springy underactuated leg with a large stroke is also proposed to facilitate the robot’s jumping capability, necessary for somersault. We also develop a dynamic model and propose motion controllers to study the tail-assisted somersault. Simulation and experimental results demonstrate the effectiveness of the proposed mechanical design and controllers in doing tail-assisted somersault. Using our proposed adjustable-length tail, the stabilized body rotation speed during flight phase can become twice of that with a fixed-length tail. Lastly, for the first time, our robot successfully completed an advanced somersault (somersault with a twist) by utilizing the tail’s 3-D inertial adjustment capability.