In this work, we describe a novel framework for planning and execution of semi-autonomous tissue retraction in robotic minimally invasive surgery. The approach is aimed at autonomously removing flaps of organs or connective tissue from the surgical area, thus exposing the underlying anatomy. Initially, a deep neural network is used to analyse the endoscopic image and detect candidate flaps of tissue that obstruct the scene. Subsequently, a procedural algorithm, aimed at planning and executing the retraction gesture, is developed from extended discussions with clinicians. Experimental validation, carried out on a DaVinci Research Kit, shows an enhancement of the visible background ranging from 151.9% to 235.2%. Another significant contribution of this paper is a dataset, containing 1080 labelled surgical stereo images and the associated depth maps, representing tissue flaps in different scenarios. The work described in this paper is a fundamental step towards the autonomous execution of tissue retraction and the first example based on the simultaneous use of deep learning and procedural algorithms. The same framework could be applied to a wide range of autonomous tasks, such as debridement and placement of laparoscopic clips.

In this letter, explicit model predictive control is applied in conjunction with nonlinear optimization to a magnetically actuated flexible endoscope for the first time. The approach is aimed at computing the motion of the external permanent magnet, given the desired forces and torques. The strategy described here takes advantage of the nonlinear nature of the magnetic actuation and explicitly considers the workspace boundaries, as well as the actuation constraints. Initially, a simplified dynamic model of the tethered capsule, based on the Euler-Lagrange equations is developed. Subsequently, the explicit model predictive control is described and a novel approach for the external magnet positioning, based on a single step, nonlinear optimisation routine, is proposed. Finally, the strategy is implemented on the experimental platform, where bench-top trials are performed on a realistic colon phantom, showing the effectiveness of the technique. The work presented here constitutes an initial exploration for model-based control techniques applied to magnetically manipulated payloads; the techniques described here may be applied to a wide range of devices, including flexible endoscopes and wireless capsules. To our knowledge, this is the first example of advanced closed-loop control of magnetic capsules.

SUMMARY: In this paper, a closed-form dynamic model of flexible manipulators is developed, based on the Newton–Euler formulation of motion equations of flexible links and on the adoption of the spatial vector notation. The proposed model accounts for two main innovations with respect to the state of the art: it is obtained in closed form with respect to the joints and modal coordinates (including the quadratic velocity terms) and motion equations of the whole manipulator can be computed for any arbitrary shape of the links and any possible link cardinality starting from the output of several commercial (finite element analysis) FEA codes. The Newton–Euler formulation of motion equations in terms of the joint and elastic variables greatly improves the simulation performances and makes the model suitable for real-time control and active vibration damping. The model has been compared with literature benchmarks obtained by the classical multibody approach and further validated by comparison with experiments collected on an experimental manipulator.

Magnetic fields enable remote manipulation of objects and are ideal for medical applications because they pass through human tissue harmlessly. This capability is promising for surgical robots, allowing navigation deeper into the human anatomy and accessing organs beyond the reach of current technologies. However, magnetic manipulation is typically limited to a maximum two–degrees-of-freedom orientation, restricting complex motions, especially those including rolling around the main axis of the magnetic robot. To address this challenge, we introduce a robot design inspired by embodied intelligence and the unique geometry of developable rollers, leveraging the oloid shape. The oloid, with its axial asymmetry and sinusoidal motion, facilitates rolling when precisely controlled by an external magnetic field. We present a versatile closed-loop control model to ensure precise magnetic manipulation of an oloid-shaped robot. This capability was validated in endoluminal applications through the integration of a 28-megahertz micro-ultrasound array to perform virtual biopsies, noninvasive real-time histological imaging. Extensive in vitro and in vivo tests using a porcine model showed the robot’s ability to execute sweeping motions, identify lesions, and generate detailed three-dimensional scans of gastrointestinal subsurface tissue. This research not only restores a critical movement capability to magnetic medical robots but also enables additional clinical applications deep within the human body.

Magnetically actuated endoscopes are currently transitioning in to clinical use for procedures such as colonoscopy, presenting numerous benefits over their conventional counterparts. Intelligent and easy-to-use control strategies are an essential part of their clinical effectiveness due to the un-intuitive nature of magnetic field interaction. However, work on developing intelligent control for these devices has mainly been focused on general purpose endoscope navigation. In this work, we investigate the use of autonomous robotic control for magnetic colonoscope intervention via biopsy, another major component of clinical viability. We have developed control strategies with varying levels of robotic autonomy, including semi-autonomous routines for identifying and performing targeted biopsy, as well as random quadrant biopsy. We present and compare the performance of these approaches to magnetic endoscope biopsy against the use of a standard flexible endoscope on bench-top using a colonoscopy training simulator and silicone colon model. The semi-autonomous routines for targeted and random quadrant biopsy were shown to reduce user workload with comparable times to using a standard flexible endoscope.