Surgical robots have been introduced to operating rooms over the past few decades due to their high sensitivity, small size, and remote controllability. The cable-driven nature of many surgical robots allows the systems to be dexterous and lightweight, with diameters as low as 5mm. However, due to the slack and stretch of the cables and the backlash of the gears, inevitable uncertainties are brought into the kinematics calcu-lation [1]. Since the reported end effector position of surgical robots like RAVEN-II [2] is directly calculated using the motor encoder measurements and forward kinematics, it may contain relatively large error up to 10mm, whereas semi-autonomous functions being introduced into abdominal surgeries require position inaccuracy of at most 1mm. To resolve the problem, a cost-effective, real-time and data-driven pipeline for robot end effector position precision estimation is proposed and tested on RAVEN-II. Analysis shows an improved end effector position error of around 1mm RMS traversing through the entire robot workspace without high-resolution motion tracker. The open source code, data sets, videos, and user guide can be found at //github.com/HaonanPeng/RAVEN Neural Network Estimator.

Accurate semantic image segmentation from medical imaging can enable intelligent vision-based assistance in robot-assisted minimally invasive surgery. The human body and surgical procedures are highly dynamic. While machine-vision presents a promising approach, sufficiently large training image sets for robust performance are either costly or unavailable. This work examines three novel generative adversarial network (GAN) methods of providing usable synthetic tool images using only surgical background images and a few real tool images. The best of these three novel approaches generates realistic tool textures while preserving local background content by incorporating both a style preservation and a content loss component into the proposed multi-level loss function. The approach is quantitatively evaluated, and results suggest that the synthetically generated training tool images enhance UNet tool segmentation performance. More specifically, with a random set of 100 cadaver and live endoscopic images from the University of Washington Sinus Dataset, the UNet trained with synthetically generated images using the presented method resulted in 35.7% and 30.6% improvement over using purely real images in mean Dice coefficient and Intersection over Union scores, respectively. This study is promising towards the use of more widely available and routine screening endoscopy to preoperatively generate synthetic training tool images for intraoperative UNet tool segmentation.

Accurate joint position estimation is crucial for the control of cable-driven laparoscopic surgical robots like the RAVEN-II. However, any slack and stretch in the cable can lead to errors in kinematic estimation, complicating precise control. This work proposes an efficient data-driven calibration method, requiring no additional sensors post-training. The calibration takes 8–21 min and maintains high accuracy during a 6-hour heavily loaded operating. The Deep Neural Network (DNN) model reduces errors by 76%, achieving accuracy of 0.104^∘, 0.120^∘, and 0.118 mm for joints 1, 2, and 3, respectively. Compared to end-to-end models, the DNN achieves better accuracy and faster convergence by correcting original inaccurate joint positions. Additionally, a linear regression model offers 160 times faster inference speed than the DNN, suitable for RAVEN’s 1000 Hz control loop, with slight compromises in accuracy. This approach should significantly enhance the accuracy of similar cable-driven robots.

Virtual Reality (VR) has emerged as a promising medium, enhancing human-robot interaction by offering a more immersive and intuitive interface. This paper presents a novel approach utilizing VR, particularly the Meta Quest 2 device, as an interface for remote control of a robot performing a pick-and-place task in a proxy data center environment. A user interface was created in VR, enabling users to control the Fetch Mobile Manipulator robot and monitor its pose for secure and safe remote operation. This work integrates virtual reality, computer vision, and human-robot interaction methodologies. The system demonstrates the effectiveness of this VR-based human-robot interaction approach, showcasing its potential to enhance productivity and safety in industrial applications involving remote robot control.

Surgical robots equipped with cable-driven mechanisms have flexible, light, and compact arms and tools. However, cable slack, stretch, and gear backlash introduce unavoidable errors from motor positions to joint positions and the end-effector pose. This paper presents a learning-based joint position calibration method for the RAVEN-II surgical robot, employing deep neural networks and gated recurrent units. Compared to fixed offset compensation, the learning-based calibrations reduce the joint position errors by over 62.4% (unloaded) and 54.8% (loaded). Furthermore, removal and inaccurate ablation studies on input features identify that raw joint positions and motor torques are the most important model inputs for calibration accuracy. These studies also reveal that the models are capable of inferring joint positions from the end-effector pose and prioritize the direction of motor torques over their amplitude. When guided appropriately, the models can also compensate for encoder value inconsistencies occurring with robot re-homings. By excluding the unnecessary input features, lightweight models are developed and achieve better performance and efficiency simultaneously, reducing the training time on the CPU to 2.5 minutes. All data and code are open-source at https://github.com/uw-biorobotics/RAVEN-2-Feature-Ablation Note to Practitioners—This paper presents a data-driven neural network/AI based calibration method for accurate joint position estimation on a cable-driven research surgical robot (RAVEN-II). We aim to understand our learning calibration by studying the importance of each input feature using two ablation methods. We selectively eliminate (removal ablation) or add distorting noise to (inaccurate ablation) input features one at a time, and retrain the calibration model. Then, the increase in error suggests the importance of the target input feature. By excluding unnecessary input features and thus reducing the input dimension, the size of the machine learning model can be reduced without losing accuracy. The simpler model can be trained and inferred efficiently without GPU acceleration. All data and code used in this paper, including robot control, data collection, and model training, are available online.