Purpose of Review: This review provides an overview of the most recent robotic ultrasound systems that have contemporary emerged over the past five years, highlighting their status and future directions. The systems are categorized based on their level of robot autonomy (LORA). Recent Findings: Teleoperating systems show the highest level of technical maturity. Collaborative assisting and autonomous systems are still in the research phase, with a focus on ultrasound image processing and force adaptation strategies. However, missing key factors are clinical studies and appropriate safety strategies. Future research will likely focus on artificial intelligence and virtual/augmented reality to improve image understanding and ergonomics. Summary: A review on robotic ultrasound systems is presented in which first technical specifications are outlined. Hereafter, the literature of the past five years is subdivided into teleoperation, collaborative assistance, or autonomous systems based on LORA. Finally, future trends for robotic ultrasound systems are reviewed with a focus on artificial intelligence and virtual/augmented reality.

Real-time volumetric (4D) ultrasound has shown high potential for diagnostic and therapy guidance tasks. One of the main drawbacks of ultrasound imaging to date is the reliance on manual probe positioning and the resulting user dependence. Robotic assistance could help overcome this issue and facilitate the acquisition of long-term image data to observe dynamic processes in vivo over time. The aim of this study is to assess the feasibility of robotic probe manipulation and organ motion quantification during extended imaging sessions. The system consists of a collaborative robot and a 4D ultrasound system providing real-time data access. Five healthy volunteers received liver and prostate scans during free breathing over 30 min. Initial probe placement was performed with real-time remote control with a predefined contact force of 10 N. During scan acquisition, the probe position was continuously adjusted to the body surface motion using impedance control. Ultrasound volumes, the pose of the end-effector and the estimated contact forces were recorded. For motion analysis, one anatomical landmark was manually annotated in a subset of ultrasound frames for each experiment. Probe contact was uninterrupted over the entire scan duration in all ten sessions. Organ drift and imaging artefacts were successfully compensated using remote control. The median contact force along the probe’s longitudinal axis was 10.0 N with maximum values of 13.2 and 21.3 N for liver and prostate, respectively. Forces exceeding 11 N only occurred in 0.3% of the time. Probe and landmark motion were more pronounced in the liver, with median interquartile ranges of 1.5 and 9.6 mm, compared to 0.6 and 2.7 mm in the prostate. The results show that robotic ultrasound imaging with dynamic force control can be used for stable, long-term imaging of anatomical regions affected by motion. The system facilitates the acquisition of 4D image data in vivo over extended scanning periods for the first time and holds the potential to be used for motion monitoring for therapy guidance as well as diagnostic tasks.

In radiation therapy of abdominal targets, optimal tumor irradiation can be challenging due to intrafractional motion. Current target localization methods are mainly indirect, surrogate-based and the patient is exposed to additional radiation due to X-ray imaging. In contrast, 4D ultrasound (4DUS) imaging provides volumetric images of soft tissue tumors in real-time without ionizing radiation, facilitating a non-invasive, direct tracking method. In this study, the target was defined by features located in its local neighborhood. Features were extracted using the FAST detector and the BRISK descriptor, which were extended to 3D. To account for anatomical variability, a feature library was generated that contains manually annotated target information and relative locations of the features. During tracking, features were extracted from the current 4DUS volume and compared to the feature library. Recognized features are used to estimate feature position and shape. The developed method was evaluated in 4DUS sequences of the liver of three healthy subjects. For each dataset, a target was defined and manually contoured in a training and a test sequence. Training was used for library creation, the test sequence for target tracking. The target estimations are compared to the annotations to quantify a tracking error. The results show that binary feature libraries can be used for robust target localization in 4DUS data of the liver and could potentially serve as a tracking method less sensitive to target deformation.

Purpose: In this study, we present and validate a novel concept for target tracking in 4D ultrasound. The key idea is to replace image patch similarity metrics by distances in a latent representation. For this, 3D ultrasound patches are mapped into a representation space using sliced-Wasserstein autoencoders. Methods: A novel target tracking method for 4D ultrasound is presented that performs tracking in a representation space instead of in images space. Sliced-Wasserstein autoencoders are trained in an unsupervised manner which are used to map 3D ultrasound patches into a representation space. The tracking procedure is based on a greedy algorithm approach and measuring distances between representation vectors to relocate the target . The proposed algorithm is validated on an in vivo data set of liver images. Furthermore, three different concepts for training the autoencoder are presented to provide cross-patient generalizability, aiming at minimal training time on data of the individual patient. Results: Eight annotated 4D ultrasound sequences are used to test the tracking method. Tracking could be performed in all sequences using all autoencoder training approaches. A mean tracking error of 3.23 mm could be achieved using generalized fine-tuned autoencoders. It is shown that using generalized autoencoders and fine-tuning them achieves better tracking results than training subject individual autoencoders. Conclusion: It could be shown that distances between encoded image patches in a representation space can serve as a meaningful measure of the image patch similarity, even under realistic deformations of the anatomical structure. Based on that, we could validate the proposed tracking algorithm in an in vivo setting. Furthermore, our results indicate that using generalized autoencoders, fine-tuning on only a small number of patches from the individual patient provides promising results.

For training deep neural networks, large data sets are required. Especially in the medical 3D ultrasound (US) image domain, the amount of available data is limited. A common method to enlarge small data sets is using data augmentation. However, simple geometric augmentation techniques like rotation or sheering can lead to unrealistic US images. In this study, a novel structure-dependent deformable augmentation method for 3D US patches is proposed. The approach is based on learning realistic motion patterns from 3D liver US images using a conditional variational auto encoder (CVAE) where the condition represents the anatomical structure type. The CVAE augmentation performance is compared to a baseline variational auto encoder (VAE). It is shown that the CVAE generates deformable augmentations 18.5 percentage points more similar to realistic deformations than the VAE. Furthermore, the method is evaluated in an expert study where experienced radiologists were asked to rate the realism of US patches. The results show that the experts could not distinguish between CVAE augmented and original US patches. Applying the proposed method in a target detection application improved the performance of a neural network by 41 %. The proposed augmentation method is able to apply realistic deformation to 3D US patches which can be used to enlarge a small data set for usage in deep learning applications.

Acquiring reproducible ultrasound images of high quality is challenging in ultrasound imaging. While physicians can rely on their experience, robot-assisted systems must be able to automatically align the ultrasound probe with the correct orientation. This paper describes a method to align the central axis of the probe to the surface normal at the point of contact. This is done by analyzing the area of ultrasound volumes directly below the transducer of the probe. A convolutional neural network is trained to estimate the inclination of the probe orientation towards the direction of the surface normal. Experiments on two different phantoms indicate that the mean absolute angle error between the estimated rotation and the ground truth surface normal are 5.0 ± 2.8∘. The method is able to keep the probe-surface contact continuous and the results indicate that the method is invariant to anatomical structures.