Robots are expanding from industrial applications to daily life, in areas such as medical robotics, rehabilitative robotics, social robotics, and mobile/aerial robotics systems. In recent years, augmented reality (AR) has been integrated into many robotic applications, including medical, industrial, human–robot interactions, and collaboration scenarios. In this work, AR for both medical and industrial robot applications is reviewed and summarized. For medical robot applications, we investigated the integration of AR in (1) preoperative and surgical task planning; (2) image-guided robotic surgery; (3) surgical training and simulation; and (4) telesurgery. AR for industrial scenarios is reviewed in (1) human–robot interactions and collaborations; (2) path planning and task allocation; (3) training and simulation; and (4) teleoperation control/assistance. In addition, the limitations and challenges are discussed. Overall, this article serves as a valuable resource for working in the field of AR and robotic research, offering insights into the recent state of the art and prospects for improvement.
11 June 2023 in Bioengineering & Translational Medicine
Three‐dimensional (3D) imaging techniques (e.g., confocal microscopy) are commonly used to visualize in vitro models, especially microvasculature on‐a‐chip. Conversely, 3D analysis is not the standard method to extract quantitative information from those models. We developed the μVES algorithm to analyze vascularized in vitro models leveraging 3D data. It computes morphological parameters (geometry, diameter, length, tortuosity, eccentricity) and intravascular flow velocity. μVES application to microfluidic vascularized in vitro models shows that they successfully replicate functional features of the microvasculature in vivo in terms of intravascular fluid flow velocity. However, wall shear stress is lower compared to in vivo references. The morphological analysis also highlights the model's physiological similarities (vessel length and tortuosity) and shortcomings (vessel radius and surface‐over‐volume ratio). The addition of the third dimension in our analysis produced significant differences in the metrics assessed compared to 2D estimations. It enabled the computation of new indices, such as vessel eccentricity. These μVES capabilities can find application in analyses of different in vitro vascular models, as well as in vivo and ex vivo microvasculature.
16 February 2024 in IEEE Robotics & Automation Magazine
During percutaneous nephrolithotomy (PCNL) operations, the surgeon is required to define the incision point on the patient’s back, align the needle to a preplanned path, and perform puncture operations afterward. The procedure is currently performed manually using ultrasound or fluoroscopy imaging for needle orientation, which, however, implies limited accuracy and low reproducibility. This work incorporates augmented reality (AR) visualization with an optical see-through head-mounted display (OST-HMD) and human–robot collaboration (HRC) framework to empower the surgeon’s task completion performance. In detail, eye-to-hand calibration, system registration, and hologram model registration are performed to realize visual guidance. A Cartesian impedance controller is used to guide the operator during the needle-puncture task execution. Experiments are conducted to verify the system performance compared with conventional manual puncture procedures and a 2D monitor-based visualization interface. The results showed that the proposed framework achieves the lowest median and standard deviation error of 3.17 ± 1.36 mm in translation and 1.2 ± .9° in orientation across all of the experimental groups, respectively. Furthermore, the National Aeronautics and Space Administration task load index (NASA-TLX) user evaluation results indicate that the proposed framework requires the lowest workload score of 42.5 ± 13.7 for task completion compared to other experimental setups. The proposed framework exhibits significant potential for clinical application in the PCNL task, as it enhances the surgeon’s perception capability, facilitates collision-free needle insertion path planning, and minimizes errors in task completion.
Ovarian cancer is the deadliest gynecologic malignancy worldwide, with a 5-year overall survival rate of approximately 49%. Although complete resection is associated with the most favorable prognosis following primary debulking surgery, accurately assessing tumor resectability at diagnosis remains a major clinical challenge. We propose a decision support system (DSS) designed to predict residual tumor after primary debulking surgery, based on clinical and imaging data available at diagnosis. The system was developed and validated using a retrospective cohort of 465 patients with high-grade serous ovarian cancer, collected at the European Institute of Oncology in Milan, Italy. We developed a deep learning (DL) model that combines pre-trained Vision Transformer encoders with an attention mechanism and a classification head. When evaluated on an independent test cohort of 75 cases, our best-performing model achieved an area under the curve (AUC) of 0.80 (95% CI: 0.68–0.89) and a recall of 0.86 (95% CI: 0.71–0.97), demonstrating discriminatory ability with a particularly low rate of false negatives. Clinically, the model correctly identified 24 out of 28 patients (86%) with non-resectable disease, who would not have benefited from primary debulking surgery. Notably, within this subgroup, the model accurately predicted 93% (13 out of 14) of cases in which surgery was aborted intraoperatively due to unforeseen unresectable disease. These findings suggest that the model could have potential in preventing unnecessary and inappropriate surgical interventions. The proposed DSS is a fully automated, DL–based system for predicting tumor resectability at diagnosis, without the need for manual segmentation or expert evaluation of radiological features and structured clinical parameters. This approach could facilitate and accelerate personalized treatment planning in ovarian cancer. The work is part of the project Under-XAI: understanding ovarian cancer initiation and progression through explainable AI. Project code: PNRR-MAD-2022-12376574.
Specular highlights distort appearance, obscure texture, and hinder geometric reasoning in both natural and surgical imagery. We present UnReflectAnything, an RGB-only framework that removes highlights from a single image by predicting a highlight map together with a reflection-free diffuse reconstruction. The model uses a frozen vision transformer encoder to extract multi-scale features, a lightweight head to localize specular regions, and a token-level inpainting module that restores corrupted feature patches before producing the final diffuse image. To overcome the lack of paired supervision, we introduce a Virtual Highlight Synthesis pipeline that renders physically plausible specularities using monocular geometry, Fresnel-aware shading, and randomized lighting which enables training on arbitrary RGB images with correct geometric structure. UnReflectAnything generalizes across natural and surgical domains where non-Lambertian surfaces and non-uniform lighting create severe highlights and it achieves competitive performance with state-of-the-art results on several benchmarks. Project Page: https://alberto-rota.github.io/UnReflectAnything/
Accurate spatial understanding is essential for image-guided surgery, augmented reality integration and context awareness. In minimally invasive procedures, where visual input is the sole intraoperative modality, establishing precise pixel-level correspondences between endoscopic frames is critical for 3D reconstruction, camera tracking, and scene interpretation. However, the surgical domain presents distinct challenges: weak perspective cues, non-Lambertian tissue reflections, and complex, deformable anatomy degrade the performance of conventional computer vision techniques. While Deep Learning models have shown strong performance in natural scenes, their features are not inherently suited for fine-grained matching in surgical images and require targeted adaptation to meet the demands of this domain. This research presents a novel Deep Learning pipeline for establishing feature correspondences in endoscopic image pairs, alongside a self-supervised optimization framework for model training. The proposed methodology leverages a novel-view synthesis pipeline to generate ground-truth inlier correspondences, subsequently utilized for mining triplets within a contrastive learning paradigm. Through this self-supervised approach, we augment the DINOv2 backbone with an additional Transformer layer, specifically optimized to produce embeddings that facilitate direct matching through cosine similarity thresholding. Experimental evaluation demonstrates that our pipeline surpasses state-of-the-art methodologies on the SCARED datasets improved matching precision and lower epipolar error compared to the related work. The proposed framework constitutes a valuable contribution toward enabling more accurate high-level computer vision applications in surgical endoscopy.