In the last decade, various robotic platforms have been introduced that could support delicate retinal surgeries. Concurrently, to provide semantic understanding of the surgical area, recent advances have enabled microscope-integrated intraoperative Optical Coherent Tomography (iOCT) with high-resolution 3D imaging at near video rate. The combination of robotics and semantic understanding enables task autonomy in robotic retinal surgery, such as for subretinal injection. This procedure requires precise needle insertion for best treatment outcomes. However, merging robotic systems with iOCT intro-duces new challenges. These include, but are not limited to high demands on data processing rates and dynamic registration of these systems during the procedure. In this work, we propose a framework for autonomous robotic navigation for subretinal injection, based on intelligent real-time processing of iOCT volumes. Our method consists of an instrument pose estimation method, an online registration between the robotic and the iOCT system, and trajectory planning tailored for navigation to an injection target. We also introduce intelligent virtual B-scans, a volume slicing approach for rapid instrument pose estimation, which is enabled by Convolutional Neural Networks (CNNs). Our experiments on ex-vivo porcine eyes demonstrate the precision and repeatability of the method. Finally, we discuss identified challenges in this work and suggest potential solutions to further the development of such systems.
The “Remote Interactive Surgery Platform” (RISP) is an augmented reality (AR)-based platform for surgical telementoring. It builds upon recent advances of mixed reality head-mounted displays (MR-HMD) and associated immersive visualization technologies to assist the surgeon during an operation. It enables an interactive, real-time collaboration with a remote consultant by sharing the operating surgeon’s field of view through the Microsoft (MS) HoloLens2 (HL2). Development of the RISP started during the Medical Augmented Reality Summer School 2021 and is currently still ongoing. It currently includes features such as three-dimensional annotations, bidirectional voice communication and interactive windows to display radiographs within the sterile field. This manuscript provides an overview of the RISP and preliminary results regarding its annotation accuracy and user experience measured with ten participants.
Optical coherence tomography (OCT) is a medical imaging modality that is commonly used to diagnose retinal diseases. In recent years, linear and radial scanning patterns have been proposed to acquire three-dimensional OCT data. These patterns show differences in A-scan acquisition density across the generated volumes, and thus differ in their suitability for the diagnosis of retinal diseases. While radial OCT volumes exhibit a higher A-scan sampling rate around the scan center, linear scans contain more information in the peripheral scan areas. In this paper, we propose a method to combine a linearly and radially acquired OCT volume to generate a single compound volume, which merges the advantages of both scanning patterns to increase the information that can be gained from the three-dimensional OCT data. We initially generate 3D point clouds of the linearly and radially acquired OCT volumes and use an Iterative Closest Point (ICP) variant to register both volumes. After registration, the compound volume is created by selectively exploiting linear and radial scanning data, depending on the A-scan density of the individual scans. Fusing regions from both volumes with respect to their local A-scan sampling density, we achieve improved overall anatomical OCT information in a high-resolution compound volume. We demonstrate our method on linear and radial OCT volumes for the visualization and analysis of macular holes and the surrounding anatomical structures.
13 January 2025 in IEEE Transactions on Medical Imaging
Four-dimensional microscope-integrated optical coherence tomography enables volumetric imaging of tissue structures and tool-tissue interactions in ophthalmic surgery at interactive update rates. This enables surgeons to undertake particular surgical steps under four-dimensional optical coherence tomography (4D OCT) guidance. However, current 4D OCT systems are limited by their field of view and signal quality. Both are attributable to the emphasis on high volume acquisition rates, which is critical for smooth visual perception by the surgeon. Existing 3D volume mosaicing methods are developed in the context of diagnostic imaging and do not take dynamic surgical interactions and real-time processing into account. In this paper, we propose a novel volume mosaicing and visualization methodology that not only aims at leveraging the temporal information to overcome some of the current limitations and imaging artifacts of 4D OCT, but also is aware of the surgical context and dynamic instrument motion implicitly during registration and explicitly for visualization. We propose a rapid 4-degrees of freedom volume registration, integrating an innovative approach for volume mosaicing that takes temporal recency and semantic information into account for enhanced surgical visualization. Our experiments on 4D OCT datasets demonstrate high registration accuracy and illustrate the benefits for visualization by reducing imaging artifacts and dynamically expanding the surgical view.
Medical imaging maps tissue characteristics into image intensity values, enhancing human perception. However, comprehending this data, especially in high-stakes scenarios such as surgery, is prone to errors. Additionally, current multimodal methods do not fully leverage this valuable data in their design. We introduce "From Tissue to Sound," a new paradigm for medical sonic interaction design. This paradigm establishes a comprehensive framework for mapping tissue characteristics to auditory displays, providing dynamic and intuitive access to medical images that complement visual data, thereby enhancing multimodal perception. "From Tissue to Sound" provides an advanced and adaptable framework for the interactive sonification of multimodal medical imaging data. This framework employs a physics-based sound model composed of a network of multiple oscillators, whose mechanical properties-such as friction and stiffness-are defined by tissue characteristics extracted from imaging data. This approach enables the representation of anatomical structures and the creation of unique acoustic profiles in response to excitations of the sound model. This method allows users to explore data at a fundamental level, identifying tissue characteristics ranging from rigid to soft, dense to sparse, and structured to scattered. It facilitates intuitive discovery of both general and detailed patterns with minimal preprocessing. Unlike conventional methods that transform low-dimensional data into global sound features through a parametric approach, this method utilizes model-based unsupervised mapping between data and an anatomical sound model, enabling high-dimensional data processing. The versatility of this method is demonstrated through feasibility experiments confirming the generation of perceptually discernible acoustic signals. Furthermore, we present a novel application developed based on this framework for retinal surgery. This new paradigm opens up possibilities for designing multisensory applications for multimodal imaging data. It also facilitates the creation of interactive sonification models with various auditory causality approaches, enhancing both directness and richness.
In the last decade, various robotic platforms have been introduced that could support delicate retinal surgeries. Concurrently, to provide semantic understanding of the surgical area, recent advances have enabled microscope-integrated intraoperative Optical Coherent Tomography (iOCT) with high-resolution 3D imaging at near video rate. The combination of robotics and semantic understanding enables task autonomy in robotic retinal surgery, such as for subretinal injection. This procedure requires precise needle insertion for best treatment outcomes. However, merging robotic systems with iOCT introduces new challenges. These include, but are not limited to high demands on data processing rates and dynamic registration of these systems during the procedure. In this work, we propose a framework for autonomous robotic navigation for subretinal injection, based on intelligent real-time processing of iOCT volumes. Our method consists of an instrument pose estimation method, an online registration between the robotic and the iOCT system, and trajectory planning tailored for navigation to an injection target. We also introduce intelligent virtual B-scans, a volume slicing approach for rapid instrument pose estimation, which is enabled by Convolutional Neural Networks (CNNs). Our experiments on ex-vivo porcine eyes demonstrate the precision and repeatability of the method. Finally, we discuss identified challenges in this work and suggest potential solutions to further the development of such systems.