We develop a stereo-multispectral endoscopic prototype in which a filter-wheel is used for surgical guidance to remove cholesteatoma tissue in the middle ear. Cholesteatoma is a destructive proliferating tissue. The only treatment for this disease is surgery. Removal is a very demanding task, even for experienced surgeons. It is very difficult to distinguish between bone and cholesteatoma. In addition, it can even reoccur if not all tissue particles of the cholesteatoma are removed, which leads to undesirable follow-up operations. Therefore, we propose an image-based method that combines multispectral tissue classification and 3D reconstruction to identify all parts of the removed tissue and determine their metric dimensions intraoperatively. The designed multispectral filter-wheel 3D-endoscope prototype can switch between narrow-band spectral and broad-band white illumination, which is technically evaluated in terms of optical system properties. Further, it is tested and evaluated on three patients. The wavelengths 400 nm and 420 nm are identified as most suitable for the differentiation task. The stereoscopic image acquisition allows accurate 3D surface reconstruction of the enhanced image information. The first results are promising, as the cholesteatoma can be easily highlighted, correctly identified, and visualized as a true-to-scale 3D model showing the patient-specific anatomy.

This contribution is intended to provide researchers with a comprehensive overview of the current state-of-the-art concerning real-time 3D reconstruction methods suitable for medical endoscopy. Over the past decade, there have been various technological advancements in computational power and an increased research effort in many computer vision fields such as autonomous driving, robotics, and unmanned aerial vehicles. Some of these advancements can also be adapted to the field of medical endoscopy while coping with challenges such as featureless surfaces, varying lighting conditions, and deformable structures. To provide a comprehensive overview, a logical division of monocular, binocular, trinocular, and multiocular methods is performed and also active and passive methods are distinguished. Within these categories, we consider both flexible and non-flexible endoscopes to cover the state-of-the-art as fully as possible. The relevant error metrics to compare the publications presented here are discussed, and the choice of when to choose a GPU rather than an FPGA for camera-based 3D reconstruction is debated. We elaborate on the good practice of using datasets and provide a direct comparison of the presented work. It is important to note that in addition to medical publications, publications evaluated on the KITTI and Middlebury datasets are also considered to include related methods that may be suited for medical 3D reconstruction.

Background: Nasal septum perforations (NSP) have many uncomfortable symptoms for the patient and a highly negative impact on quality of life. NSPs are closed using patient-specific implants or surgery. Implants are created either under anesthesia using silicone impressions or using 3D models from CT data. Disadvantages for patient safety are the increased risk of morbidity or radiation exposure. Materials and methods: In the context of otorhinolaryngologic surgery, we present a gentle approach to treating NSP with a new image-based, contactless, and radiation-free measurement method using a 3D endoscope. The method relies on image information only and makes use of real-time capable computer vision algorithms to compute 3D information. This endoscopic method can be repeated as often as desired in the clinical course and has already proven its accuracy and robustness for robotic-assisted surgery (RAS) and surgical microscopy. We expand our method for nasal surgery, as there are additional spatial and stereoperspective challenges. Results: After measuring 3 relevant parameters (NSP extension: axial, coronal, and NSP circumference) of 6 patients and comparing the results of 2 stereoendoscopes with CT data, it was shown that the image-based measurements can achieve comparable accuracies to CT data. One patient could be only partially evaluated because the NSP was larger than the endoscopic field of view. Conclusion: Based on the very good measurements, we outline a therapeutic procedure which should enable the production of patient-specific NSP implants based on endoscopic data only.

Zusammenfassung: Hintergrund: Die vielfältigen unangenehmen Symptome von Nasenseptumdefekten (NSD) führen zu einer deutlichen Einschränkung der Lebensqualität. NSD können mittels patientenspezifischer Implantate oder durch eine Op. verschlossen werden. Implantate werden dabei durch Silikonabformungen unter Vollnarkose, ggf. in lokaler Betäubung, oder anhand von 3D-Modellen aus computertomographischen (CT-)Daten erstellt. Nachteile für die Patientensicherheit sind ein erhöhtes Morbiditätsrisiko oder die Strahlenbelastung. Methodik: Für die Hals‑, Nasen- und Ohren-(HNO)-Chirurgie wird ein schonender Ansatz zur Behandlung von NSD mit einer neuen bildbasierten, kontakt- und strahlungsfreien Messmethode unter Nutzung eines Stereoendoskops vorgestellt. Das Verfahren beruht ausschließlich auf Bilddaten und nutzt echtzeitfähige Bildverarbeitungsalgorithmen zur Berechnung von 3D-Informationen. Es ist beliebig oft wiederholbar und wurde bereits erfolgreich in der robotergestützten Chirurgie und in der OP-Mikroskopie eingesetzt. Daher wurde diese Methode für die Nasenchirurgie erweitert, für die es zusätzliche anatomische und stereoskopische Herausforderungen gibt. Ergebnisse: Nach Auswertung von 3 relevanten Messgrößen (NSD-Ausdehnung: axial, koronal und Umfang) von 6 Patienten und Vergleich der Ergebnisse von 2 Stereoendoskopen mit vorhandenen CT-Daten zeigte sich: Die bildbasierten Messergebnisse können vergleichbare Genauigkeiten wie CT-Daten erzielen. Bei einem Patienten wurden die Daten nur teilweise ausgewertet, da der NSD größer als das endoskopische Sichtfeld war. Schlussfolgerung: Aufbauend auf den sehr guten Messwerten wird ein Therapieverfahren skizziert, welches die Herstellung von patientenspezifischen NSD-Implantaten auf Basis endoskopischer Daten ermöglicht.

We propose a new benchmark for evaluating stereoscopic visual-inertial computer vision algorithms (SLAM/ SfM/ 3D Reconstruction/ Visual-Inertial Odometry) for minimally invasive surgical (MIS) interventions in the abdomen. Our MITI Dataset available at [https://mediatum.ub.tum.de/1621941] provides all the necessary data by a complete recording of a handheld surgical intervention at Research Hospital Rechts der Isar of TUM. It contains multimodal sensor information from IMU, stereoscopic video, and infrared (IR) tracking as ground truth for evaluation. Furthermore, calibration for the stereoscope, accelerometer, magnetometer, the rigid transformations in the sensor setup, and time-offsets are available. We wisely chose a suitable intervention that contains very few cutting and tissue deformation and shows a full scan of the abdomen with a handheld camera such that it is ideal for testing SLAM algorithms. Intending to promote the progress of visual-inertial algorithms designed for MIS application, we hope that our clinical training dataset helps and enables researchers to enhance algorithms.

We propose a novel method to tackle the visual-inertial localization problem for constrained camera movements. We use residuals from the different modalities to jointly optimize a global cost function. The residuals emerge from IMU measurements, stereoscopic feature points, and constraints on possible solutions in SE(3). In settings where dynamic disturbances are frequent, the residuals reduce the complexity of the problem and make localization feasible. We verify the advantages of our method in a suitable medical use case and produce a dataset capturing a minimally invasive surgery in the abdomen. Our novel clinical dataset MITI is comparable to state-of-the-art evaluation datasets, contains calibration and synchronization and is available at https://mediatum.ub.tum.de/1621941.