Contemporary endoscopic simultaneous localization and mapping (SLAM) methods accurately compute endoscope poses; however, they only provide a sparse 3-D reconstruction that poorly describes the surgical scene. We propose a novel dense SLAM method whose qualities are: 1) monocular, requiring only RGB images of a handheld monocular endoscope; 2) fast, providing endoscope positional tracking and 3-D scene reconstruction, running in parallel threads; 3) dense, yielding an accurate dense reconstruction; 4) robust, to the severe illumination changes, poor texture and small deformations that are typical in endoscopy; and 5) self-contained, without needing any fiducials nor external tracking devices and, therefore, it can be smoothly integrated into the surgical workflow. It works as follows. First, accurate cluster frame poses are estimated using the sparse SLAM feature matches. The system segments clusters of video frames according to parallax criteria. Next, dense matches between cluster frames are computed in parallel by a variational approach that combines zero mean normalized cross correlation and a gradient Huber norm regularizer. This combination copes with challenging lighting and textures at an affordable time budget on a modern GPU. It can outperform pure stereo reconstructions, because the frames cluster can provide larger parallax from the endoscope's motion. We provide an extensive experimental validation on real sequences of the porcine abdominal cavity, both in-vivo and ex-vivo. We also show a qualitative evaluation on human liver. In addition, we show a comparison with the other dense SLAM methods showing the performance gain in terms of accuracy, density, and computation time.
15 January 2015 in IEEE Transactions on Pattern Analysis and Machine Intelligence
We study a problem that we call Shape-from-Template, which is the problem of reconstructing the shape of a deformable surface from a single image and a 3D template. Current methods in the literature address the case of isometric deformations, and relax the isometry constraint to the convex inextensibility constraint, solved using the so-called maximum depth heuristic. We call these methods zeroth-order since they use image point locations (the zeroth-order differential structure) to solve the shape inference problem from a perspective image. We propose a novel class of methods that we call first-order. The key idea is to use both image point locations and their first-order differential structure. The latter can be easily extracted from a warp between the template and the input image. We give a unified problem formulation as a system of PDEs for isometric and conformal surfaces that we solve analytically. This has important consequences. First, it gives the first analytical algorithms to solve this type of reconstruction problems. Second, it gives the first algorithms to solve for the exact constraints. Third, it allows us to study the well-posedness of this type of reconstruction: we establish that isometric surfaces can be reconstructed unambiguously and that conformal surfaces can be reconstructed up to a few discrete ambiguities and a global scale. In the latter case, the candidate solution surfaces are obtained analytically. Experimental results on simulated and real data show that our isometric methods generally perform as well as or outperform state of the art approaches in terms of reconstruction accuracy, while our conformal methods largely outperform all isometric methods for extensible deformations.
28 September 2020 in IEEE Transactions on Medical Imaging
A major research area in Computer Assisted Intervention (CAI) is to aid laparoscopic surgery teams with Augmented Reality (AR) guidance. This involves registering data from other modalities such as MR and fusing it with the laparoscopic video in real-time, to reveal the location of hidden critical structures. We present the first system for AR guided laparoscopic surgery of the uterus. This works with pre-operative MR or CT data and monocular laparoscopes, without requiring any additional interventional hardware such as optical trackers. We present novel and robust solutions to two main sub-problems: the initial registration, which is solved using a short exploratory video, and update registration, which is solved with real-time tracking-by-detection. These problems are challenging for the uterus because it is a weakly-textured, highly mobile organ that moves independently of surrounding structures. In the broader context, our system is the first that has successfully performed markerless real-time registration and AR of a mobile human organ with monocular laparoscopes in the OR.
1 January 2019 in Surgical Oncology Clinics of North America
Hyperspectral imaging (HSI) is a novel optical imaging modality, which has recently found diverse applications in the medical field. HSI is a hybrid imaging modality, combining a digital photographic camera with a spectrographic unit, and it allows for a contactless and non-destructive biochemical analysis of living tissue. HSI provides quantitative and qualitative information of the tissue composition at molecular level in a contrast-free manner, hence making it possible to objectively discriminate between different tissue types and between healthy and pathological tissue. Over the last two decades, HSI has been increasingly used in the medical field, and only recently it has found an application in the operating room. In the last few years, several research groups have used this imaging modality as an intraoperative guidance tool within different surgical disciplines. Despite its great potential, HSI still remains far from being routinely used in the daily surgical practice, since it is still largely unknown to most of the surgical community. The aim of this study is to provide clinical surgeons with an overview of the capabilities, current limitations, and future directions of HSI for intraoperative guidance.