Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a specific emphasis on method robustness and generalization capabilities. Our data set comprises 30 laparoscopic videos and corresponding sensor data from medical devices in the operating room for three different types of laparoscopic surgery. Annotations include surgical phase labels for all video frames as well as information on instrument presence and corresponding instance-wise segmentation masks for surgical instruments (if any) in more than 10,000 individual frames. The data has successfully been used to organize international competitions within the Endoscopic Vision Challenges 2017 and 2019.

Continual Learning (CL) is a novel AI paradigm in which tasks and data are made available over time; thus, the trained model is computed on the basis of a stream of data. CL-based approaches are able to learn new skills and knowledge without forgetting the previous ones, with no guaranteed access to previously encountered data, and mitigating the so-called “catastrophic forgetting” phenomenon. Interestingly, by making AI systems able to learn and improve over time without the need for large amounts of new data or computational resources, CL can help at reducing the impact of computationally-expensive and energy-intensive activities; hence, CL can play a key role in the path towards more green AIs, enabling more efficient and sustainable uses of resources. In this work, we describe different methods proposed in the literature to solve CL tasks; we survey different applications, highlighting strengths and weaknesses, with a particular focus on the biomedical context. Furthermore, we discuss how to make the methods more robust and suitable for a wider range of applications.

Challenges have become the state-of-the-art approach to benchmark image analysis algorithms in a comparative manner. While the validation on identical data sets was a great step forward, results analysis is often restricted to pure ranking tables, leaving relevant questions unanswered. Specifically, little effort has been put into the systematic investigation on what characterizes images in which state-of-the-art algorithms fail. To address this gap in the literature, we (1) present a statistical framework for learning from challenges and (2) instantiate it for the specific task of instrument instance segmentation in laparoscopic videos. Our framework relies on the semantic meta data annotation of images, which serves as foundation for a General Linear Mixed Models (GLMM) analysis. Based on 51,542 meta data annotations performed on 2,728 images, we applied our approach to the results of the Robust Medical Instrument Segmentation Challenge (ROBUST-MIS) challenge 2019 and revealed underexposure, motion and occlusion of instruments as well as the presence of smoke or other objects in the background as major sources of algorithm failure. Our subsequent method development, tailored to the specific remaining issues, yielded a deep learning model with state-of-the-art overall performance and specific strengths in the processing of images in which previous methods tended to fail. Due to the objectivity and generic applicability of our approach, it could become a valuable tool for validation in the field of medical image analysis and beyond.

We present a novel framework for disease classification from high-dimensional gene expression data or from several characteristic of patients. We take advantage of Principle Component Analysis to perform dimensionality reduction and heatmaps for embedding the complex information in a 2-D image, and we make use of a convolutional neural network to make classification of different tumor types. Experimental analyses show that the proposed method achieves good performance, and encourages its application to other genomic data or pathological context.