In the last five years, deep learning has attracted great interest in computer-assisted systems for Minimally Invasive Surgery. The straightforward accessibility to images in surgical interventions makes deep neural networks enormously powerful for solving classification problems in complex surgical scenarios. The objective of this work is to provide readers a survey on deep learning models applied to minimally invasive surgery, identifying the different architectures used depending on the application, the results achieved until now, and the publicly available surgical datasets that can be used for validating new studies. A total of 85 publications have been extracted from manual research from four databases (IEEE Xplorer, Springer Link, Science Direct, and ACM Digital Library). After analyzing all these studies, they have been classified into four applications: surgical image analysis, surgical task analysis, surgical skill assessment, and automation of surgical tasks. This work provides a technical description of these works and a comparison among them. Finally, promising research directions to advance in this field are identified.

Robotic platforms are taking their place in the operating room because they provide more stability and accuracy during surgery. Although most of these platforms are teleoperated, a lot of research is currently being carried out to design collaborative platforms. The objective is to reduce the surgeon workload through the automation of secondary or auxiliary tasks, which would benefit both surgeons and patients by facilitating the surgery and reducing the operation time. One of the most important secondary tasks is the endoscopic camera guidance, whose automation would allow the surgeon to be concentrated on handling the surgical instruments. This paper proposes a novel autonomous camera guidance approach for laparoscopic surgery. It is based on learning from demonstration (LfD), which has demonstrated its feasibility to transfer knowledge from humans to robots by means of multiple expert showings. The proposed approach has been validated using an experimental surgical robotic platform to perform peg transferring, a typical task that is used to train human skills in laparoscopic surgery. The results show that camera guidance can be easily trained by a surgeon for a particular task. Later, it can be autonomously reproduced in a similar way to one carried out by a human. Therefore, the results demonstrate that the use of learning from demonstration is a suitable method to perform autonomous camera guidance in collaborative surgical robotic platforms.

This paper describes the mechanical design and a cognition system for a novel concept-of-camera robotic assistant. The system combines the advantages of intra-abdominal devices and autonomous camera navigation. The robotic assistant is composed of a magnetic intra-abdominal camera robot with two internal cable-driven degrees of freedom and an external robot that handles an external magnet. The intelligence of the robot is implemented in a cognitive architecture based on a long-term memory that stores the robot's knowledge and learning capabilities to improve the robot's behavior. The navigation strategy combines a reactive control based on instrument tracking and a proactive control based on predefined behaviors, depending on the actual state of the task. The robot's learning capabilities include a semantic customization, to adapt the camera's behavior to the surgeon's preferences, and reinforcement learning, to improve the camera navigation strategy. This paper details both the hardware implementation of the system and the software implementation in a robotic operating system architecture. The cognition system and the performance of the cable-driven mechanism have been validated with a set of in vitro experiments. Moreover, the camera robot has been evaluated through an in-vivo experiment in a pig.

Minimally invasive surgery (MIS) techniques are growing in quantity and complexity to cover a wider range of interventions. More specifically, hand-assisted laparoscopic surgery (HALS) involves the use of one surgeon’s hand inside the patient whereas the other one manages a single laparoscopic tool. In this scenario, those surgical procedures performed with an additional tool require the aid of an assistant. Furthermore, in the case of a human–robot assistant pairing a fluid communication is mandatory. This human–machine interaction must combine both explicit orders and implicit information from the surgical gestures. In this context, this paper focuses on the development of a hand gesture recognition system for HALS. The recognition is based on a hidden Markov model (HMM) algorithm with an improved automated training step, which can also learn during the online surgical procedure by means of a reinforcement learning process.

Endonasal surgery is a minimally invasive approach for the removal of pituitary tumors (sarcomas). In this type of procedure, the surgeon has to complete the surgical maneuvers for sarcoma resection with extreme precision, as there are many vital structures in this area. Therefore, the use of robots for this type of intervention could increase the success of the intervention by providing accurate movements. Research has focused on the development of teleoperated robots to handle a surgical instrument, including the use of virtual fixtures to delimit the working area. This paper aims to go a step further with a platform that includes a teleoperated robot and an autonomous robot dedicated to secondary tasks. In this way, the aim is to reduce the surgeon’s workload so that he can concentrate on his main task. Thus, the article focuses on the description and implementation of a navigator that coordinates both robots via a force/position control. Finally, both the navigation and control scheme were validated by in-vitro tests.

Haptic guidance is a promising method for assisting an operator in solving robotic remote operation tasks. It can be implemented through different methods, such as virtual fixtures, where a predefined trajectory is used to generate guidance forces, or interactive guidance, where sensor measurements are used to assist the operator in real-time. During the last years, the use of learning from demonstration (LfD) has been proposed to perform interactive guidance based on simple tasks that are usually composed of a single stage. However, it would be desirable to improve this approach to solve complex tasks composed of several stages or gestures. This paper extends the LfD approach for object telemanipulation where the task to be solved is divided into a set of gestures that need to be detected. Thus, each gesture is previously trained and encoded within a Gaussian mixture model using LfD, and stored in a gesture library. During telemanipulation, depending on the sensory information, the gesture that is being carried out is recognized using the same LfD trained model for haptic guidance. The method was experimentally verified in a teleoperated peg-in-hole insertion task. A KUKA LWR4+ lightweight robot was remotely controlled with a Sigma.7 haptic device with LfD-based shared control. Finally, a comparison was carried out to evaluate the performance of Gaussian mixture models with a well-established gesture recognition method, continuous hidden Markov models, for the same task. Results show that the Gaussian mixture models (GMM)-based method slightly improves the success rate, with lower training and recognition processing times.