Complex teleoperative tasks, such as surgery, generally require human control. However, teleoperating a robot using indirect visual information poses many technical challenges because the user is expected to control the movement(s) of the camera(s) in addition to the robot’s arms and other elements. For humans, camera positioning is difficult, error-prone, and a drain on the user’s available resources and attention. This paper reviews the state of the art of autonomous camera control with a focus on surgical applications. We also propose potential avenues of research in this field that will support the transition from direct slaved control to truly autonomous robotic camera systems.

An augmented reality (AR) cueing method designed to improve teleoperator performance under conditions of display-control misalignment is investigated. The teleoperation task was designed to mimic the operation of space robot arms, which are manipulated using hand controllers (HCs) to orient and translate the body-fixed coordinates at the end effector (EE). Cameras provide visual feedback. However, the pose of the EE seen through the camera hinders operator performance due to misalignments between the displayed EE axes and the HC axes. In this paper, the coordinate system of the EE is graphically overlaid in three dimensions on the video views with uniquely colored axes using AR. The same color scheme is used to label the corresponding axes on the HCs. Operators use these color cues to map each axis on the HCs to the corresponding colored axis of the augmented coordinates at the EE to obtain EE movement in the desired direction. Between-groups and within-participant experiments comparing EE trajectory distance, deviation from path, navigation errors, and HC axis usage were used to determine the effectiveness of the augmented coordinates over conventional teleoperation without augmented coordinates. Significant reductions in EE trajectory distance, deviation from path, navigation errors, and single-axis HC usage were observed when participants manipulated the remote robot with augmented coordinates. The results demonstrate that the use of simple AR cues in remote robot arm teleoperation is beneficial to the operator.

Robotic-assisted surgery holds significant promise to improve patient treatment by allowing surgeons to perform many types of complex operations with greater precision and flexibility than before. In order to facilitate automation of robotic surgery and more practical training for surgeons, more detailed comprehension of the surgical procedures is needed. In this regard, a key step is to develop techniques that segment and recognize surgical tasks intelligently. Surgeries involve complex continuous activities that may contain superfluous, repeated actions, and temporal variation. Therefore, any segmentation approach that has the capability to account for all these characteristics is of increased interest. Toward this goal, we develop a new segmentation algorithm, namely soft-boundary unsupervised gesture segmentation (Soft-UGS), to segment the temporal sequence of surgical gestures and model gradual transitions between them using fuzzy membership scores. The proposed framework is evaluated using a real robotic surgery dataset. Our extensive set of experiments and evaluation metrics show that the proposed Soft-UGS method is able to match manual annotations with upto 83% sensitivity, 81% precision, and 73% segmentation score. The results show that the proposed soft boundary approach can provide more insight into the surgical activities and can contribute to the automation of robotic surgeries.

It is important to monitor manufacturing processes in order to improve product quality and reduce production cost. Statistical Process Control (SPC) is the most commonly used method for process monitoring, in particular making distinctions between variations attributed to normal process variability to those caused by ‘special causes’. Most SPC and multivariate SPC (MSPC) methods are parametric in that they make assumptions about the distributional properties and autocorrelation structure of in-control process parameters, and, if satisfied, are effective in managing false alarms/-positives and false-negatives. However, when processes do not satisfy these assumptions, the effectiveness of SPC methods is compromised. Several non-parametric control charts based on sequential ranks of data depth measures have been proposed in the literature, but their development and implementation have been rather slow in industrial process control. Several non-parametric control charts based on machine learning principles have also been proposed in the literature to overcome some of these limitations. However, unlike conventional SPC methods, these non-parametric methods require event data from each out-of-control process state for effective model building. The paper presents a new non-parametric multivariate control chart based on kernel distance that overcomes these limitations by employing the notion of one-class classification based on support vector principles. The chart is non-parametric in that it makes no assumptions regarding the data probability density and only requires ‘normal’ or in-control data for effective representation of an in-control process. It does, however, make an explicit provision to incorporate any available data from out-of-control process states. Experimental evaluation on a variety of benchmarking datasets suggests that the proposed chart is effective for process monitoring.