Deep learning-based techniques have gained unprecedented success for object detection tasks. The state of the art object detection accuracy and robustness have been achieved by Faster R-CNN framework based algorithms. However, no attempts have been made to detect seagrasses from underwater images mostly due to lack of labelled ground truth dataset, and additional challenges imposed by underwater photographs and low boundary differences among the seagrass and surrounding vegetation. We have created a dataset consisting of 2,699 underwater images of Halophila ovalis (one of the common type of seagrasses from Indo-Pacific saltwater environments [1]). We have labelled the seagrass and implemented Faster R-CNN based object detector to detect them from underwater images. We have used Inception V2 network in the Faster R-CNN pipeline and found, this network showed a high mean average precision (mAP) of 0.3464 on laboratory images only, and 0.261 on a test set consists of both field and laboratory images.

Long-distance, high latency teleoperation tasks are difficult, highly stressful for teleoperators, and prone to over-corrections, which can lead to loss of control. At higher latencies, or when teleoperating at higher vehicle speed, the situation becomes progressively worse. To explore potential solutions, this research work investigates two 2D visual feedback-based assistive interfaces (sliding-only and sliding-and-zooming windows) that apply simple but effective video transformations to enhance teleoperation. A teleoperation simulator that can replicate teleoperation scenarios affected by high and adjustable latency has been developed to explore the effectiveness of the proposed assistive interfaces. Three image comparison metrics have been used to fine-tune and optimise the proposed interfaces. An operator survey was conducted to evaluate and compare performance with and without the assistance. The survey has shown that a 900ms latency increases task completion time by up to 205% for an on-road and 147% for an off-road driving track. Further, the overcorrection-induced oscillations increase by up to 718% with this level of latency. The survey has shown the sliding-only video transformation reduces the task completion time by up to 25.53%, and the sliding-and-zooming transformation reduces the task completion time by up to 21.82%. The sliding-only interface reduces the oscillation count by up to 66.28%, and the sliding-and-zooming interface reduces it by up to 75.58%. The qualitative feedback from the participants also shows that both types of assistive interfaces offer better visual situational awareness, comfort, and controllability, and significantly reduce the impact of latency and intermittency on the teleoperation task.

High latency in teleoperation has a significant negative impact on operator performance. While deep learning has revolutionized many domains recently, it has not previously been applied to teleoperation enhancement. We propose a novel approach to predict video frames deep into the future using neural networks informed by synthetically generated optical flow information. This can be employed in teleoperated robotic systems that rely on video feeds for operator situational awareness. We have used the image‐to‐image translation technique as a basis for the prediction of future frames. The Pix2Pix conditional generative adversarial network (cGAN) has been selected as a base network. Optical flow components reflecting real‐time control inputs are added to the standard RGB channels of the input image. We have experimented with three data sets of 20,000 input images each that were generated using our custom‐designed teleoperation simulator with a 500‐ms delay added between the input and target frames. Structural Similarity Index Measures (SSIMs) of 0.60 and Multi‐SSIMs of 0.68 were achieved when training the cGAN with three‐channel RGB image data. With the five‐channel input data (incorporating optical flow) these values improved to 0.67 and 0.74, respectively. Applying Fleiss' κ gave a score of 0.40 for three‐channel RGB data, and 0.55 for five‐channel optical flow‐added data. We are confident the predicted synthetic frames are of sufficient quality and reliability to be presented to teleoperators as a video feed that will enhance teleoperation. To the best of our knowledge, we are the first to attempt to reduce the impacts of latency through future frame prediction using deep neural networks.

Predicting future frames through image‐to‐image translation and using these synthetically generated frames for high‐speed ground vehicle teleoperation is a new concept to address latency and enhance operational performance. In the immediate previous work, the image quality of the predicted frames was low and a lot of scene detail was lost. To preserve the structural details of objects and improve overall image quality in the predicted frames, several novel ideas are proposed herein. A filter has been designed to remove noise from dense optical flow components resulting from frame rate inconsistencies. The Pix2Pix base network has been modified and a structure‐aware SSIM‐based perpetual loss function has been implemented. A new dataset of 20 000 training input images and 2000 test input images with a 500 ms delay between the target and input frames has been created. Without any additional video transformation steps, the proposed improved model achieved PSNR of 23.1; SSIM of 0.65; and MS‐SSIM of 0.80, a substantial improvement over our previous work. A Fleiss’ kappa score of >0.40 (0.48 for the modified network and 0.46 for the perpetual loss function) proves the reliability of the model.