In hazardous environments, where conditions present risks for humans, the maintenance and interventions are often done with teleoperated remote systems or mobile robotic manipulators to avoid human exposure to dangers. The increasing need for safe and efficient teleoperation requires advanced environmental awareness and collision avoidance. The up-to-date screen-based 2D or 3D interfaces do not fully allow the operator to immerse in the controlled scenario. This problem can be addressed with the emerging Mixed Reality (MR) technologies with Head-Mounted Devices (HMDs) that offer stereoscopic immersion and interaction with virtual objects. Such human-robot interfaces have not yet been demonstrated in telerobotic interventions in particle physics accelerators. Moreover, robotic operations often require a few experts to collaborate, which increases the system complexity and requires sharing a multi-user Augmented Reality (AR) workspace. The multi-user telerobotics with shared control in the AR has not yet been approached in the industrial state-of-the-art. In this work, the developed MR human-robot interface using the AR HMD is presented. The interface adapts to the constrained wireless networks in particle accelerator facilities and provides reliable high-precision interaction and specialized visualization. The multimodal operation uses hands, eyes and user motion tracking, and voice recognition for control, as well as offers video, 3D point cloud and audio feedback from the robot. Multiple experts can collaborate in the AR workspace locally or remotely, share the robot’s control and monitor robotic teleoperation. Ten (10) operators tested the interface in intervention scenarios in the European Organization for Nuclear Research (CERN) with complete network characterization and measurements to conclude if operational requirements were met and if the network architecture could support single and multi-user communication load. The interface system has proved to be operationally ready at the Technical Readiness Level (TRL) 8 - and was validated through successful tests and demonstration in single and multi-user missions. Some areas of system limitations and further work were identified, such as optimising the network architecture for multi-user scenarios or high-level interface actions applying automatic interaction strategies with the robot depending on network conditions.

Human-Robot Interfaces have a key role in the design of secure and efficient robotic systems. Great effort has been put during the past decades on the design of advanced interfaces for domestic and industrial robots. However, robots for intervention in unplanned and hazardous scenarios still need further research, especially when the mission requires the use of multiple robotic systems, to obtain an acceptable level of usability and safety. This paper describes the design and the software engineering process behind the development of a modular and multimodal Human-Robot Interface for intervention with a cooperative team of robots, as well as its validation and commissioning, as it is being used in real operations at CERN’s accelerators complex. The proposed Human-Robot Interface allows the control of a heterogeneous set of robots homogeneously, providing the operator, among other features, with live scripting functionalities which can be programmed and adapted in run-time, for example, to increase operator’s multi-tasking in a multi-agent scenario. The operator is given the capability to enter in the control loop between the HRI and the robot and customize the control commands according to the operation. To provide such functionalities, well-defined software development approaches have been adopted, for guaranteeing the modularity and the safety of the system during its continuous development. The paper describes the modules offered by the HRI, such as the multimodality, multi-robot control, safety, operators training, and communications architecture, among others. The HRI and the CERN Robotic Framework where it belongs are designed in a modular manner, in order to be able to adapt both, software and hardware architecture in a short time, to the next planned mission. Results present the experience gained with the system, demonstrating a high level of usability, learnability and safety when operated by both, non-experts and qualified robotic operators. The multimodal user interface has demonstrated to be very accurate and secure, providing a unique system to control, in a teleoperated or supervised manner, both single and multiple heterogeneous mobile manipulators. At the moment of writing, the user interface has been successfully used in 100 real interventions in radioactive industrial environments. The presented HRI is a novel research contribution in terms of multimodality, adaptability and modularity for mobile manipulator robotic teams in radioactive environments, especially for its software architecture, as part of the CERN Robotic Framework.

Intelligent robotic systems are becoming essential for industries, nuclear plants and for harsh environments in general, such as the European Organization for Nuclear Research (CERN) particles accelerator complex and experiments. In order to increase safety and machine availability, robots can perform repetitive, unplanned and dangerous tasks, which humans either prefer to avoid or are unable to carry out due to hazards, size constraints, or the extreme environments in which they take place. A novel robotic framework for autonomous inspections and supervised teleoperations in harsh environments is presented. The proposed framework covers all aspects of a robotic intervention, from the specification and operator training, the choice of the robot and its material in accordance with possible radiological contamination risks, to the realization of the intervention, including procedures and recovery scenarios. The robotic solution proposed in this paper is able to navigate autonomously, inspecting unknown environments in a safe way. A new real-time control system was implemented in order to guarantee a fast response to environmental changes and adaptation to different type of scenarios the robot may find in a semi-structured and hazardous environment. Components of the presented framework are: a novel bilateral master-slave control, a new robotic platform named CERNbot as well as an advanced user-friendly multimodal human-robot interface, also used for the operators’ offline training, allowing technicians not expert in robot operation to perform inspection/maintenance tasks. The proposed system has been tested and validated with real robotic interventions in the CERN hazardous particle accelerator complex.

Remote robotic systems are employed in the CERN accelerator complex to perform different tasks, such as the safe handling of cables and their connectors. Without dedicated control, these kinds of actions are difficult and require the operators’ intervention, which is subjected to dangerous external agents. In this paper, two novel modules of the CERNTAURO framework are presented to provide a safe and usable solution for managing optical fibres and their connectors. The first module is used to detect touch and slippage, while the second one is used to regulate the grasping force and contrast slippage. The force reference was obtained with a combination of object recognition and a look-up table method. The proposed strategy was validated with tests in the CERN laboratory, and the preliminary experimental results demonstrated statistically significant increases in time-based efficiency and in the overall relative efficiency of the tasks.

Robotic interventions with redundant mobile manipulators pose a challenge for telerobotics in hazardous environments, such as underwater, underground, nuclear facilities, particle accelerators, aerial or space. Communication issues can lead to critical consequences, such as imprecise manipulation resulting in collisions, breakdowns and mission failures. The research presented in this paper was driven by the needs of a real robotic intervention scenario in the Large Hadron Collider (LHC) at the European Organization for Nuclear Research (CERN). The goal of the work was to develop a framework for network optimisation in order to help facilitate Mixed Reality techniques such as 3D collision detection and avoidance, trajectories planning, real-time control, and automatized target approach. The teleoperator was provided with immersive interactions while preserving precise positioning of the robot. These techniques had to be adapted to delays, bandwidth limitation and their volatility in the 4G shared network of the real underground particle accelerator environment. The novel application-layer congestion control with automatic settings was applied for video and point cloud feedback. Twelve automatic setting modes were proposed with algorithms based on the camera frame rate, resolution, point cloud subsampling, network round-trip time and throughput to bandwidth ratio. Each mode was thoroughly characterized to present its specific use-case scenarios and the improvements it brings to the adaptive camera feedback control in teleoperation. Finally, the framework was presented according to which designers can optimize their Human-Robot Interfaces and sensor feedback depending on the network characteristics and task.

The reliability of wireless communication in a network of mobile wireless robot nodes depends on the received radio signal strength (RSS). When the robot nodes are deployed in hostile environments with ionizing radiations (such as in some scientific facilities), there is a possibility that some electronic components may fail randomly (due to radiation effects), which causes problems in wireless connectivity. The objective of this paper is to maximize robot mission capabilities by maximizing the wireless network capacity and to reduce the risk of communication failure. Thus, in this paper, we consider a multi-node wireless tethering structure called the “server-relay-client” framework that uses (multiple) relay nodes in between a server and a client node. We propose a robust stochastic optimization (RSO) algorithm using a multi-sensor-based RSS sampling method at the relay nodes to efficiently improve and balance the RSS between the source and client nodes to improve the network capacity and to provide redundant networking abilities. We use pre-processing techniques, such as exponential moving averaging and spatial averaging filters on the RSS data for smoothing. We apply a receiver spatial diversity concept and employ a position controller on the relay node using a stochastic gradient ascent method for self-positioning the relay node to achieve the RSS balancing task. The effectiveness of the proposed solution is validated by extensive simulations and field experiments in CERN facilities. For the field trials, we used a youBot mobile robot platform as the relay node, and two stand-alone Raspberry Pi computers as the client and server nodes. The algorithm has been proven to be robust to noise in the radio signals and to work effectively even under non-line-of-sight conditions.