The growing demand for robots able to act autonomously in complex scenarios has widely accelerated the introduction of Reinforcement Learning (RL) in robots control applications. However, the trial and error intrinsic nature of RL may result in long training time on real robots and, moreover, it may lead to dangerous outcomes. While simulators are useful tools to accelerate RL training and to ensure safety, they often are provided only with an approximated model of robot dynamics and of its interaction with the surrounding environment, thus resulting in what is called the reality gap (RG): a mismatch of simulated and real control-law performances caused by the inaccurate representation of the real environment in simulation. The most undesirable result occurs when the controller learnt in simulation fails the task on the real robot, thus resulting in an unsuccessful sim-to-real transfer. The goal of the present survey is threefold: (1) to identify the main approaches to face the RG problem in the context of robot control with RL, (2) to point out their shortcomings, and (3) to outline new potential research areas.

Optimal tuning of particle accelerators is a challenging task. Many different approaches have been proposed in the past to solve two main problems—attainment of an optimal working point and performance recovery after machine drifts. The most classical model-free techniques (e.g., Gradient Ascent or Extremum Seeking algorithms) have some intrinsic limitations. To overcome those limitations, Machine Learning tools, in particular Reinforcement Learning (RL), are attracting more and more attention in the particle accelerator community. We investigate the feasibility of RL model-free approaches to align the seed laser, as well as other service lasers, at FERMI, the free-electron laser facility at Elettra Sincrotrone Trieste. We apply two different techniques—the first, based on the episodic Q-learning with linear function approximation, for performance optimization; the second, based on the continuous Natural Policy Gradient REINFORCE algorithm, for performance recovery. Despite the simplicity of these approaches, we report satisfactory preliminary results, that represent the first step toward a new fully automatic procedure for the alignment of the seed laser to the electron beam. Such an alignment is, at present, performed manually.

We address the problem of coverage of a given environment by a team of quadrotor unmanned aerial vehicles (UAVs) with limited sensing capabilities. The objective is to develop a control strategy for spreading the UAVs across the environment for monitoring purposes. We do not assume the availability of a central controller for UAV coordination, and the UAVs cooperate by exchanging information via a communication network. By considering possible cyberattacks acting on the UAV network, we propose a resilient distributed control scheme such that the UAVs locally compute their position set points to achieve an overall coverage configuration despite the malicious behaviors of some attacked UAVs. By relying on the Voronoi diagram-based coverage technique, we first address possible attack scenarios that can deteriorate a covering mission. Then, we consider a generalized Voronoi diagram-based coverage scheme that shows resilience against cyberattack scenarios. We also design an adaptive control strategy ensuring the convergence of the UAVs to a desired coverage configuration, whereas the UAV parameters (inertial matrix, drag coefficient, and so on) are not known. Mathematical analysis, simulation, and real-world experimental results show the effectiveness of the proposed distributed resilient control scheme.

The optimization of particle accelerators is a challenging task, and many different approaches have been proposed in years, to obtain an optimal tuning of the plant and to keep it optimally tuned despite drifts or disturbances. Indeed, the classical model-free approaches (such as Gradient Ascent or Extremum Seeking algorithms) have intrinsic limitations. To overcome those limitations, Machine Learning techniques, in particular, the Reinforcement Learning, are attracting more and more attention in the particle accelerator community. The purpose of this paper is to apply a Reinforcement Learning model-free approach to the alignment of a seed laser, based on a rather general target function depending on the laser trajectory. The study focuses on the alignment of the lasers at FERMI, the free-electron laser facility at Elettra Sincrotrone Trieste. In particular, we employ Q-learning with linear function approximation and report experimental results obtained in two setups, which are the actual setups where the final application has to be deployed. Despite the simplicity of the approach, we report satisfactory preliminary results, that represent the first step toward a fully automatic procedure for seed laser to the electron beam. Such a superimposition is, at present, performed manually.

Estimating the wear of the single electrical parts of a home appliance without resorting to a large number of sensors is desirable for ensuring a proper level of maintenance by the manufacturers. Deep learning techniques can be effective tools for such estimation from relatively poor measurements, but their computational demands must be carefully considered, for the actual deployment. In this work, we employ one-dimensional Convolutional Neural Networks and Long Short-Term Memory networks to infer the status of some electrical components of different models of washing machines, from the electrical signals measured at the plug. These tools are trained and tested on a large dataset (502 washing cycles $$\approx$$≈ 1000 h) collected from four different washing machines and are carefully designed in order to comply with the memory constraints imposed by available hardware selected for a real implementation. The approach is end-to-end; i.e., it does not require any feature extraction, except the harmonic decomposition of the electrical signals, and thus it can be easily generalized to other appliances.