The growing demand for robots able to act autonomously in complex scenarios has widely accelerated the introduction of Reinforcement Learning (RL) in robots control applications. However, the trial and error intrinsic nature of RL may result in long training time on real robots and, moreover, it may lead to dangerous outcomes. While simulators are useful tools to accelerate RL training and to ensure safety, they often are provided only with an approximated model of robot dynamics and of its interaction with the surrounding environment, thus resulting in what is called the reality gap (RG): a mismatch of simulated and real control-law performances caused by the inaccurate representation of the real environment in simulation. The most undesirable result occurs when the controller learnt in simulation fails the task on the real robot, thus resulting in an unsuccessful sim-to-real transfer. The goal of the present survey is threefold: (1) to identify the main approaches to face the RG problem in the context of robot control with RL, (2) to point out their shortcomings, and (3) to outline new potential research areas.

To objectively evaluate the image quality obtained with toric intraocular lenses (IOLs) when misaligned from the intended axis. University Eye Clinic and the Department of Industrial and Information Engineering, University of Trieste, Trieste, Italy. Experimental study. An experimental optoelectronic test bench was created. It consisted of a high-resolution monitor to project target images and an artificial eye. The system simulates the optical and geometric characteristics of the human eye with an implanted toric IOL. A 3.00 diopters corneal astigmatism was simulated. Images reproduced by the optical system were captured according to different IOL axis positions. The quality of each image was analyzed using the visual information fidelity (VIF) criterion. The VIF reduction was calculated at each IOL rotational step. A 5-degree IOL axis rotation from the intended position determined a decay in the image quality of 7.03%. Ten degrees of IOL rotation caused an 11.09% decay of relative VIF value. For a 30-degree rotation, the VIF decay value was 45.85%. Finally, the image decay with no toric correction was 56.70%. The more the objective quality of the image decays progressively, the further the axis of the IOL is rotated from its intended position. The reduction in image quality obtained after 30 degrees of toric IOL rotation was less than 50% and after 45 degrees, the image quality was the same as that of no toric correction.

Optimal tuning of particle accelerators is a challenging task. Many different approaches have been proposed in the past to solve two main problems—attainment of an optimal working point and performance recovery after machine drifts. The most classical model-free techniques (e.g., Gradient Ascent or Extremum Seeking algorithms) have some intrinsic limitations. To overcome those limitations, Machine Learning tools, in particular Reinforcement Learning (RL), are attracting more and more attention in the particle accelerator community. We investigate the feasibility of RL model-free approaches to align the seed laser, as well as other service lasers, at FERMI, the free-electron laser facility at Elettra Sincrotrone Trieste. We apply two different techniques—the first, based on the episodic Q-learning with linear function approximation, for performance optimization; the second, based on the continuous Natural Policy Gradient REINFORCE algorithm, for performance recovery. Despite the simplicity of these approaches, we report satisfactory preliminary results, that represent the first step toward a new fully automatic procedure for the alignment of the seed laser to the electron beam. Such an alignment is, at present, performed manually.

The objective of this work consists in the offline approximation of possibly discontinuous model predictive control laws for nonlinear discrete-time systems, while enforcing hard constraints on state and input variables. Obtaining an offline approximation of the receding horizon control law may lead to a very significant reduction of the online computational burden with respect to algorithms based on iterated optimization, thus allowing the application to fast dynamics plants. The proposed approximation scheme allows to cope with discontinuous control laws, such as those arising from constrained nonlinear finite horizon optimal control problems. A detailed stability analysis of the closed-loop system driven by the approximated state-feedback controller shows that the devised technique guarantees the input-to-state practical stability with respect to the (non-fading) approximation-induced errors. Two examples are provided to show the effectiveness of the method when the approximator is chosen either as a discontinuous nearest point function or as a smooth neural network.

Given a static plant described by a differentiable input-output function, which is completely unknown, but whose Jacobian takes values in a known polytope in the matrix space, this paper considers the problem of tuning (i.e., driving to a desired value) the output, by suitably choosing the input. It is shown that, if the polytope is robustly nonsingular (or has full rank, in the nonsquare case), then a suitable tuning scheme drives the output to the desired point. The proof exploits a Lyapunov-like function and applies a well-known game-theoretic result, concerning the existence of a saddle point for a min-max zero-sum game. When the plant output is represented in an implicit form, it is shown that the same result can be obtained, resorting to a different Lyapunov-like function. The case in which proper input or output constraints must be enforced during the transient is considered as well. Some application examples are proposed to show the effectiveness of the approach.