In this article, we propose a simple, practical, and intuitive approach to improve the performance of a conventional controller in uncertain environments using deep reinforcement learning while maintaining safe operation. Our approach is motivated by the observation that conventional controllers in industrial motion control value robustness over adaptivity to deal with different operating conditions and are suboptimal as a consequence. Reinforcement learning, on the other hand, can optimize a control signal directly from input–output data and thus adapts to operational conditions but lacks safety guarantees, impeding its use in industrial environments. To realize adaptive control using reinforcement learning in such conditions, we follow a residual learning methodology, where a reinforcement learning algorithm learns corrective adaptations to a base controller’s output to increase optimality. We investigate how constraining the residual agent’s actions enables to leverage the base controller’s robustness to guarantee safe operation. We detail the algorithmic design and propose to constrain the residual actions relative to the base controller to increase the method’s robustness. Building on Lyapunov stability theory, we prove stability for a broad class of mechatronic closed-loop systems. We validate our method experimentally on a slider crank setup and investigate how the constraints affect the safety during learning and optimality after convergence.
Cascaded control structures are prevalent in industrial systems with many disturbances to obtain stable control but are cumbersome and challenging to tune. In this work, we propose cascaded constrained residual reinforcement learning (RL), an intuitive method that allows to improve the performance of a cascaded control structure while maintaining safe operation at all times. We draw inspiration from the constrained residual RL framework, in which a constrained reinforcement learning agent learns corrective adaptations to a base controller’s output to increase optimality. We first revisit the interplay between the residual agent and the baseline controller and subsequently extend this to the cascaded case. We analyze the differences and challenges this structure brings and derive some principle insights from this into the stability and operation of the cascaded residual architecture. Next, we propose a novel actor structure to enable efficient learning under the cascaded setting. We show that the standard algorithm is suboptimal for application to cascaded control structures and validate our method on a high-fidelity simulator of a dual motor drivetrain, resulting in a performance improvement of 14.7% on average, with only a minor decrease in performance occurring during the training phase. We study the different principles constituting the method and examine and validate their contribution to the algorithm’s performance under the considered cascaded control structure.
The limited ability of deep learning models to generalize to regions outside of the training data distribution impedes their use for mechatronic applications, with high requirements of safe and robust operation in multiple operating conditions. We draw inspiration from the fields of Multi-Task Learning and distributed computing and propose an adaptation to Elastic Averaging Stochastic Gradient Descent that makes it possible to leverage upon the information of a fleet of systems to extend the generalization capabilities of the individual models, without having access to the full dataset. We demonstrate in simulation that our method enables models to generalize even outside of the joint training data distribution of the fleet. We compare our method to vanilla Elastic Averaging Stochastic Gradient Descent and demonstrate the importance of our adaptation for convergence in the Multi-Task Learning setting. Finally we investigate the interplay between the elastic force and the individual gradients in the update rules as a determining force for its performance.
17 June 2026 in Engineering Applications of Artificial Intelligence