4 October 2022 in IEEE Transactions on Automatic Control
Control Lyapunov function is a central tool in stabilization. It generalizes an abstract energy function – a Lyapunov function – to the case of controlled systems. It is a known fact that most control Lyapunov functions are non-smooth – so is the case in non-holonomic systems, like wheeled robots and cars. Frameworks for stabilization using non-smooth control Lyapunov functions exist, like Dini aiming and steepest descent. This work generalizes the related results to the stochastic case. As the groundwork, sampled control scheme is chosen in which control actions are computed at discrete moments in time using discrete measurements of the system state. In such a setup, special attention should be paid to the sample-to-sample behavior of the control Lyapunov function. A particular challenge here is a random noise acting on the system. The central result of this work is a theorem that states, roughly, that if there is a, generally non-smooth, control Lyapunov function, the given stochastic dynamical system can be practically stabilized in the sample-and-hold mode meaning that the control actions are held constant within sampling time steps. A particular control method chosen is based on Moreau-Yosida regularization, in other words, inf-convolution of the control Lyapunov function, but the overall framework is extendable to further control schemes. It is assumed that the system noise be bounded almost surely, although the case of unbounded noise is briefly addressed.
27 February 2023 in IEEE Transactions on Automatic Control
A common setting of reinforcement learning (RL) is a Markov decision process (MDP) in which the environment is a stochastic discrete-time dynamical system. Whereas MDPs are suitable in such applications as video games or puzzles, physical systems are time continuous. A general variant of RL is of digital format, where updates of the value (or cost) and policy are performed at discrete moments in time. The agent–environment loop then amounts to a sampled system, whereby sample-and-hold is a specific case. In this article, we propose and benchmark two RL methods suitable for sampled systems. Specifically, we hybridize model predictive control with critics learning the optimal Q- and value (or cost-to-go) function. Optimality is analyzed and performance comparison is done in an experimental case study with a mobile robot.
17 August 2023 in IEEE Access
Online actor-critic reinforcement learning is concerned with training an agent on-the-fly via dynamic interaction with the environment. Due to the specifics of the application, it is not generally possible to perform long pre-training, as it is commonly done in off-line, tabular or Monte-Carlo mode. Such applications may be found more frequently in industry, rather than in pure digital fields, such as cloud services, video games, database management, etc., where reinforcement learning has been demonstrating success. Stability of the closed-loop of the agent plus the environment is a major challenge here, and not only in terms of the environment safety and integrity, but also in terms of sparing resources on failed training episodes. In this paper, we tackle the problem of environment stability under an actor-critic reinforcement learning agent by integration of the Lyapunov stability theory tools. Under the presented approach, the closed-loop stability is secured in all episodes without pre-training. It was observed in a case study with a mobile robot that the suggested agent could always successfully achieve the control goal, while significantly reducing the cost. While many approaches may be exploited for mobile robot control, we suggest that the experiments showed the promising potential of actor-critic reinforcement learning agents based on Lyapunov-like constraints. The presented methodology may be utilized in safety-critical, industrial applications where stability is necessary.
26 May 2025 in IEEE Control Systems Letters
Reinforcement learning (RL) benchmarking has long relied on learning curves and cumulative reward tables, yet these metrics fail to capture critical design challenges, such as environment sensitivity, robustness, and reproducibility. This paper introduces a novel benchmarking paradigm that rigorously evaluates these understudied properties through a generative framework grounded in converse optimality. By constructing families of control problems with provably optimal solutions, our method enables statistical hypothesis testing of an algorithms response to environment perturbations both during training and post-deployment. Central to this approach is a problem generator that synthesizes parametric variations of dynamics and costs, paired with a sensitivity evaluation pipeline. To illustrate the frameworks usefulness, we apply it to compare environment sensitivity between Adam and RAdam optimizers – a case study designed to clarify how exactly the methodology provides the means to evaluate reinforcement learning algorithms in an informative way. This work contributes to RL evaluation by shifting focus from sample efficiency alone to methodologies that assess algorithms under conditions reflective of practical engineering challenges.