16 April 2024 in IEEE Transactions on Systems, Man, and Cybernetics: Systems
In order to solve the two-player Stackelberg differential game (SDG) for the continuous-time nonlinear Markov jump system (MJS), this article defines a unique $Q$ -function and suggests a novel adaptive dynamic programming (ADP) method which is completely independent of system information. First, the optimal policies for the leader and follower are determined from down to the top, and it is further demonstrated that these policies are what make up the Stackelberg–Nash equilibrium point. Then, a novel action-dependent $Q$ -function is established in order to attain completely model-free learning, which is the first attempt for SDG-based nonlinear MJS. Furthermore, the Lyapunov direct approach is employed to guarantee the stability of the closed-loop uncertain nonlinear MJS under the control scheme based on ADP, ensuring uniform ultimate boundedness (UUB). Ultimately, a numerical simulation is presented to validate the efficacy of the aforementioned ADP-based control approach.
31 October 2023 in IEEE Transactions on Automation Science and Engineering
In this paper, to tackle the optimal control problem, we propose a $\mathcal{Q}$ -Learning approach for continuous-time nonlinear systems without any dynamic information. Primarily, the Hamiltonian and optimum cost functions are utilized to articulate the $\mathcal{Q}$ -function of continuous-time affine systems. To reduce the dependence of algorithms on system information, a novel $\mathcal{Q}$ -Learning approach is derived to obtain optimal solutions of nonlinear continuous-time systems without requiring knowledge of either the drift information $p(x)$ or input gain $q(x)$ . To implement this approach, critic and actor neural networks can be iterated alternately using an integral reinforcement learning method to estimate the $\mathcal{Q}$ -function. Furthermore, all signals in closed-loop system are demonstrated to be ultimate uniform bounded (UUB). It is worth noting that there exist rare literatures focused on the optimal control problem of continuous-time nonlinear uncertain systems via the $\mathcal{Q}$ -Learning for actor/critic networks iteration. Finally, two simulations are used to confirm the effectiveness of the proposed algorithm. Note to Practitioners —Nonlinear continuous-time systems, being ubiquitous in engineering practice, are widely employed due to their versatility and effectiveness. Aiming at such systems, a $\mathcal{Q}$ -learning approach with optimal feature is proposed to strengthen control efficiency while reduce costs. However, it is well known that accurately capturing all the dynamic information of the system is a formidable task in practical operation. This defect inevitably weakens the feasibility of model-based control algorithms. Since the $\mathcal{Q}$ -learning algorithm presented in this paper does not require any dynamic knowledge of systems, it is promising enabler in enhancing the effectiveness and flexibility of engineering activities.
25 July 2023 in IEEE Transactions on Fuzzy Systems
In this paper, fractional order non-autonomous system(FONAS) with the input delay and nonlinear terms are considered and investigated using adaptive fuzzy control method based on Q-learning. With the novel estimation model, the defined predictions for the error system determines the weights of the fuzzy logic system(FLS). On this basis, an error derivative-based cost function is introduced, which not only deals with the classic problem that quadratic term cost function is unbounded in infinite time, but also resolves the challenge that the the exponential discount factor cost function fails to stabilize the asymptotically. For the unmeasurable part of the state, the designed fuzzy observer eliminates the restriction on the gain parameters. Furthermore, based on the measured information and the actor-critic architecture of the online training FLSs, the improved adaptive FTC input approximate the optimal control. Utilizing fractional order Lyapunov method, the stability of FONAS with actuator faults is discussed, a sufficient criteria for stability is obtained, easier to perform with convex optimization tools. Finally, numerical simulations are shown to display the effectiveness of the optimal adaptive fuzzy FTC strategy.
17 December 2024 in IEEE Transactions on Systems, Man, and Cybernetics: Systems
This article explores the challenge of triggered optimal control for random differential equations (RDEs) with Markovian switching. We initially address the inherent contradiction between whether to comply with or bypass the event-triggered mechanism. By navigating this challenge, we ensure noise-to-state stability (NSS) for RDEs through event-triggered control (ETC). Furthermore, we establish that random nonlinear systems utilizing self-triggered control (STC) can achieve NSS, by setting a minimum triggering time to prevent Zeno behavior. Lastly, by adopting the adaptive dynamic programming (ADP) strategy, we develop self-triggered optimal control for random systems with Markovian switching, ensuring the uniform ultimate boundedness (UUB) of the signals in all closed-loop systems. This article addresses three key gaps in the field of RDE optimal control, contributing substantially to both theoretical and practical advancements. To demonstrate the method’s feasibility, we include a representative example with simulation results.
21 February 2025 in IEEE Transactions on Systems, Man, and Cybernetics: Systems
In this article, an adaptive dynamic programming (ADP)-based optimal control strategy for a series of fractional-order nonlinear systems (FONS) with unknown control directions is investigated. To eliminate the challenges posed by unknown control directions, fractional-order Nussbaum-type functions are introduced for FONS, expanding the range of possible applications. Additionally, since system performance is compromised by disturbances, a fractional-order disturbance observer is designed to counteract the effects of external disturbances and enhance system robustness. Furthermore, differential geometric methods are employed to investigate FONS, constructing appropriate diffeomorphism that provide equivalent systems for decoupled linearization. Then, an optimal control method is studied for a class of strictly feedback FONS, in which Nussbaum-type functions are combined with ADP theory during the backstepping design process. Finally, based on fractional Lyapunov stability theory and backstepping method, it is guaranteed that all signals of the closed-loop FONS are uniformly ultimately bounded (UUB). Numerical simulation and a PMSM model are utilized to verify the effectiveness of the presented method.
14 March 2025 in IEEE Transactions on Circuits and Systems I: Regular Papers
In this paper, the problem of adaptive fault-tolerant optimal control is investigated for nonstrict-feedback nonlinear systems with deferred output and performance constraints (DOPCs). By skillfully constructing the shifting transformation and finite-time constraining function, a novel error-dependent barrier function is innovated to achieve the preassigned time tracking performance without the conservative initial limitations, helping the property of DOPCs be preserved. Neural networks (NNs)-based reinforcement learning (RL) is exploited, which takes advantage of approximation means to solve the Hamiltonian equation under the backstepping framework, optimally reconciling the tracking performance and control behavior. By utilizing NNs to online estimate the unaware faults and uncertainty, an adaptive fault-tolerant optimal controller is designed. Finally, the efficiency of the proposed method is confirmed by simulations.