Adaptive dynamic programming (ADP) and reinforcement learning are quite relevant to each other when performing intelligent optimization. They are both regarded as promising methods involving important components of evaluation and improvement, at the background of information technology, such as artificial intelligence, big data, and deep learning. Although great progresses have been achieved and surveyed when addressing nonlinear optimal control problems, the research on robustness of ADP-based control strategies under uncertain environment has not been fully summarized. Hence, this survey reviews the recent main results of adaptive-critic-based robust control design of continuous-time nonlinear systems. The ADP-based nonlinear optimal regulation is reviewed, followed by robust stabilization of nonlinear systems with matched uncertainties, guaranteed cost control design of unmatched plants, and decentralized stabilization of interconnected systems. Additionally, further comprehensive discussions are presented, including event-based robust control design, improvement of the critic learning rule, nonlinear H _∞ control design, and several notes on future perspectives. By applying the ADP-based optimal and robust control methods to a practical power system and an overhead crane plant, two typical examples are provided to verify the effectiveness of theoretical results. Overall, this survey is beneficial to promote the development of adaptive critic control methods with robustness guarantee and the construction of higher level intelligent systems.

Reinforcement learning (RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming (ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively. Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks, showing how they promote ADP formulation significantly. Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has demonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

The wastewater treatment is an important avenue of resources cyclic utilization when coping with the modern urban diseases. However, there always exist obvious nonlinearities and uncertainties within wastewater treatment systems, such that it is difficult to accomplish proper optimization objectives towards these complex unknown platforms. In this paper, a data-driven iterative adaptive critic (IAC) strategy is developed to address the nonlinear optimal control problem. The iterative algorithm is constructed with a general framework, followed by convergence analysis and neural network implementation. Remarkably, the derived IAC control policy with an additional steady control input is also applied to a typical wastewater treatment plant, rendering that the dissolved oxygen concentration and the nitrate level are maintained at desired setting points. When compared with the incremental proportional-integral-derivative method, it is found that faster response and less oscillation can be obtained during the IAC control process.

The model-free optimal control problem of general discrete-time nonlinear systems is considered in this paper, and a data-based policy gradient adaptive dynamic programming (PGADP) algorithm is developed to design an adaptive optimal controller method. By using offline and online data rather than the mathematical system model, the PGADP algorithm improves control policy with a gradient descent scheme. The convergence of the PGADP algorithm is proved by demonstrating that the constructed Q-function sequence converges to the optimal Q-function. Based on the PGADP algorithm, the adaptive control method is developed with an actor-critic structure and the method of weighted residuals. Its convergence properties are analyzed, where the approximate Q-function converges to its optimum. Computer simulation results demonstrate the effectiveness of the PGADP-based adaptive control method.

In this paper, the problem of adaptive neural state-feedback tracking control is considered for a class of stochastic nonstrict-feedback nonlinear switched systems with completely unknown nonlinearities. In the design procedure, the universal approximation capability of radial basis function neural networks is used for identifying the unknown compounded nonlinear functions, and a variable separation technique is employed to overcome the design difficulty caused by the nonstrict-feedback structure. The most outstanding novelty of this paper is that individual Lyapunov function of each subsystem is constructed by flexibly adopting the upper and lower bounds of the control gain functions of each subsystem. Furthermore, by combining the average dwell-time scheme and the adaptive backstepping design, a valid adaptive neural state-feedback controller design algorithm is presented such that all the signals of the switched closed-loop system are in probability semiglobally uniformly ultimately bounded, and the tracking error eventually converges to a small neighborhood of the origin in probability. Finally, the availability of the developed control scheme is verified by two simulation examples.

The self-learning optimal regulation for discrete-time nonlinear systems under event-driven formulation is investigated. An event-based adaptive critic algorithm is developed with convergence discussion of the iterative process. The input-to-state stability (ISS) analysis for the present nonlinear plant is established. Then, a suitable triggering condition is proved to ensure the ISS of the controlled system. An iterative dual heuristic dynamic programming (DHP) strategy is adopted to implement the event-driven framework. Simulation examples are carried out to demonstrate the applicability of the constructed method. Compared with the traditional DHP algorithm, the even-based algorithm is able to substantially reduce the updating times of the control input while still maintains an impressive performance.