7 July 2022 in IEEE Transactions on Intelligent Vehicles
This article investigates the attitude control problem for a novel foldable wave-energy powered autonomous underwater vehicle (FWEPAUV) with model uncertainties and external disturbances. To find a trade-off between the control accuracy and control energy consumption, a robust performance-prescribed optimized backstepping control (RPOBC) scheme is designed for optimal tracking problem by combining the reinforcement learning (RL) technique. The performance-prescribed mechanism is devised to ensure the prescribed tracking accuracy. To make the controller robust against the model uncertainties and external disturbances, an auxiliary system with a novel cost function is introduced into the optimized backstepping (OB) framework. Besides, the theoretical analysis indicates that the RPOBC scheme can guarantee tracking errors converge to predefined performance with an optimal control performance. At last, the simulation results are presented and analyzed to demonstrate the effectiveness of the designed RPOBC strategy.
12 February 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Offline reinforcement learning (RL) aims at learning an optimal policy from a static offline data set, without interacting with the environment. However, the theoretical understanding of the existing offline RL methods needs further studies, among which the conservatism of the learned Q-function and the learned policy is a major issue. In this article, we propose a simple and efficient offline RL with relaxed conservatism (ORL-RC) framework for addressing this concern by learning a Q-function that is close to the true Q-function under the learned policy. The conservatism of learned Q-functions and policies of offline RL methods is analyzed. The analysis results support that the conservatism can lead to policy performance degradation. We establish the convergence results of the proposed ORL-RC, and the bounds of learned Q-functions with and without sampling errors, respectively, suggesting that the gap between the learned Q-function and the true Q-function can be reduced by executing the conservative policy improvement. A practical implementation of ORL-RC is presented and the experimental results on the D4RL benchmark suggest that ORL-RC exhibits superior performance and substantially outperforms existing state-of-the-art offline RL methods.
26 April 2024 in IEEE Transactions on Cybernetics
Offline reinforcement learning (offline RL) aims to find task-solving policies from prerecorded datasets without online environment interaction. It is unfortunate that extrapolation errors can cause over-optimistic $Q$ -value estimates when learning with a fixed dataset, limiting the performance of the learned policy. To tackle this issue, this article proposes an offline actor–critic with behavior value regularization (OAC-BVR) method. In the policy evaluation stage, the difference between the $Q$ -function and the value of the behavior policy is considered as the regularization term, driving the learned value function to approach the value of the behavior policy. The convergence of the proposed policy evaluation with behavior value regularization (PE-BVR) and the value function difference are analyzed, respectively. Compared with existing offline actor–critic methods, the proposed OAC-BVR method integrates the value of the behavior policy, thereby simultaneously alleviating over-optimistic $Q$ -value estimates and reducing $Q$ -function bias. Experimental results on the D4RL MuJoCo and Maze2d datasets demonstrate the validity of the proposed PE-BVR and the performance advantage of OAC-BVR over the state-of-the-art offline RL algorithms. The code of OAC-BVR is available at https://github.com/LongyangHuang/OAC-BVR.
31 January 2023 in Ocean Engineering
7 September 2023 in IEEE Transactions on Neural Networks and Learning Systems
In offline actor–critic (AC) algorithms, the distributional shift between the training data and target policy causes optimistic $Q$ value estimates for out-of-distribution (OOD) actions. This leads to learned policies skewed toward OOD actions with falsely high $Q$ values. The existing value-regularized offline AC algorithms address this issue by learning a conservative value function, leading to a performance drop. In this article, we propose a mild policy evaluation (MPE) by constraining the difference between the $Q$ values of actions supported by the target policy and those of actions contained within the offline dataset. The convergence of the proposed MPE, the gap between the learned value function and the true one, and the suboptimality of the offline AC with MPE are analyzed, respectively. A mild offline AC (MOAC) algorithm is developed by integrating MPE into off-policy AC. Compared with existing offline AC algorithms, the value function gap of MOAC is bounded by the existence of sampling errors. Moreover, in the absence of sampling errors, the true state value function can be obtained. Experimental results on the D4RL benchmark dataset demonstrate the effectiveness of MPE and the performance superiority of MOAC compared to the state-of-the-art offline reinforcement learning (RL) algorithms.
29 October 2024 in IEEE Transactions on Neural Networks and Learning Systems
The challenge of the exploration–exploitation dilemma persists in off-policy reinforcement learning (RL) algorithms, impeding the improvement of policy performance and sample efficiency. To tackle this challenge, a novel historical decision-making regularized maximum entropy (HDMRME) RL algorithm is developed to strike the balance between exploration and exploitation. Built upon the maximum entropy RL framework, the historical decision-making regularization method is proposed to enhance the exploitation capability of RL policies. The theoretical analysis involves proving the convergence of HDMRME, investigating the tradeoff between exploration and exploitation of HDMRME, examining the disparity between the Q-function learned through HDMRME and the classic one, and analyzing the suboptimality of the trained policy. The performance of HDMRME is evaluated across various continuous-action control tasks from Mujoco and OpenAI Gym platforms. Comparative experiments demonstrate that HDMRME exhibits superior sample efficiency and achieves more competitive performance compared with other state-of-the-art RL algorithms.