In this article, we address the difficulty of controlling unmanned surface vehicles (USVs) under unforeseeable and unobservable external disturbances using model-based reinforcement learning (MBRL) without human’s prior knowledge. A novel MBRL approach, filtered probabilistic model predictive control (FPMPC) is proposed to iteratively learn the USV model and an MPC-based policy in a probabilistic way through trial-and-error interactions. Compared with existing MBRL approaches that model the unobservable disturbances as system noise, FPMPC introduces a Bayesian filter process to implicitly translate the system dynamics to a partially-observed Markov decision process to present those disturbances as hidden states. An adaptive sample selection is proposed to remove the redundant learning samples based on the filter belief. Equipped with bias compensation and parallel computation, an FPMPC system, specific for USV, is developed. Evaluated by both position holding and target reaching tasks in a real USV data-driven simulation, FPMPC shows its significant superiority in control performances, generalization capability, and sample efficiency under large disturbances compared with the baseline approaches.

Autonomous vehicles have received considerable attention, yet high-speed path following control remains a critical and challenging issue. At high speeds, achieving perfect control based on an accurate dynamic model of autonomous vehicles is difficult due to the nonlinear model parameters that change at the driving limit. Moreover, the strong coupling relationship between longitudinal control and lateral control also affects control performance. To address these challenges, a reinforcement learning-based control approach (RL-controller) is developed in this paper and applied to the high-speed path following task. The proposed RL-controller overcomes the challenges of explicit modeling and tight coupling control. Firstly, to obtain efficient learning features, our RL-controller is developed using a deep soft actor-critic method that incorporates multi-layer dense connection networks with skip connections and feature reuse. Secondly, an estimation method of the lane curvature is developed, which is chosen as prior knowledge to augment the input state of reinforcement learning. We model the lane using a transformer-based network structure that utilizes a third-order polynomial and train the model on our self-collected dataset. Thirdly, intuitive reward functions are built from the cross-track error, the angle error and the sideslip angle. At last, we verify our proposed RL-controller utilizing an open autonomous driving simulator. Compared with imitation learning and model-based optimal control methods, our RL-controller outperforms them in terms of the driving speed and compound error.

In this article, we propose a novel reinforcement learning (RL) approach specialized for autonomous boats: sample‐efficient probabilistic model predictive control (SPMPC), to iteratively learn control policies of boats in real ocean environments without human prior knowledge. SPMPC addresses difficulties arising from large uncertainties in this challenging application and the need for rapid adaptation to dynamic environmental conditions, and the extremely high cost of exploring and sampling with a real vessel. SPMPC combines a Gaussian process model and model predictive control under a model‐based RL framework to iteratively model and quickly respond to uncertain ocean environments while maintaining sample efficiency. A SPMPC system is developed with features including quadrant‐based action search rule, bias compensation, and parallel computing that contribute to better control capabilities. It successfully learns to control a full‐sized single‐engine boat equipped with sensors measuring GPS position, speed, direction, and wind, in a real‐world position holding task without models from human demonstration.

Graphical Abstract In this study, a new value function based Reinforcement learning (RL) algorithm, Local Update Dynamic Policy Programming (LUDPP), is proposed. It exploits the nature of smooth policy update using Kullback–Leibler divergence to update its value function locally and considerably reduces the computational complexity. We firstly investigated the learning performance of LUDPP and other algorithms without smooth policy update for tasks of pendulum swing up and n DOFs manipulator reaching in simulation. Only LUDPP could efficiently and stably learn good control policies in high dimensional systems with limited number of training samples. In real word application, we applied LUDPP to control Pneumatic Artificial Muscles (PAMs) driven robots without the knowledge of model which is challenging for traditional methods due to the high nonlinearities of PAM’s air pressure dynamics and mechanical structure. LUDPP successfully achieved one finger control of Shadow Dexterous Hand, a PAM-driven humanoid robot hand, with far lower computational resource compared with other conventional value function based RL algorithms.

In this article, a novel reinforcement learning (RL) approach, continuous dynamic policy programming (CDPP), is proposed to tackle the issues of both learning stability and sample efficiency in the current RL methods with continuous actions. The proposed method naturally extends the relative entropy regularization from the value function-based framework to the actor–critic (AC) framework of deep deterministic policy gradient (DDPG) to stabilize the learning process in continuous action space. It tackles the intractable softmax operation over continuous actions in the critic by Monte Carlo estimation and explores the practical advantages of the Mellowmax operator. A Boltzmann sampling policy is proposed to guide the exploration of actor following the relative entropy regularized critic for superior learning capability, exploration efficiency, and robustness. Evaluated by several benchmark and real-robot-based simulation tasks, the proposed method illustrates the positive impact of the relative entropy regularization including efficient exploration behavior and stable policy update in RL with continuous action space and successfully outperforms the related baseline approaches in both sample efficiency and learning stability.