In this paper, we propose an end-to-end modular reinforcement learning architecture for a navigation task in complex dynamic environments with rapidly moving obstacles. In this architecture, the main task is divided into two subtasks: local obstacle avoidance and global navigation. For obstacle avoidance, we develop a two-stream Q-network, which processes spatial and temporal information separately and generates action values. The global navigation subtask is resolved by a conventional Q-network framework. An online learning network and an action scheduler are introduced to first combine two pretrained policies, and then continue exploring and optimizing until a stable policy is obtained. The two-stream Q-network obtains better performance than the conventional deep Q-learning approach in the obstacle avoidance subtask. Experiments on the main task demonstrate that the proposed architecture can efficiently avoid moving obstacles and complete the navigation task at a high success rate. The modular architecture enables parallel training and also demonstrates good generalization capability in different environments.

In this article, a reinforcement learning (RL)-based strategy for unmanned surface vehicle (USV) path following control is developed. The proposed method learns integrated guidance and heading control policy, which directly maps the USV’s navigation states to motor control commands. By introducing a twin-critic design and an integral compensator to the conventional deep deterministic policy gradient (DDPG) algorithm, the tracking accuracy and robustness of the controller can be significantly improved. Moreover, a pretrained neural network-based USV model is built to help the learning algorithm efficiently deal with unknown nonlinear dynamics. The self-learning and path following capabilities of the proposed method were validated in both simulations and real sea experiments. The results show that our control policy can achieve better performance than a traditional cascade control policy and a DDPG-based control policy.

Existing approaches to constrained-input optimal control problems mainly focus on systems with input saturation, whereas other constraints, such as combined inequality constraints and state-dependent constraints, are seldom discussed. In this article, a reinforcement learning (RL)-based algorithm is developed for constrained-input optimal control of discrete-time (DT) systems. The deterministic policy gradient (DPG) is introduced to iteratively search the optimal solution to the Hamilton-Jacobi-Bellman (HJB) equation. To deal with input constraints, an action mapping (AM) mechanism is proposed. The objective of this mechanism is to transform the exploration space from the subspace generated by the given inequality constraints to the standard Cartesian product space, which can be searched effectively by existing algorithms. By using the proposed architecture, the learned policy can output control signals satisfying the given constraints, and the original reward function can be kept unchanged. In our study, the convergence analysis is given. It is shown that the iterative algorithm is convergent to the optimal solution of the HJB equation. In addition, the continuity of the iterative estimated Q-function is investigated. Two numerical examples are provided to demonstrate the effectiveness of our approach.