Traditional unmanned aerial vehicle path planning methods focus on addressing planning issues in static scenes, struggle to balance optimality and real-time performance, and are prone to local optima. In this paper, we propose an improved deep reinforcement learning approach for UAV path planning in dynamic scenarios. Firstly, we establish a task scenario including an obstacle assessment model and model the UAV’s path planning problem using the Markov Decision Process. We translate the MDP model into the framework of reinforcement learning and design the state space, action space, and reward function while incorporating heuristic rules into the action exploration policy. Secondly, we utilize the Q function approximation of an enhanced D3QN with a prioritized experience replay mechanism and design the algorithm’s network structure based on the TensorFlow framework. Through extensive training, we obtain reinforcement learning path planning policies for both static and dynamic scenes and innovatively employ a visualized action field to analyze their planning effectiveness. Simulations demonstrate that the proposed algorithm can accomplish UAV dynamic scene path planning tasks and outperforms classical methods such as A*, RRT, and DQN in terms of planning effectiveness.
A novel homing guidance law against maneuvering targets based on the deep deterministic policy gradient (DDPG) is proposed. The proposed guidance law directly maps the engagement state information to the acceleration of the interceptor, which is an end-to-end guidance policy. Firstly, the kinematic model of the interception process is described as a Markov decision process (MDP) that is applied to the deep reinforcement learning (DRL) algorithm. Then, an environment of training, state, action, and network structure is reasonably designed. Only the measurements of line-of-sight (LOS) angles and LOS rotational rates are used as state inputs, which can greatly simplify the problem of state estimation. Then, considering the LOS rotational rate and zero-effort-miss (ZEM), the Gaussian reward and terminal reward are designed to build a complete training and testing simulation environment. DDPG is used to deal with the RL problem to obtain a guidance law. Finally, the proposed RL guidance law’s performance has been validated using numerical simulation examples. The proposed RL guidance law demonstrated improved performance compared to the classical true proportional navigation (TPN) method and the RL guidance policy using deep-Q-network (DQN).
22 August 2022 in International Journal of Aerospace Engineering
A homing guidance law for exoatmospheric interceptor based on the Deep Q Network (DQN) algorithm is proposed in this paper. Aiming at the exoatmospheric interception problem, the guidance agent is built with the help of the deep reinforcement learning theory, and the action command is given according to the measurement information of the exoatmospheric interceptor for the accurate interception of the target. The homing guidance problem is first transformed into a Markov decision process, and a three-dimensional (3D) interception scenario is established. Then, the reward function considering the line-of-sight (LOS) rate and the final zero-effort-miss (ZEM) is designed, and the homing guidance problem is transferred to the reinforcement learning framework. After that, DQN is utilized to solve the exoatmospheric interception problem, and the guidance agent is obtained through a large amount of training. Finally, the guidance performance of DQN homing guidance law is verified by numerical simulation examples and compared with the classical true proportional navigation (TPN) guidance law. The results show that the guidance performance of the homing guidance law is better than that of TPN.