Active tracking control is essential for UAVs to perform autonomous operations in GPS-denied environments. In the active tracking task, UAVs take high-dimensional raw images as input and execute motor actions to actively follow the dynamic target. Most research focuses on three-stage methods, which entail perception first, followed by high-level decision-making based on extracted spatial information of the dynamic target, and then UAV movement control, using a low-level dynamic controller. Perception methods based on deep neural networks are powerful but require considerable effort for manual ground truth labeling. Instead, we unify the perception and decision-making stages using a high-level controller and then leverage deep reinforcement learning to learn the mapping from raw images to the high-level action commands in the V-REP-based environment, where simulation data are infinite and inexpensive. This end-to-end method also has the advantages of a small parameter size and reduced effort requirements for parameter turning in the decision-making stage. The high-level controller, which has a novel architecture, explicitly encodes the spatial and temporal features of the dynamic target. Auxiliary segmentation and motion-in-depth losses are introduced to generate denser training signals for the high-level controller’s fast and stable training. The high-level controller and a conventional low-level PID controller constitute our hierarchical active tracking control framework for the UAVs’ active tracking task. Simulation experiments show that our controller trained with several augmentation techniques sufficiently generalizes dynamic targets with random appearances and velocities, and achieves significantly better performance, compared with three-stage methods.

Deep reinforcement learning technology applied to three-dimensional Unmanned Aerial Vehicle (UAV) air game maneuver decision-making often results in low utilization efficiency of training data and algorithm convergence difficulties. To address these issues, this study proposes an expert experience storage mechanism that improves the algorithm’s performance with less experience replay time. Based on this mechanism, a maneuver decision algorithm using the Dueling Double Deep Q Network is introduced. Simulation experiments demonstrate that the proposed mechanism significantly enhances the algorithm’s performance by reducing the experience by 81.3% compared to the prioritized experience replay mechanism, enabling the UAV agent to achieve a higher maximum average reward value. The experimental results suggest that the proposed expert experience storage mechanism improves the algorithm’s performance with less experience replay time. Additionally, the proposed maneuver decision algorithm identifies the optimal policy for attacking target UAVs using different fixed strategies.

Autonomous manoeuvre decision-making is essential for enhancing the survivability and operational effectiveness of unmanned aerial vehicles in high-risk and dynamic air combat scenarios. To address the limitations of traditional air combat decision-making methods in dealing with complex and rapidly changing environments, this paper proposes an autonomous air combat decision-making algorithm based on hybrid temporal difference error-reward prioritised experience replay with twin delayed deep deterministic policy gradient. This algorithm constructs a closed-loop learning system from environmental interaction to policy optimisation, addressing the key challenges of slow convergence and insufficient identification of critical tactical decisions in autonomous air combat. A hybrid priority metric leveraging reward backpropagation and temporal difference error filter is introduced to optimise the learning of high-value experiences while balancing sample diversity and the reuse of critical experiences. To reduce excessive trial and error in the initial phase, an integrated reward function combining task rewards and auxiliary guidance rewards is designed using the reward reshaping method to guide the agent on how to choose a manoeuvre strategy. Based on the established three-dimensional close-range air combat game model, simulation validations were conducted for both basic manoeuvre and expert system engagements. The results demonstrate that the proposed autonomous air combat manoeuvre decision-making algorithm exhibits higher learning efficiency and convergence stability. It can rapidly identify high-value manoeuvres and effectively formulate rational yet superior tactical strategies in the face of complex battlefield scenarios, demonstrating obvious benefits in enhancing combat effectiveness and tactical adaptability.