This study proposes an unmanned combat aerial vehicle (UCAV)-oriented hierarchical reinforcement learning framework to address the temporal abstraction challenge in autonomous within-visual-range air combat (WVRAC) for UCAVs. The incorporation of maximum-entropy objectives within the MEOL framework facilitates the optimization of both autonomous low-level tactical discovery and high-level option selection. At the low level, three tactical policies (angle, snapshot, and energy tactics) are designed with reward functions informed by expert knowledge, while the high-level policy dynamically terminates current tactics and selects new ones through sparse reward learning, thus overcoming the limitations of fixed-duration tactical execution. Furthermore, a novel automatic curriculum generation mechanism based on Wasserstein Generative Adversarial Networks (WGANs) is introduced to enhance training efficiency and adaptability to diverse initial combat conditions. Extensive experiments conducted in UCAV air combat simulations have shown that MEOL not only achieves significantly better win rates than other policies when training against rule-based opponents, but also that MEOC achieves superior results in tests against tactical intra-option policies as well as other option learning policies. The framework facilitates dynamic termination and switching of tactics, thereby addressing the limitations of fixed-duration hierarchical methods. Ablation studies confirm the effectiveness of WGAN-based curricula in accelerating policy convergence. Additionally, the visual analysis of UCAVs’ flight logs validates the learned hierarchical decision-making process, showcasing the interplay between tactical selection and manoeuvring execution. This research provides novel methodologies combining hierarchical reinforcement learning with tactical domain knowledge for the autonomous decision-making of UCAVs in complex air combat scenarios.

Deep reinforcement learning (DRL) can significantly improve the autonomy and effectiveness of air combat maneuver decision (ACMD). The design of reward functions faces significant challenges due to the long duration and large state space of air combat. Most existing methods rely on subjective judgment and cumbersome parameter tuning. Therefore, this paper proposed a reward design paradigm that provides practical guidance for the agent to win within visual range (WVR) air combat. Firstly, we proposed a dynamic situation assessor(DSA) based on the improved entropy weight method, which can improve the accuracy, contrast, and objectivity of air combat situation evaluation. In addition, we design a parameter optimizer(PO) based on the improved sparrow search algorithm (SSA) to achieve the automation of parameter tuning for proximal policy optimization (PPO), and applied it in the reward optimization, which improves the training efficiency significantly. Finally, the effectiveness of our work is evaluated through comparative experiments with state-of-the-art (SOTA) methods on a high-fidelity air combat simulation platform (JSBSim). The training process and win rate tests confirm the effectiveness of our reward design paradigm and show good training efficiency compared to existing methods.

Options, the temporally extended courses of actions that can be taken at varying time scale, have provided a concrete, key framework for learning levels of temporal abstraction in hierarchical tasks. While methods of learning options end-to-end is well researched, how to explore good options and actions simultaneously is still challenging. We address this issue by maximizing reward augmented with entropies of both option and action selection policy in options learning. To this end, we reveal our novel optimization objective by reformulating options learning from perspective of probabilistic inference and propose a soft options iteration method to guarantee convergence to the optimum. In implementation, we propose an off-policy algorithm called the maximum-entropy options critic (MEOC) and evaluate it on series of continuous control benchmarks. Comparative results demonstrate that our method outperforms baselines in efficiency and final result on most benchmarks, and the performance exhibits superiority and robustness especially on complex tasks. Ablated studies further explain that entropy maximization on hierarchical exploration promotes learning performance through efficient options specialization and multimodality in action level.