The field of autonomous air combat has witnessed a surge in interest propelled by the rapid progress of artificial intelligence technology. A persistent challenge within this domain pertains to autonomous decision-making for dogfighting, especially when dealing with intricate, high-fidelity nonlinear aircraft dynamic models and insufficient information. In response to this challenge, this paper introduces reinforcement learning (RL) to train maneuvering strategies. In the context of RL for dogfighting, the method by which opponents are sampled assumes significance in determining the efficacy of training. Consequently, this paper proposes a novel automatic opponent sampling (AOS)-based RL framework where proximal policy optimization (PPO) is applied. This approach encompasses three pivotal components: a phased opponent policy pool with simulated annealing (SA)-inspired curriculum learning, an SA-inspired Boltzmann Meta-Solver, and a Gate Function based on the sliding window. The training outcomes demonstrate that this improved PPO algorithm with an AOS framework outperforms existing reinforcement learning methods such as the soft actor–critic (SAC) algorithm and the PPO algorithm with prioritized fictitious self-play (PFSP). Moreover, during testing scenarios, the trained maneuvering policy displays remarkable adaptability when confronted with a diverse array of opponents. This research signifies a substantial stride towards the realization of robust autonomous maneuvering decision systems in the context of modern air combat.

With the increasing use of unmanned aerial vehicles (UAVs) in military operations, autonomous cooperative decision-making for multiple UAVs in aerial confrontations has become a critical research challenge. This paper presents an attention-based multi-agent actor-critic (AMAAC) algorithm for UAV aerial confrontation decision-making. The algorithm combines multi-head attention and self-play within the centralized training-distributed execution (CTDE) framework, extending the actor-critic approach based on the missile hit probability prediction model (MHPAC) to multi-UAV scenarios. A fighter observation encoder (OFE) and a centralized critic network based on the attention mechanism are introduced to adapt to varying number of UAVs (different scales) and enhance training performance. Additionally, self-play-based extended training is used to generalize offensive and defensive strategies from small-scale aerial confrontations to larger scenarios. Experimental results demonstrate that the AMAAC algorithm achieves superior training effectiveness, and the strategies it produces perform well across various confrontation scales, even beyond the training scenario's scale. Compared to other decision-making algorithms, such as Multi-agent Proximal Policy Optimization (MAPPO), Multi-agent Hierarchical Policy Gradient (MAHPG), and the State-Event-Condition-Action (SECA) algorithm, the AMAAC-trained strategies yield higher win ratios and kill-death ratios in different scenarios, validating the algorithm's effectiveness and scalability.

The complexities inherent in beyond-visual-range (BVR) air combat research stem from various intertwined factors, including dynamic nonlinear interactions, evolving dynamics during confrontations, and significant delays in countermeasure outcomes. These complexities create challenges in managing system state variables and navigating a vast strategic solution space. Applying algorithms directly to address these challenges often worsens the complexities, leading to considerable deviations and convergence issues. This paper tackles these challenges by initially dissecting the core elements intrinsic to the confrontation process. It establishes a detailed BVR confrontation environment, scrutinizes the stage characteristics of typical confrontations, introduces a framework for BVR confrontation decision-making, and devises a rule-based state-switch decision method. Moreover, it extracts and analyzes key decision parameters, deliberating on the tactical significance of the chosen parameters. The proposed method offers robust interpretability and demonstrates strong decision-making capabilities through simulation results, showcasing its advantages in autonomous decision-making during confrontations.

Deep reinforcement learning has gained increasing popularity in the beyond visual range engagement of unmanned combat aerial vehicles. Regardless of the success of the maneuvering decision, existing methods either utilize sparse rewards to optimize fire control or adopt a predefined fire policy, severely limiting the engagement performance. This paper presents a supervision-enhanced technique to address the challenges of the hybrid action space. A dual-head policy is developed, with one head dedicated to maneuvering decisions and the other to fire control. Alongside the reinforcement learning memory that stores state transitions, supervised learning memory is introduced to store pairs of states where missiles are launched and the hit outcomes. The maneuvering head is trained with the soft policy gradient, while the fire control head is optimized with the binary cross-entropy loss. This method enables agents to make maneuvering decisions and predict missile hit probabilities simultaneously without prior expert knowledge. Simulation demonstrates the superiority of the proposed method.