Autonomous maneuver decision by an unmanned combat air vehicle (UCAV) is a critical part of air combat that requires both flight safety and tactical maneuvering. In this paper, an unmanned combat air vehicle air combat maneuver decision method based on a proximal policy optimization algorithm (PPO) is proposed. Firstly, a motion model of the unmanned combat air vehicle and a situation assessment model of air combat was established to describe the motion situation of the unmanned combat air vehicle. An enemy maneuver policy based on a situation assessment with a greedy algorithm was also proposed for air combat confrontation, which aimed to verify the performance of the proximal policy optimization algorithm. Then, an action space based on a basic maneuver library and a state observation space of the proximal policy optimization algorithm were constructed, and a reward function with situation reward shaping was designed for accelerating the convergence rate. Finally, a simulation of air combat confrontation was carried out, which showed that the agent using the proximal policy optimization algorithm learned to combine a series of basic maneuvers, such as diving, climb and circling, into tactical maneuvers and eventually defeated the enemy. The winning rate of the proximal policy optimization algorithm reached 62%, and the corresponding losing rate was only 11%.

Traditional model-based control methods typically require accurate system dynamics. However, when controlling a complex non-linear system such as a quadrotor unmanned aerial vehicle (QUAV), the dynamics are unknown and it is challenging to tune the control parameters manually. This paper proposes a novel model-free learning method that combines the advantages of a model-based method, i.e., sliding mode control (SMC), with the iterative learning control (ILC) method. Specifically, we selected a designed sliding surface to obtain the expected tracking error trajectory as the learning objective, and the system tracking errors of the angles of the QUAV constitute the state space. Then, the policy of converging to the sliding surface is learned by an ILC algorithm.We have provided theoretical proof of the convergence, and validated the proposed method with real-world experiments where the sine wave signals of roll and pitch angles were tracked. The results have demonstrated the effectiveness of the method with less tracking errors as well as faster learning speed compared with a baseline PID controller and a sliding mode controller.

Fast and agile flying through a gap is challenging for a quadrotor if the gap is narrow, tilted, and moving. Due to the strict time-variant position and attitude constraints, collision-free traversal trajectories under the under-actuated quadrotor dynamics are sparse and difficult to solve. To achieve this challenging task in the real world, we propose a Gap-Traversing Adaptive Curriculum Learning (GTACL) approach, which consists of adaptive curriculum reinforcement learning (ACRL) and online thrust updating (OTU). First, ACRL is introduced to improve sample efficiency, and the policy training is accelerated by designing a curriculum adapted to the agent's capability. Second, OTU is proposed to map the acceleration commands to low-level throttle signals by estimating the thrust model during flight, which reduces the intermediate control variables and helps sim2real transfer. We use the prioritized experience replay mechanism that considers both policy update contribution and data acquisition time to adapt to the changing tasks. GTACL is trained entirely in simulation and can be transferred to other quadrotors with different dynamics. Furthermore, we achieve zero-shot transfer to the real-world quadrotor without fine-tuning. The average success rates of 98% and 87.8% in simulation and real-world experiments for different task conditions demonstrate the robustness of the proposed approach. Comparative results with traditional and related learning-based approaches show the advantages of GTACL in terms of learning efficiency, control performance, and generalization.

The flight controllability and safety of unmanned compound rotorcraft are closely related to their aerodynamic characteristics. During forward flight, complex aerodynamic interference effects arise among the rotor, propeller, wing, fuselage, and horizontal–vertical tail. These interactions change dramatically with variations in forward speed, which may have a substantial impact on flight performance. This paper investigates aerodynamic interference related to the rotor, propeller, and fuselage of a sample unmanned compound rotorcraft with a novel configuration. On this basis, a flight dynamics model that incorporates the identified aerodynamic interference is formulated. Firstly, an analysis of rotor/propeller/fuselage aerodynamic interference is performed using the momentum source method (MSM). Subsequently, the aerodynamic models for the wing, fuselage, and horizontal–vertical tail are updated by integrating aerodynamic interference factors, leading to the development of a nonlinear flight dynamics model for the sample unmanned compound rotorcraft. Finally, to validate the updated flight dynamics model, numerical simulation results are systematically compared against wind tunnel test results. The results reveal a significant correlation between the numerical simulation data and wind tunnel test results, which indicates that the updated flight dynamics model possesses high accuracy and reliability and can characterize the dynamic characteristics of the sample unmanned compound rotorcraft within the flight speed envelope.

Options, the temporally extended courses of actions that can be taken at varying time scale, have provided a concrete, key framework for learning levels of temporal abstraction in hierarchical tasks. While methods of learning options end-to-end is well researched, how to explore good options and actions simultaneously is still challenging. We address this issue by maximizing reward augmented with entropies of both option and action selection policy in options learning. To this end, we reveal our novel optimization objective by reformulating options learning from perspective of probabilistic inference and propose a soft options iteration method to guarantee convergence to the optimum. In implementation, we propose an off-policy algorithm called the maximum-entropy options critic (MEOC) and evaluate it on series of continuous control benchmarks. Comparative results demonstrate that our method outperforms baselines in efficiency and final result on most benchmarks, and the performance exhibits superiority and robustness especially on complex tasks. Ablated studies further explain that entropy maximization on hierarchical exploration promotes learning performance through efficient options specialization and multimodality in action level.

In this paper, we extend an existing real-time planner to avoid dynamic obstacles in unknown cluttered environments for Micro-UAVs. We use an optimization-based approach to generate the local trajectory represented by B-splines. We model dynamic obstacles and design a new optimization cost function. Currently, few methods can generate a trajectory avoiding moving obstacles, although most achieve high success rates in static environments We compare our method with the original planer in typical scenarios and the results show that our method is able to deal with dynamic obstacles more effectively.