Distributed multi-agent collaborative decision-making technology is the key to general artificial intelligence. This paper takes the self-developed Unity3D collaborative combat environment as the test scenario, setting a task that requires heterogeneous unmanned aerial vehicles (UAVs) to perform a distributed decision-making and complete cooperation task. Aiming at the problem of the traditional proximal policy optimization (PPO) algorithm’s poor performance in the field of complex multi-agent collaboration scenarios based on the distributed training framework Ray, the Critic network in the PPO algorithm is improved to learn a centralized value function, and the muti-agent proximal policy optimization (MAPPO) algorithm is proposed. At the same time, the inheritance training method based on course learning is adopted to improve the generalization performance of the algorithm. In the experiment, MAPPO can obtain the highest average accumulate reward compared with other algorithms and can complete the task goal with the fewest steps after convergence, which fully demonstrates that the MAPPO algorithm outperforms the state-of-the-art.

Autonomous umanned aerial vehicle (UAV) manipulation is necessary for the defense department to execute tactical missions given by commanders in the future unmanned battlefield. A large amount of research has been devoted to improving the autonomous decision-making ability of UAV in an interactive environment, where finding the optimal maneuvering decision-making policy became one of the key issues for enabling the intelligence of UAV. In this paper, we propose a maneuvering decision-making algorithm for autonomous air-delivery based on deep reinforcement learning under the guidance of expert experience. Specifically, we refine the guidance towards area and guidance towards specific point tasks for the air-delivery process based on the traditional air-to-surface fire control methods. Moreover, we construct the UAV maneuvering decision-making model based on Markov decision processes (MDPs). Specifically, we present a reward shaping method for the guidance towards area and guidance towards specific point tasks using potential-based function and expert-guided advice. The proposed algorithm could accelerate the convergence of the maneuvering decision-making policy and increase the stability of the policy in terms of the output during the later stage of training process. The effectiveness of the proposed maneuvering decision-making policy is illustrated by the curves of training parameters and extensive experimental results for testing the trained policy.

This paper reports on the formation and transformation of multiple fixed-wing unmanned aerial vehicles (UAVs) in three-dimensional space. A cooperative guidance law based on the classic missile-type parallel-approach method is designed for the multi-UAV formation control problem. Additionally, formation transformation strategies for multi-UAV autonomous assembly, disbandment, and special circumstances are formed, effective for managing and controlling the formation. When formulating the management strategy for formation establishment, its process is divided into three steps: (i) selecting and allocating target points, (ii) forming loose formations, and (iii) forming short-range formations. The management of disbanding the formation is formulated through reverse thinking: the assembly process is split and recombined in reverse, and a formation disbanding strategy that can achieve a smooth transition from close to lose formation is proposed. Additionally, a strategy is given for adjusting the formation transformation in special cases, and the formation adjustment is completed using the adjacency matrix. Finally, a hardware-in-the-loop simulation and measured flight verification using a simulator show the practicality of the guidance law in meeting the control requirements of UAV formation flight for specific flight tasks.

It’s difficult for traditional solutions consisting of path planning and trajectory tracking algorithms to solve UAV maneuvering decision-making problem while executing long-distance UAV airdrop mission, because it’s necessary for obtaining an available path to prepare detailed information about mission area, such as digital elevation model, mountain, and other all possible threats. Therefore, it’s difficult for traditional methods to achieve UAV guidance in an interactive environment with an uncertain number of threats. An end-to-end algorithm is needed to perform long-distance UAV airdrop mission, where finding an optimal maneuvering decision-making policy becomes one of the key issues for improving the autonomy of UAV. In this paper, an experience knowledge driven learning algorithm based on deep reinforcement learning for long-distance UAV airdrop is proposed, which introduces expert experience to modify reward function for fully utilizing the potential value of experience transitions. Moreover, the training scheme of UAV maneuvering policy is designed based on Twin Delayed Deep Deterministic Policy Gradient with Prioritized Experience Replay (PER-TD3). Particularly, the training scheme incorporates Curriculum Learning (CL) to accelerate the convergence of policy. Compared with the traditional reward model without the inspiration of experience knowledge, our proposed algorithm could accelerate the convergence of policy and improve the performance of trained policy.