Multi-Agent Reinforcement Learning (MARL) is widely used to solve various problems in real life. In the multi-agent reinforcement learning tasks, there are multiple agents in the environment, the existing Proximal Policy Optimization (PPO) algorithm can be applied to multi-agent reinforcement learning. However, it cannot deal with the communication problem between agents. In order to resolve this issue, we propose a Graph-based PPO algorithm, this approach can solve the communication problem between agents and it can enhance the exploration efficiency of agents in the environment and speed up the learning process. We apply our algorithms to the task of multi-UAV navigation for communication coverage to verify the functionality and performance of our proposed algorithms.

In recent years, unmanned aerial vehicles (UAVs) have shown significant potential across diverse applications, drawing attention from both academia and industry. In specific scenarios, UAVs are expected to achieve formation flying without relying on communication or external assistance. In this context, our work focuses on the classic leader-follower formation and presents a learning-based UAV chasing control method that enables a quadrotor UAV to autonomously chase a highly maneuverable fixed-wing UAV. The proposed method utilizes a neural network called Vision Follow Net (VFNet), which integrates monocular visual data with the UAV’s flight state information. Utilizing a multi-head self-attention mechanism, VFNet aggregates data over a time window to predict the waypoints for the chasing flight. The quadrotor’s yaw angle is controlled by calculating the line-of-sight (LOS) angle to the target, ensuring that the target remains within the onboard camera’s field of view during the flight. A simulation flight system is developed and used for neural network training and validation. Experimental results indicate that the quadrotor maintains stable chasing performance through various maneuvers of the fixed-wing UAV and can sustain formation over long durations. Our research explores the use of end-to-end neural networks for UAV formation flying, spanning from perception to control.

In most existing studies on large-scale multi-agent coordination, the control methods aim to learn discrete policies for agents with finite choices. They rarely consider selecting actions directly from continuous action spaces to provide more accurate control; therefore, they are normally unsuitable for more complex tasks. To solve the control issue of large-scale multi-agent systems with continuous action spaces, we propose a novel MARL coordination control method named EHCAMA to derive stable continuous policies, by constructing a new network architecture in an actor–critic framework. By optimizing policies with maximum entropy learning, agents improve their exploration ability in training and acquire an excellent performance in execution. Further, we employ hierarchical graph attention networks (HGATs) and gated recurrent units (GRUs) to improve the scalability and transferability of our method. We simulate the performance of EHCAMA for cooperative tasks with both homogeneous and heterogeneous agents, and compare it with SAC-HGRN, HAMA, AHAC and AGGAT-Comm. The experimental results show that our method consistently outperforms the baselines in large-scale multi-agent scenarios.