Unmanned aerial vehicles as base stations (UAV-BSs) are recognized as effective means for tackling eruptive communication service requirements especially when terrestrial infrastructures are unavailable. Quality of service (QoS) received by ground terminals (GTs) highly depends on the spatial movement of UAV-BSs. In this paper, we investigate the cooperative trajectory design problem of multiple UAV-BSs towards fair throughput maximization of GTs. Considering the restriction of coverage and sensing, we first propose a heterogeneous-graph-based formulation of relations between GTs and UAV-BSs. Subsequently, we design a framework named graph vision and communication (GVis&Comm) to 1) let each UAV-BS efficiently manage time-varying local observations; 2) facilitate cooperation between UAV-BSs through explicit information exchange. To further reduce the overhead of over-the-air cooperation, we realize discretization of the message passing process among UAV-BSs while still enabling end-to-end training. By leveraging multi-agent reinforcement learning (MARL), UAV-BSs as agents learn a distributed trajectory design policy. Extensive numerical simulation shows that our framework on the one hand achieves remarkable efficiency in processing local observations of each UAV-BS, and on the other improves the overall network performance via close cooperation among UAV-BSs.

The evolution of artificial intelligence and Internet of Things (IoT) envision a highly integrated artificial IoT (AIoT) network. Flocking and cooperation with multiple unmanned aerial vehicles (UAVs) are expected to play a vital role in industrial AIoT networks. In this article, we formulate the collision-free flocking problem of fixed-wing UAVs as a Markov decision process and solve it in the deep reinforcement learning (DRL) framework. Our method can deal with a variable number of followers by encoding the dynamic environmental state into a fixed-length embedding tensor. Specifically, each follower constructs a fixed-size local situation map that describes the collision risks with other followers nearby. The local situation maps are used by a proposed DRL algorithm to learn the collision-free flocking behavior. To further improve the learning efficiency, we design a reference-point-based action selection strategy and an adaptive mechanism. We compare the proposed MA2D3QN algorithm with several benchmark DRL algorithms through numerical simulation, and we verify its advantages in learning efficiency and performance. Finally, we demonstrate the scalability and adaptability of MA2D3QN in a semiphysical simulation experiment.

Multiple unmanned aerial vehicles (UAVs) are able to efficiently accomplish a variety of tasks in complex scenarios. However, developing a collision-avoiding flocking policy for multiple fixed-wing UAVs is still challenging, especially in obstacle-cluttered environments. In this article, we propose a novel curriculum-based multiagent deep reinforcement learning (MADRL) approach called task-specific curriculum-based MADRL (TSCAL) to learn the decentralized flocking with obstacle avoidance policy for multiple fixed-wing UAVs. The core idea is to decompose the collision-avoiding flocking task into multiple subtasks and progressively increase the number of subtasks to be solved in a staged manner. Meanwhile, TSCAL iteratively alternates between the procedures of online learning and offline transfer. For online learning, we propose a hierarchical recurrent attention multiagent actor–critic (HRAMA) algorithm to learn the policies for the corresponding subtask(s) in each learning stage. For offline transfer, we develop two transfer mechanisms, i.e., model reload and buffer reuse, to transfer knowledge between two neighboring stages. A series of numerical simulations demonstrate the significant advantages of TSCAL in terms of policy optimality, sample efficiency, and learning stability. Finally, the high-fidelity hardware-in-the-loop (HITL) simulation is conducted to verify the adaptability of TSCAL. A video about the numerical and HITL simulations is available at https://youtu.be/R9yLJNYRIqY.