A novel framework is proposed for the trajectory design of multiple unmanned aerial vehicles (UAVs) based on the prediction of users' mobility information. The problem of joint trajectory design and power control is formulated for maximizing the instantaneous sum transmit rate while satisfying the rate requirement of users. In an effort to solve this pertinent problem, a three-step approach is proposed which is based on machine learning techniques to obtain both the position information of users and the trajectory design of UAVs. Firstly, a multi-agent Q-learning based placement algorithm is proposed for determining the optimal positions of the UAVs based on the initial location of the users. Secondly, in an effort to determine the mobility information of users based on a real dataset, their position data is collected from Twitter to describe the anonymous user-trajectories in the physical world. In the meantime, an echo state network (ESN) based prediction algorithm is proposed for predicting the future positions of users based on the real dataset. Thirdly, a multi-agent Q-learning based algorithm is conceived for predicting the position of UAVs in each time slot based on the movement of users. The algorithm is proved to be able to converge to an optimal state. In this algorithm, multiple UAVs act as agents to find optimal actions by interacting with their environment and learn from their mistakes. Numerical results are provided to demonstrate that as the size of the reservoir increases, the proposed ESN approach improves the prediction accuracy. Finally, we demonstrate that throughput gains of about $17\%$ are achieved.

A novel framework is proposed for quality of experience (QoE)-driven deployment and dynamic movement of multiple unmanned aerial vehicles (UAVs). The problem of joint non-convex three-dimensional (3D) deployment and dynamic movement of the UAVs is formulated for maximizing the sum mean opinion score (MOS) of ground users, which is proved to be NP-hard. In the aim of solving this pertinent problem, a three-step approach is proposed for attaining 3D deployment and dynamic movement of multiple UAVs. Firstly, genetic algorithm based K-means (GAK-means) algorithm is utilized for obtaining the cell partition of the users. Secondly, Q-learning based deployment algorithm is proposed, in which each UAV acts as an agent, making their own decision for attaining 3D position by learning from trial and mistake. In contrast to conventional genetic algorithm based learning algorithms, the proposed algorithm is capable of training the direction selection strategy offline. Thirdly, Q-learning based movement algorithm is proposed in the scenario that the users are roaming. The proposed algorithm is capable of converging to an optimal state. Numerical results reveal that the proposed algorithms show a fast convergence rate after a small number of iterations. Additionally, the proposed Q-learning based deployment algorithm outperforms K-means algorithms and Iterative-GAKmean (IGK) algorithms with a low complexity.

The effective trajectory design of multiple unmanned aerial vehicles (UAVs) is investigated for improving the capacity of the communication system. The aim is for maximizing real-time downlink capacity under the coverage constraint by reaping the mobility benefits of UAVs. The problem of three-dimension (3D) dynamic movement of UAVs under coverage constraint is formulated as a Constrained Markov Decision Process (CMDP) problem, while a constrained Deep Q-Network (cDQN) algorithm is proposed for solving the formulated problem. In the proposed cDQN model, each UAV acts as an agent to explore and learn its 3D deploying policy. The aim of the proposed cDQN model is for obtaining the maximum capacity while attempting to guarantee that all ground terminals (GTs) are covered. In order to satisfy the coverage constraint, a primal-dual method is adopted for training primal variable and dual variable (lagrangian multiplier) in turn. Furthermore, in an effort to reduce the action space of the cDQN algorithm, prior information is utilized for eliminating the invalid actions by the action filter. Experiment results demonstrate that the cDQN algorithm is capable of converging after some training steps. Additionally, the UAVs are capable of adapting the movement of GTs under the coverage constraint according to the 3D deploying policy derived from the proposed cDQN algorithm.

This article mainly focus on addressing the non-orthogonal multiple access (NOMA) enhanced unmanned aerial vehicle (UAV)-to-Everything (U2X) networks. A novel three dimension (3D) network for providing wireless services to randomly roaming NOMA receivers (Rxs) in the sphere space is proposed by utilizing stochastic geometry tools. In an effort to evaluate the performance of the proposed network, we first derive the closed-form expressions for the outage probability and the ergodic rate of the paired NOMA Rxs. Both the diversity orders and the high-signal-to-noise (SNR) slopes of the paired NOMA Rxs are derived. The spectrum efficiency (SE) in both the NOMA and orthogonal multiple access (OMA) enhanced U2X networks are derived. Our analytical results demonstrate that the diversity order and the high-SNR slope of the proposed network are fading parameter m and one, respectively. Numerical results are provided to confirm that: i) the proposed NOMA enhanced U2X networks have superior outage performance and SE compared with the OMA enhanced U2X networks; and ii) for the case of fixed LoS probability, the outage performance of the paired NOMA Rxs mainly depends on the poorer channel conditions.

This article investigates a non-orthogonal multiple access (NOMA) enhanced Internet of Things (IoT) network. In order to provide connectivity, a novel cluster strategy is proposed, where multiple devices can be served simultaneously. Two potential scenarios are investigated: 1) NOMA enhanced terrestrial IoT networks and 2) NOMA enhanced aerial IoT networks. We utilize stochastic geometry tools to model the spatial randomness of both terrestrial and aerial devices. New channel statistics are derived for both terrestrial and aerial devices. The exact and the asymptotic expressions in terms of coverage probability are derived. In order to obtain further engineering insights, short-packet communication scenarios are investigated. From our analysis, we show that the performance of NOMA enhanced IoT networks is capable of outperforming OMA enhanced IoT networks. Moreover, based on simulation results, there exists an optimal value of the transmit power that maximizes the coverage probability.

A communication enabled indoor intelligent robots (IRs) service framework is proposed, where non-orthogonal multiple access (NOMA) technique is adopted to enable highly reliable communications. In cooperation with the ultramodern indoor channel model recently proposed by the International Telecommunication Union (ITU), the Lego modeling method is proposed, which can deterministically describe the indoor layout and channel state in order to construct the radio map. The investigated radio map is invoked as a virtual environment to train the reinforcement learning agent, which can save training time and hardware costs. Build on the proposed communication model, motions of IRs who need to reach designated mission destinations and their corresponding down-link power allocation policy are jointly optimized to maximize the mission efficiency and communication reliability of IRs. In an effort to solve this optimization problem, a novel reinforcement learning approach named deep transfer deterministic policy gradient (DT-DPG) algorithm is proposed. Our simulation results demonstrate in the following: 1) with the aid of NOMA techniques, the communication reliability of IRs is effectively improved; 2) radio map is qualified to be a virtual training environment, and its statistical channel state information improves training efficiency by about 30%; 3) proposed DT-DPG algorithm is superior to the conventional deep deterministic policy gradient (DDPG) algorithm in terms of optimization performance, training time, and anti-local optimum ability.