Air ground integrated mobile cloud computing (MCC) provides unmanned aerial vehicles (UAVs) the capability to act as an aerial relay with more flexibility and resilience. In the cloud computing architecture, the data generated by ground users (GUs) can be offloaded to the remote server for fast processing. However, the heterogeneity of mobile tasks makes the data size distributed among GUs unbalanced. Besides, the energy efficiency of UAVs movement should be carefully considered for sustainable flight and obstacle avoidance. In general, such a joint trajectory issue can hardly be formulated as a convex optimization in unpredictable and dynamic environments. This paper proposes a potential game combined multi-agent deep deterministic policy gradient (MADDPG) approach to optimize multiple UAVs' trajectory with the consideration of GUs' offloading delay, energy efficiency as well as obstacle avoidance system. In specific, we first model the issue as a mixed integer non-linear problem (MINP), in which the service assignment between multi-user and multi-UAV is solved by potential game. The convergence to a Nash Equilibrium (NE) can be achieved by distributive service assignment update with infinite iteration. Then, we optimize the trajectory with obstacle avoidance at each UAV by MADDPG approach, which has a great advantage of centralized-training and decentralized-execution to reduce the global synchronized communication overhead. UAVs movement can be optimized in continuity rather than other deep reinforcement learning (DRL) approaches generating discrete simple actions. Experiments demonstrate the proposed game-combined learning algorithm can minimize the offloading delay, enhance UAVs’ energy efficiency and avoid the obstacles at the same time.

Cognitive radio networks (CRNs) can greatly improve the temporal and spatial spectrum utilization by identifying and exploring spectrum holes of the licensed primary users (PUs). However, since the occupation of primary channels changes dynamically, a swift and accurate spectrum sensing is crucial especially in the multi-channel multi-secondary users (SUs) environment, where the number of channels is much larger than that of SUs. To improve the sensing accuracy, a cooperative sensing algorithm is proposed in this letter, where multiple SUs can share their spectrum detection results for a more effective spectrum holes search. This letter further employs multi-agent deep deterministic policy gradient (MADDPG) algorithm with the feature of centralized training and decentralized execution to reduce the synchronization and communication overhead caused by the sensing cooperation of SUs. The numerical simulation demonstrates that with the combination of cooperative sensing and multi-agent reinforcement learning, the proposed algorithm can greatly enhance the sensing accuracy in comparison to other non-cooperative learning or centralized learning approaches.

Wireless powered mobile edge computing (MEC) networks, where wireless devices (WDs) are allowed to offload parts of computation-intensive tasks to remote servers and charge the built-in batteries over the air, have been envisaged as a promising technology to ensure the ultra-low-power requirement and enhance the continuous work capacity of WDs. However, when multiple WDs coexist in the network, it is non-trivial to minimize the total tasks delay since the optimization variables are intrinsically coupled. Even more, channels are dynamically varying from time to time and the tasks are unpredictable, which aggravates the difficulty to obtain the closed-form solution. Although reinforcement learning (RL) has been proved to be effective for such complex optimization problems, there is still the challenge that the training of neural networks is time-consuming. This paper considers a challenging hybrid tasks offloading scenario, where offloading tasks can be partially executed locally and remotely in parallel, and each WD is endowed to take both the active RF-transmission and passive backscatter communication (BackCom) for remote tasks offloading. Furthermore, a game-combined multi-agent deep deterministic policy gradient (MADDPG) algorithm is proposed to minimize the total tasks delay with the fairness consideration of multiple WDs, i.e., potential game for offloading decision and MADDPG for time scheduling and harvested energy splitting. The introduction of potential game which can be proved to converge with finite iterations, helps to accelerate the training and reduce the computation complexity. Equipped with the feature of ‘centralized training with decentralized execution,’ once well trained, each agent in MADDPG can figure out the proper time scheduling and harvested energy splitting independently without sharing information with others. Besides the unilateral contention among WDs for the offloading decision by potential game, a fully decentralized framework is finally designed for the proposed algorithm. Numerical results demonstrate that the game-combined MADDPG algorithm can achieve the near-optimal performance compared with existing centralized approaches, and reduce the convergence time compared with other no-game learning approaches.

Recent years, unmanned aerial vehicles (UAVs) have attracted much attention for providing intermediate relay to ground mobile user equipments (UEs) for their flexible mobility. UEs can offload computing-intensive task to mobile cloud computing (MCC) or mobile edge computing (MEC) for fast processing. However, with multi-UAV and ground mobile UEs in the system, heterogeneous performance requirement as well as fast-changing communication condition make the system more complicated. Meanwhile, both UEs and UAVs are battery-driven. How to optimize the energy efficiency for UEs' transmission and UAVs' position should be carefully considered. Since this is a non-convex and mixed-integer optimization problem, a heuristic joint power and quality of experience (HJPQ) algorithm is proposed in this article, where the UEs' offloading delay, MIMO channel, transmission power, as well as UAVs' placement are jointly optimized. The numeral simulations not only reveal the effectiveness of HJPQ, but also guarantee the great quality of experience (QoE) performance for UEs with different priorities. Furthermore, the comparison experiments with random assignment and deep deterministic policy gradient (DDPG) show the superiority of HJPQ in lower complexity, faster convergence, shorter offloading delay as well as higher energy efficiency.

Cooperative spectrum sensing (CSS) technology has been widely studied to enhance the spectrum sharing efficiency spatially and temporally in cognitive radio networks (CRNs), where the secondary users (SUs) can opportunistically reuse the channels already licensed to the primary users (PUs) for transmission by sensing spectrum holes. SUs are endowed with the global awareness of channels state by cooperating with each other without sweeping across the whole frequency bands. Since the channels occupation of PUs changes dynamically, the accurate sensing and swift information sharing are crucial for CRNs. The paper proposes a multi-agent deep reinforcement learning (DRL) based CSS method to help SUs efficiently finding a vacant channel by the cooperation with their partners. 1 Two partner selection algorithms are proposed named as Reliable Partner CSS and Adaptive Partner CSS, respectively. For the former, the partner selection is facilitated based on the historical sensing accuracy of SUs. While the latter takes the comprehensive consideration of both the reliability and geographical distribution of SUs to further improve the sensing accuracy. 2 Multi-agent deep deterministic policy gradient (MADDPG) is adopted to resist the dynamically varying channels condition as well as the high-dimension solution space. With the feature of ‘centralized training and decentralized execution’, each SU learns to interact with the environment and select a vacant channel for transmission by its partial observation, which greatly reduces the communication overhead caused by the cooperative spectrum sensing. 3 Numerical simulation demonstrates the convergence and availability of the proposed algorithms. No matter Reliable Partner CSS or Adaptive Partner CSS, the sensing accuracy can be greatly enhanced comparing with other non-cooperative or centralized learning approaches. Besides, the attention mechanism is introduced to MADDPG for Adaptive Partner CSS to reveal the behavior of SUs by the visualization of attention weight, which helps to partially interpret the ‘black box’ issue of DRL.

Air ground integrated Mobile Cloud Computing (MCC) provides UAVs with more flexibility and resilience from the cloud computing architecture. However, the increasing aerial mobile data requires heterogeneous quality of experience (QoE) for aerial accessing network. Additionally, for the persistent flying, energy efficiency during the computation offloading should also be under consideration. This paper proposes an energy-efficient resource allocation scheme with the ability of QoE enhancement. Various aerial offloading data with different QoE requirements is stored and relayed in the multi-queueing architecture. Hence offloading rate differentiation is utilized to ensure the high-priority data a better QoE. The satisfaction function is designed with respect to energy efficiency and actual performance experienced by UAV. By using Lyapunov optimization technique, the problem can be decoupled into two independent sub-problems. The first one is rate control associated with multi-queueing architecture in ground base-station (GBS) that manages the aerial offloading data from UAVs according to the queue state information. The second one is resource allocation associated with the strategy of subcarrier assignment and power allocation according to the channel state information. Experiments demonstrate the algorithm has great properties such as maximization of UAVs’ satisfaction, reliable heterogeneous QoE support and enhancement of UAVs’ transmission energy efficiency.