Due to the high maneuverability and flexibility, unmanned aerial vehicles (UAVs) have been considered as a promising paradigm to assist mobile edge computing (MEC) in many scenarios including disaster rescue and field operation. Most existing research focuses on the study of trajectory and computation-offloading scheduling for UAV-assisted MEC in stationary environments, and could face challenges in dynamic environments where the locations of UAVs and mobile devices (MDs) vary significantly. Some latest research attempts to develop scheduling policies for dynamic environments by means of reinforcement learning (RL). However, as these need to explore in high-dimensional state and action space, they may fail to cover in large-scale networks where multiple UAVs serve numerous MDs. To address this challenge, we leverage the idea of “divide-and-conquer” and propose HT3O, a scalable scheduling approach for large-scale UAV-assisted MEC. First, HT3O is built with neural networks via deep RL to obtain real-time scheduling policies for MEC in dynamic environments. More importantly, to make HT3O more scalable, we decompose the scheduling problem into two-layered subproblems and optimize them alternately via hierarchical RL. This not only substantially reduces the complexity of each subproblem, but also improves the convergence efficiency. Experimental results show that HT3O can achieve promising performance improvements over state-of-the-art approaches.

Mobile edge computing (MEC) has been considered as a promising paradigm to support the growing popularity of mobile devices (MDs) with similar capabilities as cloud computing. Most existing research focuses on MEC enabled by terrestrial base stations (BSs), which is unable to work in certain scenarios, e.g., disaster rescue and field operation. Some researchers have been making efforts on studying MEC assisted by unmanned-aerial-vehicles (UAVs) and developed lots of efficient scheduling algorithms. However, MEC assisted only by UAVs has limited capability and is unsuitable for heavy-computation applications. To address the issue, this paper proposes a novel UAV-and-BS hybrid enabled MEC system, where multiple UAVs and one BS are deployed to facilitate the provisioning of MEC services either directly from UAVs or indirectly from the BS. Considering maximizing the lifetime of all MDs, the energy-efficient scheduling problem is formulated as minimizing the energy consumption of all MDs by jointly optimizing UAV trajectories, task associations, computing-and-transmitting resource allocations. The optimization problem is further decomposed into three sub-problems and solved by the proposed hybrid heuristic and learning based scheduling algorithm to reduce the complexity. Experimental results show that the proposed algorithm can achieve promising performance improvements over baseline algorithms.

Mobile edge computing (MEC) has been envisioned as a promising paradigm that could effectively enhance the computational capacity of wireless user devices (WUDs) and quality of experience of mobile applications. One of the most crucial issues of MEC is computation offloading, which decides how to offload WUDs’ tasks to edge severs for further intensive computation. Conventional mathematical programming-based offloading approaches could face troubles in dynamic MEC environments due to the time-varying channel conditions (caused primarily by WUD mobility). To address the problem, reinforcement learning (RL) based offloading approaches have been proposed, which develop offloading policies by mapping MEC states to offloading actions. However, these approaches could fail to converge in large-scale MEC due to the exponentially-growing state and action spaces. In this article, we propose a novel online computation offloading approach that could effectively reduce task latency and energy consumption in dynamic MEC with large-scale WUDs. First, a RL-based computation offloading and energy transmission algorithm is proposed to accelerate the learning process. Then, a joint optimization method is adopted to develop the allocating algorithm, which obtains near-optimal solutions for energy and computation resources allocation. Simulation results show that the proposed approach can converge efficiently and achieve significant performance improvements over baseline approaches.

Mobile devices (MDs) have undergone a booming development, yet are still capacity limited in computation and energy resources and thus could face troubles when serving computation-intensive and delay-sensitive applications. Mobile-edge computing (MEC) has been proposed to accommodate MDs with both satisfactory latency and acceptable resources, by offloading MDs’ tasks to near-deployed edge servers (ESs). Whereas, offloading tasks solely to ESs are difficult to meet distinct requirements of various applications, which leads to the emergence of end–edge–cloud orchestrated computing (EECOC). However, studies on EECOC are still insufficient, most existing of which do not consider MDs’ dynamical movements partly due to the intractability of associating optimal ESs with moving MDs. To address the issue, a novel deep reinforcement learning (DRL)-based mobility-aware (MA) EECOC scheduling approach is proposed in this article. With the goal of minimizing maximal task latency, we first formulate and transform the optimization problem into a Markov decision problem (MDP). Then, we enable DRL with elaborately designed reward functions and integrate it with NoisyNet to obtain near-optimal solutions. Furthermore, a MA component based on ConvLSTM is developed to extract MDs’ temporal–spatial distribution features and predict their movements, which are further utilized to facilitate the decision making of computation-offloading and resource-allocation actions. Extensive experimental results indicate the promising performance improvements of our approach against the state-of-the-art approaches in various scenarios.

One of the key issues in mobile edge computing (MEC) is computation offloading, most policies of which are developed based on mathematical programming (MP). Due to the high computational complexity of iterative programming in MP-based policies, recent years have seen a popular trend to develop offloading policies based on deep reinforcement learning (DRL). However, on account of the poor generalization ability of DRL models in MEC environments with different network sizes and settings, it is difficult to directly apply DRL-based offloading policies in unseen MEC environments. Motivated by this, we propose a DRL-based environment-adaptive offloading framework (DEAT), including a size-adaptive scheme (SIED) and setting-adaptive component (SEAL). SIED leverages the idea of ‘time division multiplexing’ to adapt to varying MEC network sizes and order-unaware feature extraction to mitigate impacts of different size-changing orders. SEAL adopts system dynamics embedding and offloading policy embedding, which guide the finding of the closest pre-training MEC environment and offloading policy, respectively, to achieve fast setting-adaptation with only few exploring interactions in unseen MEC environments. Extensive experiments are conducted via both simulation and testbed to demonstrate the adaptation performance advantages of DEAT in unseen MEC environments compared to the state-of-the-art offloading approaches.

Large-scale canals are worldwidely constructed to divert water from rich to arid areas to mitigate water shortages. Efficient control of canals is essential to improve water-diversion performance. Numerous model-based approaches have been proposed and made great progress for canal control. However, when the predictive model is unpromising for long time-step predictions, model-free approaches could be considered as a possible way. Since most existing model-free approaches are focused on control of small canals or reservoirs, this paper proposes a new control approach named PARL for large-scale canals. We leverage the idea of divide-and-conquer to decompose the control task of large-scale canals into policy-learning and action-learning sub-tasks, and develop PARL by means of hierarchical reinforcement learning. Extensive experiments are conducted via numerical simulation on the case study of Chinese South to North Water Transfer Project, and experimental results show that PARL can achieve desirable performance improvements over other model-free learning approaches.