In this paper, we consider the problem of making an optimal offloading decision for a mobile user in an ad-hoc mobile cloud in which the mobile user can offload his computation tasks to nearby mobile cloudlets via a device-to-device (D2D) communication-enabled cellular network. We propose a deep reinforcement learning (DRL)-based offloading scheme which enables the user to make near-optimal offloading decisions by taking into account uncertainties of user's and cloudlets' movements and the cloudlets' resource availabilities. We first propose a Markov decision process (MDP)-based offloading problem formulation which considers the composite states of the user's and cloudlets' queue states and the distance states between the user and cloudlets as the system state space. The objective of the formulated MDP-based problem is to determine the optimal actions on how many tasks the user should process locally and how many tasks to offload to each cloudlet at each observed system state such that the user's utility obtained by task execution is maximized while minimizing the energy consumption, task processing delay, task loss probability and required payment. Then, we use a deep reinforcement learning scheme, called deep Q-network (DQN) to learn an efficient solution for the proposed MDP-based offloading problem. Extensive simulations were performed to evaluate the performance of the proposed offloading scheme. The simulation results validate the effectiveness of the offloading policies obtained by the proposed scheme.

Integrated sensing and communication (ISAC) allows the same hardware platform and resources to function sensing and communication simultaneously, which reduces the hardware size and addresses the spectrum congestion concerns. However, the sharing of the hardware and resources of the sensing and communication functions raises resource managements. Traditional optimization approaches are developed based on rigid mathematical models in ISAC systems. However, they face computation complexity and may not achieve the desired performance under the dynamics of the ISAC system environments. Machine learning with ability in learning features/patterns of data as well as approximating mathematical models has recently proposed to effectively solve the complicated ISAC problems. In this survey, we thus provide a comprehensive literature review on applications of learning algorithms for ISAC systems. Particularly, we review learning approaches proposed for emerging issues in ISAC systems, including beamforming designing/tracking, waveform design, spectrum allocation, time allocation, and power allocation, angle of arrival (AoA)/angle of departure (AoD) estimation, signal classification, and security issues. Moreover, we present applications of advanced learning methods for wireless sensing, which is considered to be an emerging sensing service of the next-generation networks. We conclude the survey with highlighting technical issues of learning algorithms and discussing future research directions.

In this paper, we propose a unmanned aerial vehicle (UAV)-assisted multi-hop edge computing (UAV-assisted MEC) system in which a UE can offload its task to multiple UAVs in a multi-hop fashion. In particular, the UE offloads a task to its nearby UAV, and this UAV can execute a part of the received task and offload the remaining part to its neighboring UAV. The offloading process continues until the task execution is finished. The benefit of this multihop offloading is that the task execution can be finished faster, and the computing load can be shared among multiple UAVs, thus avoiding overloading and congestion. Each node, i.e., the UE or the UAV, needs to determine the task size for offloading to minimize the cumulative energy consumption and latency over the nodes. We formulate a stochastic optimization problem under the dynamics and uncertainty of the UAV-assisted MEC system. Then, we propose a deep reinforcement learning (DRL) algorithm to solve this problem. Simulation results are provided to demonstrate the effectiveness of the DRL algorithm.

Recently, an ad-hoc mobile cloud has emerged as a promising architecture which allows a mobile user to offload its computation tasks to nearby mobile devices, namely mobile cloudlets, with a short offloading latency and low bandwidth consumption of wireless local area network (WLAN) connections. However, due to the uncertainty in the user and cloudlet movements and the availability of cloudlets' computation resources, it is a challenge to design an effective offloading scheme that can best provide the required quality of service (QoS) for the user. In this paper, we develop a constrained Markov decision process (CMDP) formulation for the mobile user to make an optimal offloading decision while considering its QoS requirements. The objective of the proposed CMDP formulation is to maximize the user's utility obtained by the task execution, while meeting the constraints on the required payment, energy consumption, processing delay and task loss probability. Two different schemes, which are linear programming (LP)-based and Q-Learning (QL)-based CMDP, are proposed to find the optimal solution for the formulated CMDP offloading problem. The LP-based scheme can be used for the user to acquire a QoS-aware optimal offloading policy with prior knowledge of the system state transition probabilities, while the QL-based scheme enables the user to learn an effective offloading decision in an unknown ad-hoc mobile cloud system. Extensive simulations were performed to evaluate the performance of the proposed schemes. The simulation results show the effectiveness of the offloading policies obtained by the proposed schemes, which outperform baseline schemes.

Air free-cooled data centers (DCs) have not existed in the tropical zone due to the unique challenges of year-round high ambient temperature and relative humidity (RH). The increasing availability of servers that can tolerate higher temperatures and RH due to the regulatory bodies’ prompts to raise DC temperature setpoints sheds light upon the feasibility of air free-cooled DCs in the tropics. However, due to the complex psychrometric dynamics, operating the air free-cooled DC in the tropics generally requires adaptive control of supply air condition to maintain the computing performance and reliability of the servers. This article studies the problem of controlling the supply air temperature and RH in a free-cooled tropical DC below certain thresholds. To achieve the goal, we formulate the control problem as Markov decision processes and apply deep reinforcement learning (DRL) to learn the control policy that minimizes the cooling energy while satisfying the requirements on the supply air temperature and RH. We also develop a constrained DRL solution for performance improvements. Extensive evaluation based on real data traces collected from an air free-cooled testbed and comparisons among the unconstrained and constrained DRL approaches as well as two other baseline approaches show the superior performance of our proposed solutions.