This paper presents a comprehensive literature review on applications of deep reinforcement learning in communications and networking. Modern networks, e.g., Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, deep reinforcement learning, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of deep reinforcement learning from fundamental concepts to advanced models. Then, we review deep reinforcement learning approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks such as 5G and beyond. Furthermore, we present applications of deep reinforcement learning for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying deep reinforcement learning.

Recently, ambient backscatter communications has been introduced as a cutting-edge technology which enables smart devices to communicate by utilizing ambient radio frequency (RF) signals without requiring active RF transmission. This technology is especially effective in addressing communication and energy efficiency problems for low-power communications systems such as sensor networks. It is expected to realize numerous Internet-of-Things (IoT) applications. Therefore, this paper aims to provide a contemporary and comprehensive literature review on fundamentals, applications, challenges, and research efforts/progress of ambient backscatter communications. In particular, we first present fundamentals of backscatter communications and briefly review bistatic backscatter communications systems. Then, the general architecture, advantages, and solutions to address existing issues and limitations of ambient backscatter communications systems are discussed. Additionally, emerging applications of ambient backscatter communications are highlighted. Finally, we outline some open issues and future research directions.

In traditional machine learning, the central server first collects the data owners' private data together and then trains the model. However, people's concerns about data privacy protection are dramatically increasing. The emerging paradigm of federated learning efficiently builds machine learning models while allowing the private data to be kept at local devices. The success of federated learning requires sufficient data owners to jointly utilize their data, computing and communication resources for model training. In this paper, we propose an auction based market model for incentivizing data owners to participate in federated learning. We design two auction mechanisms for the federated learning platform to maximize the social welfare of the federated learning services market. Specifically, we first design an approximate strategy-proof mechanism which guarantees the truthfulness, individual rationality, and computational efficiency. To improve the social welfare, we develop an automated strategy-proof mechanism based on deep reinforcement learning and graph neural networks. The communication traffic congestion and the unique characteristics of federated learning are particularly considered in the proposed model. Extensive experimental results demonstrate that our proposed auction mechanisms can efficiently maximize the social welfare and provide effective insights and strategies for the platform to organize the federated training.

Federated learning (FL) enables resource-constrained edge nodes to collaboratively learn a global model under the orchestration of a central server while keeping privacy-sensitive data locally. The non-independent-and-identically-distributed (non-IID) data samples across participating nodes slow model training and impose additional communication rounds for FL to converge. In this paper, we propose Fed erated Ad a p tive Weighting ( FedAdp ) algorithm that aims to accelerate model convergence under the presence of nodes with non-IID dataset. We observe the implicit connection between the node contribution to the global model aggregation and data distribution on the local node through theoretical and empirical analysis. We then propose to assign different weights for updating the global model based on node contribution adaptively through each training round. The contribution of participating nodes is first measured by the angle between the local gradient vector and the global gradient vector, and then, weight is quantified by a designed non-linear mapping function subsequently. The simple yet effective strategy can reinforce positive (suppress negative) node contribution dynamically, resulting in communication round reduction drastically. Its superiority over the commonly adopted Federated Averaging ( FedAvg ) is verified both theoretically and experimentally. With extensive experiments performed in Pytorch and PySyft, we show that FL training with FedAdp can reduce the number of communication rounds by up to 54.1% on MNIST dataset and up to 45.4% on FashionMNIST dataset, as compared to FedAvg algorithm.

Future-generation wireless networks (5G and beyond) must accommodate surging growth in mobile data traffic and support an increasingly high density of mobile users involving a variety of services and applications. Meanwhile, the networks become increasingly dense, heterogeneous, decentralized, and ad hoc in nature, and they encompass numerous and diverse network entities. Consequently, different objectives, such as high throughput and low latency, need to be achieved in terms of service, and resource allocation must be designed and optimized accordingly. However, considering the dynamics and uncertainty that inherently exist in wireless network environments, conventional approaches for service and resource management that require complete and perfect knowledge of the systems are inefficient or even inapplicable. Inspired by the success of machine learning in solving complicated control and decision-making problems, in this article we focus on deep reinforcement- learning (DRL)-based approaches that allow network entities to learn and build knowledge about the networks and thus make optimal decisions locally and independently. We first overview fundamental concepts of DRL and then review related works that use DRL to address various issues in 5G networks. Finally, we present an application of DRL for 5G network slicing optimization. The numerical results demonstrate that the proposed approach achieves superior performance compared with baseline solutions.

Public blockchain networks using proof of work-based consensus protocols are considered a promising platform for decentralized resource management with financial incentive mechanisms. In order to maintain a secured, universal state of the blockchain, proof of work-based consensus protocols financially incentivize the nodes in the network to compete for the privilege of block generation through cryptographic puzzle solving. For rational consensus nodes, i.e., miners with limited local computational resources, offloading the computation load for proof of work to the cloud/fog providers becomes a viable option. In this paper, we study the interaction between the cloud/fog providers and the miners in a proof of work-based blockchain network using a game theoretic approach. In particular, we propose a lightweight infrastructure of the proof of work-based blockchains, where the computation-intensive part of the consensus process is offloaded to the cloud/fog. We formulate the computation resource management in the blockchain consensus process as a two-stage Stackelberg game, where the profit of the cloud/fog provider and the utilities of the individual miners are jointly optimized. In the first stage of the game, the cloud/fog provider sets the price of offered computing resource. In the second stage, the miners decide on the amount of service to purchase accordingly. We apply backward induction to analyze the sub-game perfect equilibria in each stage for both uniform and discriminatory pricing schemes. For uniform pricing where the same price applies to all miners, the uniqueness of the Stackelberg equilibrium is validated by identifying the best response strategies of the miners. For discriminatory pricing where the different prices are applied, the uniqueness of the Stackelberg equilibrium is proved by capitalizing on the variational inequality theory. Further, the real experimental results are employed to justify our proposed model.