The incumbent Internet of Things suffers from poor scalability and elasticity exhibiting in communication, computing, caching and control (4Cs) problems. The recent advances in deep reinforcement learning (DRL) algorithms can potentially address the above problems of IoT systems. In this context, this paper provides a comprehensive survey that overviews DRL algorithms and discusses DRL-enabled IoT applications. In particular, we first briefly review the state-of-the-art DRL algorithms and present a comprehensive analysis on their advantages and challenges. We then discuss on applying DRL algorithms to a wide variety of IoT applications including smart grid, intelligent transportation systems, industrial IoT applications, mobile crowdsensing, and blockchain-empowered IoT. Meanwhile, the discussion of each IoT application domain is accompanied by an in-depth summary and comparison of DRL algorithms. Moreover, we highlight emerging challenges and outline future research directions in driving the further success of DRL in IoT applications.

Unmanned aerial vehicles (UAVs) can be leveragedin mobile crowdsensing (MCS) to conduct sensing tasks at remote or rural areas through computation offloading and data sensing. Nonetheless, both computation offloading and data sensing have been separately investigated in most existing studies. In this paper, we propose a novel cooperative data sensing and computation offloading scheme for the UAV-assisted MCS system with an aim to maximize the overall system utility. First, a multi-objective function is formulated to evaluate the system utility by jointly considering flight direction, flight distance, task offloading proportion, and server offload selection for each UAV. Then, the problem is modeled as a partially observable Markov decision process and a multi-agent actor-critic algorithm framework is proposed to train the strategy network for UAVs. Due to high delay and energy costs caused by communications among multiple agents, we train a centralized critic network to model other agents and to seek equilibrium among all UAVs rather than adopting the explicit channel for information exchange. Furthermore, we introduce attention mechanism to enhance the convergence performance in model training phases. Finally, experimental results demonstrate the effectiveness and applicability of our scheme. Compared with baselines, our algorithm shows significant advantages in convergence performance, system utility, task costs, and task completion rate.

Accurate and rapid response in complex driving scenarios is a challenging problem in autonomous driving. If a target is detected, the vehicle will not be able to react in time, resulting in fatal safety accidents. Therefore, the application of driver assistance systems requires a model that can accurately detect targets in complex scenes and respond quickly. In this paper, a lightweight feature extraction model, ShuffDet, is proposed to replace the CSPDark53 model used by YOLOX by improving the YOLOX algorithm. At the same time, an attention mechanism is introduced into the path aggregation feature pyramid network (PAFPN) to make the network focus more on important information in the network, thereby improving the accuracy of the model. This model, which combines two methods, is called ShuffYOLOX, and it can improve the accuracy of the model while keeping it lightweight. The performance of the ShuffYOLOX model on the KITTI dataset is tested in this paper, and the experimental results show that compared to the original network, the mean average precision (mAP) of the ShuffYOLOX model on the KITTI dataset reaches 92.20%. In addition, the number of parameters of the ShuffYOLOX model is reduced by 34.57%, the Gflops are reduced by 42.19%, and the FPS is increased by 65%. Therefore, the ShuffYOLOX model is very suitable for autonomous driving applications.

Sharding technologies allow the Internet of Things (IoT) to deploy blockchains in large-scale applications with good scalability. However, conventional sharding strategies in IoT blockchain are highly restricted because most IoT devices are dynamic and heterogeneous. They fail to partition and reconfigure shards with a fine-balanced tradeoff between throughput and security. Therefore, we propose SmartChain, which is a dynamic and self-adaptive sharding framework devised for making sharding decisions on the IoT blockchain featured with dynamics and heterogeneity. Specifically, we elaborate on how SmartChain performs reconfiguration and provide a quantitative analysis of shard performance. We then formulate the long-term tradeoff of throughput and security as a Markov decision process. Considering the nature of time-varying devices (e.g., amount of computing power, location), we develop a Transferable Proximal Policy Optimization (PPO) with Demonstrations algorithm, namely TPPOD, to help quickly reconfigure shards when the environment changes. Thus, based on current state, SmartChain can adaptively and dynamically select shard number, partition structure, and primary selection mode. Evaluations show that SmartChain enables high throughput and low risk of security, and reduces 70% of the training time averaged over baselines. Our implementation of TPPOD is 8.3 times of average system reward compared with the PPO-based sharding strategy with uniform sampling.

The Solid (Social Linked Data) project focuses on data sharing and privacy security and aims to build a decentralized ecosystem that radically changes the way web applications work today. Our goal is to introduce a “trust access authentication system” to achieve secure authentication and fine-grained access control, thereby promoting the implementation of Solid. Blockchain, equipped with multiple security properties and authentication functions, is a crucial technology. In this paper, we present a blockchain-assisted system for secure authentication in Solid and for implementation of fine-grained access control policies. Specifically, we explore to integrate threshold RSA signatures in a permissioned blockchain system to enable a fault-tolerant distributed signature scheme, thereby enhancing the resilience and robustness of authentication system. Moreover, we utilize smart contract to control transaction flows and manage access control policies automatically. Experimental results show that our proposed trust access authentication system enhances security, scales well, and is efficient and economically feasible.

Federated Learning (FL) is a promising distributed machine learning framework that allows clients to collaboratively train a global model without data leakage. The synchronous FL suffers from the inefficient training caused by the slow-speed clients, which are called stragglers. Though asynchronous FL can well address the efficiency challenge, it induces massive system overheads and model degradation. As a framework considering the trade-off between synchronous and asynchronous FL, semi-asynchronous FL gains increasing attention. However, when clients' resources become a bottleneck, an unfair client scheduling may degrade global training accuracy and increase system overheads, especially in heterogeneous environments. In this paper, we propose Libra, which is a new FL framework aiming to achieve fair client scheduling in semi-asynchronous FL mode. Libra restricts devices that train overly fast according to the model discrepancy. Furthermore, it selects stale local models according to the number of participating in FL training by clients. Additionally, Libra conducts a biased client selection while considering clients' resources and local losses. The experimental results show that Libra outperforms other baselines in terms of convergence accuracy, system overhead, and fairness of client participation. We also conduct an ablation study to further prove the effectiveness of Libra. In brief, Libra can achieve fair client scheduling and reduce inefficient local updates.