As an important application scenario of the Industrial Internet of Things (IIoT), the Internet of Vehicles can significantly improve road safety, improve traffic management efficiency and improve people's travel experience. Due to the high dynamics of the Internet of Vehicles environment, traditional resource optimization technologies cannot meet the requirements of Internet of Vehicles for dynamic communication, computing and storage resources optimization management, and artificial intelligence algorithms can adaptively obtain dynamic resource allocation schemes through self-learning. Therefore, adopting artificial intelligence techniques to optimize dynamic resource of Internet of Vehicles is the research focus of this paper. This paper first models the Internet of Vehicles resource allocation problem as a Semi-Markov Decision Process which introduces a resource reservation strategy and a secondary resource allocation mechanism. Then the reinforcement learning algorithm is used to solve the model. Thereafter, it theoretically analyzes the joint optimization of computing and communication resources, models it as a hierarchical architecture, and uses Hierarchical Reinforcement Learning (HRL) to obtain the optimal system resource allocation plan. Finally, the results of simulation experiments show that the dynamic resource allocation scheme of Internet of Vehicles based on reinforcement learning in this paper greatly improve resource utilization and user Quality of Experience (QoE) with guaranteeing system Quality of Service (QoS) compared with the traditional greedy algorithm.

Heterogeneous computing powered by remote clouds and local fogs is a promising technology to improve the performance of user terminals in the Internet of Things. In this paper, two semi-Markov decision process (SMDP)-based coordinated virtual machine (VM) allocation methods are proposed to balance the tradeoff between the high cost of providing services by the remote cloud and the limited computing capacity of the local fog. We first present a model-based planning method in which it is necessary to train the state transition probabilities and the expected time intervals between adjacent decision epochs. To facilitate training them, the SMDP is degraded into a continuous-time Markov decision process (CTMDP) in which the service requests and ongoing service completions follow a continuous-time Markov chain. The relative value iterative algorithm for the CTMDP is used to find an asymptotically optimal VM allocation policy. In addition, we also propose a model-free reinforcement learning (RL) method, where an optimal coordinated VM allocation policy is approximated by learning from the states and rewards of feedback. The simulation results show that the performance of the model-free RL method can converge to a level similar to that of the model-based planning method and outperform the greedy VM allocation method.

In this paper, we propose a novel adaptive cloud resource allocation model based on Semi-Markov Decision Process (SMDP) and Reinforcement Learning (RL) algorithm in vehicular cloud system. The issue of adaptive resource allocation for vehicular request is formed as an SMDP in order to gain the dynamics of vehicular requests’ arrival and departure. An optimized decision is made to guarantee the Quality of Service (QoS) of the vehicular cloud system and the Quality of Experience (QoE) of the vehicular users as well as to maximize the total system reward of the vehicular cloud system in consideration of the balance between the vehicular cloud resource expense and the system income. Furthermore, to capture the mobility feature of the vehicular cloud system, we also apply a neural-network-based RL algorithm to resolve our proposed SMDP-based adaptive cloud resource allocation model. Firstly, we use a Planning algorithm to get the action values under certain state-action pairs, which are the initial samples to train the neural network. Then the RL is used to update the parameters of the neural network, train the neural network and adaptively improve the decision strategy. Subsequently, an adaptive vehicular cloud resource allocation scheme which can approach the optimal strategy is obtained without the knowledge of the distribution function of vehicular requests’ arrival and departure during the RL process. The simulation results show that our proposed adaptive cloud resource allocation model for vehicular cloud system can reduce the probability of delay in processing requests and achieve high system rewards in comparison with the regularly used greedy resource allocation method. The performance of the RL solution approaches that of traditional value iteration solution for our proposed adaptive cloud resource allocation model.

Due to their frequent use in unattended and hostile deployment environments, the security in wireless sensor networks (WSNs) has attracted much interest in the past two decades. However, it remains a challenge to design a lightweight authentication protocol for WSNs because the designers are confronted with a series of desirable security requirements, e.g., user anonymity, perfect forward secrecy, resistance to de-synchronization attack. Recently, the authors presented two authentication schemes that attempt to provide user anonymity and to resist various known attacks. Unfortunately, in this work we shall show that user anonymity of the two schemes is achieved at the price of an impractical search operation—the gateway node may search for every possible value. Besides this defect, they are also prone to smart card loss attacks and have no provision for perfect forward secrecy. As our main contribution, a lightweight anonymous authentication scheme with perfect forward secrecy is designed, and what we believe the most interesting feature is that user anonymity, perfect forward secrecy, and resistance to de-synchronization attack can be achieved at the same time. As far as we know, it is extremely difficult to meet these security features simultaneously only using the lightweight operations, such as symmetric encryption/decryption and hash functions.

Authentication and authorization constitute the essential security component, access control, for preventing unauthorized access to cloud services in mobile cloud computing (MCC) environments. Traditional centralized access control models relying on third party trust face a critical challenge due to a high trust cost and single point of failure. Blockchain can achieve the distributed trust for access control designs in a mutual untrustworthy scenario, but it also leads to expensive storage overhead. Considering the above issues, this work constructed an authentication and authorization scheme based on blockchain that can provide a dynamic update of access permissions by utilizing the smart contract. Compared with the conventional authentication scheme, the proposed scheme integrates an extra authorization function without additional computation and communication costs in the authentication phase. To improve the storage efficiency and system scalability, only one transaction is required to be stored in blockchain to record a user’s access privileges on different service providers (SPs). In addition, mobile users in the proposed scheme are able to register with an arbitrary SP once and then utilize the same credential to access different SPs with different access levels. The security analysis indicates that the proposed scheme is secure under the random oracle model. The performance analysis clearly shows that the proposed scheme possesses superior computation and communication efficiencies and requires a low blockchain storage capacity for accomplishing user registration and updates.

This paper studies a wireless-powered sensor network, where a sensor harvests energy from a dedicated radio-frequency (RF) energy source and transmits information to an information sink using the harvested energy. Two working modes are considered; One is the frequency division multiplexing (FDM) mode in which the sensor harvests RF energy and transmits information simultaneously over orthogonal frequency bands. The other is the time division multiplexing (TDM) mode in which energy harvesting and information transmission are implemented in the same frequency band but in different time slots. The energy harvesting channel and the information transmission channel are assumed to follow the Rician and the Rayleigh distributions, respectively, and are discretized and modeled as finite-state Markov chains. We formulate the process of energy harvesting and information transmission as an infinite-horizon discounted Markov decision process. The value iteration algorithm is used to find an asymptotically optimal energy harvesting and information transmission policy to optimize the long-term throughput. In the asymptotically optimal policy of the FDM mode, the energy transmitted from the sensor in one slot is proved to be non-decreasing with the battery state of the sensor. By contrast, such monotonicity between the transmitted energy and the battery state does not exist in the asymptotically optimal policy in the TDM mode. Simulation results verify the above findings and demonstrate that the proposed method outperforms the heuristic greedy method.