Electric vehicle (EV) has become one of the most critical components in the smart grid with the applications of the Internet of Things (IoT) technologies. Real-time charging control is pivotal to ensure the efficient operation of EVs. However, the charging control performance is limited by the uncertainty of the environment. On the other hand, it is challenging to determine a charging control strategy that is able to optimize multiple objectives simultaneously. In this paper, we formulate EV charging control model as a Markov decision process (MDP) by constructing state, action, transition function, and reward. Then, we propose a deep reinforcement learning-based approach: charging control deep deterministic policy gradient (CDDPG) to learn the optimal charging control strategy for satisfying user’s requirement of battery energy while minimizing user’s charging expense. We utilize the Long short-term memory (LSTM) network which extracts the information of previous energy price to determine the current charging control strategy. Moreover, Gaussian noise is added to the output of the actor network to prevent the agent from sticking into the non-optimal strategy. In addition, we address the limitation of sparse rewards by using two replay buffers, of which one is used to store the rewards during the charging phase, and another is used to store the rewards after the charging is completed. Simulation results prove that the CDDPG-based approach outperforms the deep Q learning-based approach (DQL) and the deep deterministic policy gradient-based approach (DDPG) in satisfying the user’s requirement for the battery energy and reducing the charging cost.

An efficient energy scheduling strategy of a charging station is crucial for stabilizing the electricity market and accommodating the charging demand of electric vehicles (EVs). Most of the existing studies on energy scheduling strategies fail to coordinate the process of energy purchasing and distribution and, thus, cannot balance the energy supply and demand. Besides, the existence of multiple charging stations in a complex scenario makes it difficult to develop a unified schedule strategy for different charging stations. In order to solve these problems, we propose a multiagent reinforcement learning (MARL) method to learn the optimal energy purchasing strategy and an online heuristic dispatching scheme to develop a energy distribution strategy in this article. Unlike the traditional scheduling methods, the two proposed strategies are coordinated with each other in both temporal and spatial dimensions to develop the unified energy scheduling strategy for charging stations. Specifically, the proposed MARL method combines the multiagent deep deterministic policy gradient (MADDPG) principles for learning purchasing strategy and a long short-term memory (LSTM) neural network for predicting the charging demand of EVs. Moreover, a multistep reward function is developed to accelerate the learning process. The proposed method is verified by comprehensive simulation experiments based on real data of the electricity market in Chicago. The experiment results show that the proposed method can achieve better performance than other state-of-the-art energy scheduling methods in the charging market in terms of the economic profits and users’ satisfaction ratio.

A smart grid integrates advanced sensors, efficient measurement methods, progressive control technologies, and other techniques and devices to achieve safe, efficient and economical operation of the grid system. However, the diversified and open environment of a smart grid makes energy and information of the smart grid vulnerable to malicious attacks. As a representative cyber-physical attack, the data integrity attack has an extremely severe impact on the grid operation for it can bypass the traditional detection mechanisms by adjusting the attack vector. In this paper, we first present the attack strategy against dynamic state estimation of power grid in the perspective of adversary and formulate the data integrity attack detection problem that has the characteristic of sequential decision making as a partially observable Markov decision process. Then, a deep reinforcement learning-based approach is proposed to detect against data integrity attacks, which utilizes the Long Short-Term Memory layer to extract the state features of previous time steps in determining whether the system is currently under attack. Moreover, the noisy networks are employed to ensure effective agent exploration, which prevents the agent from sticking to the non-optimal policy. The principle of a multi-step learning is adopted to increase the estimation accuracy of Q value. To address the sparse rewards problem, the prioritized experience replay is proposed to increase training efficiency. Simulation results demonstrated that the proposed detection approach surpasses the benchmarks in the comparison metrics: delay error rate and false rate. Note to Practitioners —In this paper, we present a deep reinforcement learning-based algorithm to defend against the data integrity attacks of smart grid. Most of the previous works discretized the system states and utilized the current state information to identify whether the system is under attack. For this reason, the detection policy may totally ignored the continuously changing characteristics of the grid states, which will lead to poor detection performance. Moreover, the attacked system states only accounts for a small part of the entire grid operation states, the probability of sampling the experience containing the attack state is extremely small, which limits the learning efficiency of previous RL-based detection approaches. In order to increase the accuracy of detection, we first present the attack strategy against power grid’s dynamic state estimation in the perspective of adversary and formulate the partially observable Markov decision process model of attack detection problem. Moreover, we propose a deep reinforcement learning-based detection approach combining the LSTM network to extract the system state features of the previous time steps to determine whether the system is currently being attacked. To address the sparse rewards problem, the prioritized experience replay is used to increase learning efficiency. The experiments demonstrate the effectiveness of proposed detection scheme compared with benchmarks in terms of detection delay as well as accuracy. In conclusion, the proposed detection scheme is helpful in defending against the data integrity attacks without obtaining the opponent’s strategy in advance and can be conveniently applied to the real-world security management system of smart grid.

In this paper, to address the issue of demand response in the smart grid with island MicroGrids (MGs), we introduce an effective and secure auction market that allows Electric Vehicles (EVs) having surplus energy to act as sellers, and the EVs having insufficient energy in the island MGs to act as buyers. There are two primary challenges in designing an effective auction market in the smart grid. First, the auction market scheme shall be online, allowing buyers and sellers to enter the market at any time, and satisfy several critical economic properties (individual rationality, incentive compatibility, etc.). Second, the sensitive information of participants shall be protected in the auction process. To address these challenges, we present a novel privacy-preserving online double auction scheme based on differential privacy. In our auction market, the MicroGrid Center Controller (MGCC) acts as the auctioneer, aiming at solving the social welfare maximization problem to match buyers and sellers. The principle of differential privacy is leveraged to protect the privacy of EVs’ sensitive bidding information. Via theoretical analysis, we demonstrate that our designed auction scheme satisfies both economic and privacy-preserving properties, including individual rationality, incentive compatibility, weak budget balance and ε-differential privacy. We conduct an extensive performance evaluation to measure the effectiveness of our proposed scheme. Our experimental results show that the proposed auction scheme can not only ensure the privacy of participants, but also effectively facilitates demand response in the smart grid, with respect to social welfare, satisfaction ratio, social efficiency, and computational overhead.

In this paper, we address the issue of demand response in microgrids via V2V (Vehicle-to-Vehicle) technology in the smart grid with consideration for location privacy protection supported by Internet of Vehicles (IoV). To enable effective demand response, the online double auction is a viable approach to support energy trading between EVs that have surplus or insufficient energy, while the utility of each participant can be considered. Nonetheless, there are three primary challenges in designing such an online double auction approach. First, as EVs are allowed to enter the market at any time, the auctioneer should make the best decision without further information about bids and asks. Second, as EVs are allowed to enter the market in different places, the auctioneer should perform routing optimization for EV charging after determining the winner. Third, there is a risk of leakage in the location of EVs that needs to be protected. To tackle these issues, we present a new truthful online double auction scheme, which features multi-unit energy trading among EVs, routing optimization for EV charging, and location privacy protection. We conduct a theoretical analysis and demonstrate that our online double auction scheme is capable of achieving several important economic properties as well as the privacy guarantee (i.e., k-anonymity). Our experimental results show that the proposed scheme can achieve good performance with respect to social welfare, satisfaction ratio, total profit of EV owners, peak load shifting, state of charge, driving distance satisfaction, and computing time, and can further ensure location privacy protection.

In the smart grid, measurement devices may be compromised by adversaries, and their operations could be disrupted by attacks. A number of schemes to efficiently and accurately detect these compromised devices remotely have been proposed. Nonetheless, most of the existing schemes detecting compromised devices depend on the incremental response time in the attestation process, which are sensitive to data transmission delay and lead to high computation and network overhead. To address the issue, in this paper, we propose a low-cost remote memory attestation scheme (LRMA), which can efficiently and accurately detect compromised smart meters considering real-time network delay and achieve low computation and network overhead. In LRMA, the impact of real-time network delay on detecting compromised nodes can be eliminated via investigating the time differences reported from relay nodes. Furthermore, the attestation frequency in LRMA is dynamically adjusted with the compromised probability of each node, and then, the total number of attestations could be reduced while low computation and network overhead can be achieved. Through a combination of extensive theoretical analysis and evaluations, our data demonstrate that our proposed scheme can achieve better detection capacity and lower computation and network overhead in comparison to existing schemes.