4 June 2020 in IEEE Transactions on Power Systems
This paper proposes a data-driven cooperative method for load frequency control (LFC) of the multi-area power system based on multi-agent deep reinforcement learning (MA-DRL) in continuous action domain. The proposed method can nonlinearly and adaptively derive the optimal coordinated control strategies for multiple LFC controllers through centralized learning and decentralized implementation. The centralized learning is achieved by MA-DRL based on a global action-value function to quantify overall LFC performance of the power system. To solve the MA-DRL problem, multi-agent deep deterministic policy gradient (DDPG) is derived to adjust control agents' parameters considering the nonlinear generator behaviors. For implementation, each individual controller only needs local information in its control area to deliver optimal control signals. Numerical simulations on a three-area power system and the fully-modeled New-England 39-bus system demonstrate that the proposed method can effectively minimize control errors against stochastic frequency variations caused by load and renewable power fluctuations.
14 November 2018 in IEEE Transactions on Power Systems
This letter proposes a data-driven, model-free method for load frequency control (LFC) against renewable energy uncertainties based on deep reinforcement learning (DRL) in continuous action domain. The proposed method can nonlinearly derive control strategies to minimize frequency deviation with faster response speed and stronger adaptability for unmolded system dynamics. It consists of (i) offline optimization of LFC strategies with DRL and continuous action search, and (ii) online control with policy network where features are extracted by stacked denoising auto-encoders (SDAE). Numerical simulations verify the effectiveness and advantages of proposed method over existing approaches.
16 April 2020 in IEEE Transactions on Power Systems
High-level penetration of intermittent renewable energy sources has introduced significant uncertainties and variabilities into modern power systems. In order to rapidly and economically respond to the changes in power system operating state, this letter proposes a real-time optimal power flow (RT-OPF) approach using Lagrangian-based deep reinforcement learning (DRL) in continuous action domain. A DRL agent to determine RT-OPF decisions is constructed and optimized using the deep deterministic policy gradient. The DRL action-value function is designed to simultaneously model RT-OPF objective and constraints. Instead of using the critic network, the deterministic gradient is derived analytically. The proposed method is tested on the IEEE 118-bus system. Compared with the state-of-the-art methods, the proposed method can achieve a high solution optimality and constraint compliance in real-time.
4 December 2023 in IEEE Transactions on Power Systems
Practical operations of a power system need to comply with Grid Codes, which are usually grounded in linguistic stipulations that may be hard to quantify in conventional optimal power flow (OPF) models. In the lights of recent breakthrough in large language models (LLM), this letter proposes an OPF model with linguistic stipulations and a solution method by integrating the generative pre-trained transformer (GPT) based LLM agent in the primal-dual deep reinforcement learning (DRL) training loop. For the first time, conventionally unquantifiable linguistic stipulations expressed in natural language can be directly modeled as the objectives and constraints in the OPF problem. The GPT-agent is used to interpret satisfactions of linguistic stipulations as rewards and constraints. The non-differentiable rewards obtained by GPT-agent are optimized through interactions with environments in the DRL process. Once sufficiently trained, the DRL agent can solve the OPF model in real-time. The proposed method is demonstrated on the IEEE 118-bus system.
1 March 2021 in IEEE Transactions on Smart Grid
This paper studies the vulnerability of deep reinforcement learning (DRL) models for power systems topology optimization under data perturbations and cyber-attack. DRL has recently solved many complex power system optimization problems. However, it has been practically proven that small perturbations of input data can lead to drastically different control decisions and induce danger. To evaluate and mitigate the security risks of DRL models in power systems, we propose a vulnerability assessment method for such DRL models under noisy data and cyber-attack. In specific, we assess the vulnerability of a DRL model in a way that perturbations are constructed to minimize the model's performance. Besides, several vulnerability indices are proposed to identify the characteristics of perturbations that may cause malfunction of DRL. Simulations on the 14-bus system and the IEEE 118-bus system for topology optimization are carried out to validate the effectiveness of the proposed vulnerability assessment method. The results show that the performance of DRL models for power systems can be significantly degraded under cyber-attack and data perturbations, especially when a proposed vulnerability index has abnormal values.
10 February 2022 in IEEE Transactions on Power Systems
This paper proposes a hybrid data-driven method for fast solutions of preventive security-constrained optimal power flow (SCOPF) of power systems. The proposed method formulates the SCOPF problem as constraints-satisfying training of a deep reinforcement learning (DRL) agent, where the action-value function of DRL is augmented by contingency security constraints. In the training process, the proposed method hybridizes the primal-dual deep deterministic policy gradient and the classic SCOPF model. Instead of building reward critic networks and cost critic networks via interacting with the environment (i.e., power flow equations), the actor gradients are approximated by solving KKT conditions of the Lagrangian. Finally, with the formulated sparse Jacobians of constraints and sparse Hessians of Lagrangians, the interior point method is incorporated in primal-dual DDPG to derive the parameters updating rule of the DRL agent. Numerical tests are carried out on a modified IEEE 57-bus system and a modified IEEE 300-bus system for critical contingencies. The results show that the well-trained DRL agent can rapidly (real-time) obtain high-quality SCOPF solutions that satisfy the security constraints.