Volt-VAR control is critical to keeping distribution network voltages within allowable range, minimizing losses, and reducing wear and tear of voltage regulating devices. To deal with incomplete and inaccurate distribution network models, we propose a safe off-policy deep reinforcement learning algorithm to solve Volt-VAR control problems in a model-free manner. The Volt-VAR control problem is formulated as a constrained Markov decision process with discrete action space, and solved by our proposed constrained soft actor-critic algorithm. Our proposed reinforcement learning algorithm achieves scalability, sample efficiency, and constraint satisfaction by synergistically combining the merits of the maximum-entropy framework, the method of multiplier, a device-decoupled neural network structure, and an ordinal encoding scheme. Comprehensive numerical studies with the IEEE distribution test feeders show that our proposed algorithm outperforms the existing reinforcement learning algorithms and conventional optimization-based approaches on a large feeder.

The increasing penetration of distributed renewable energy resources causes voltage fluctuations in distribution networks. The controllable active and reactive power resources such as energy storage (ES) systems and electric vehicles (EVs) in active distribution networks play an important role in mitigating the voltage excursions. This paper proposes a two-timescale hybrid voltage control strategy based on a mixed-integer optimization method and multi-agent reinforcement learning (MARL) to reduce power loss and mitigate voltage violations. In the slow-timescale, the active and reactive power optimization problem involving capacitor banks (CBs), on-load tap changers (OLTC), and ES systems is formulated as a mixed-integer second-order cone programming problem. In the fast-timescale, the reactive power of smart inverters connected to solar photovoltaic systems and active power of EVs are adjusted to mitigate short-term voltage fluctuations with a MARL algorithm. Specifically, we propose an experience augmented multi-agent actor-critic (EA-MAAC) algorithm with an attention mechanism to learn high-quality control policies. The control policies are executed online in a decentralized manner. The proposed hybrid voltage control strategy is validated on an IEEE testing distribution feeder. The numerical results show that our proposed control strategy is not only sample-efficient and robust but also effective in mitigating voltage fluctuations.

Volt-VAR control (VVC) is a critical application in active distribution network management system to reduce network losses and improve voltage profile. To remove dependency on inaccurate and incomplete network models and enhance resiliency against communication or controller failure, we propose consensus multi-agent deep reinforcement learning algorithm to solve the VVC problem, which determines the operation schedules for voltage regulators, on-load tap changers, and capacitors. The VVC problem is formulated as a networked multi-agent Markov decision process, which is solved using the maximum entropy reinforcement learning framework and a novel communication-efficient consensus strategy. The proposed algorithm allows individual agents to learn a group control policy using local rewards. Numerical studies on IEEE distribution test feeders show that our proposed algorithm matches the performance of single-agent reinforcement learning benchmark. In addition, the proposed algorithm is shown to be communication efficient and resilient.

Dynamic distribution network reconfiguration (DNR) algorithms perform hourly status changes of remotely controllable switches to improve distribution system performance. The problem is typically solved by physical model-based control algorithms, which not only rely on accurate network parameters but also lack scalability. To address these limitations, this paper develops a data-driven batch-constrained reinforcement learning (RL) algorithm for the dynamic DNR problem. The proposed RL algorithm learns the network reconfiguration control policy from a finite historical operational dataset without interacting with the distribution network. The numerical study results on three distribution networks show that the proposed algorithm not only outperforms state-of-the-art RL algorithms but also improves the behavior control policy, which generated the historical operational data. The proposed algorithm is also very scalable and can find a desirable network reconfiguration solution in real-time.

Active distribution networks are being challenged by frequent and rapid voltage violations due to renewable energy integration. Conventional model-based voltage control methods rely on accurate parameters of the distribution networks, which are difficult to achieve in practice. This paper proposes a novel physical-model-free two-timescale voltage control framework for active distribution systems. To achieve fast control of PV inverters, the whole network is first partitioned into several sub-networks using voltage-reactive power sensitivity. Then, the scheduling of PV inverters in the multiple sub-networks is formulated as Markov games and solved by a multi-agent soft actor-critic (MASAC) algorithm, where each sub-network is modeled as an intelligent agent. All agents are trained in a centralized manner to learn a coordinated strategy while being executed based on only local information for fast response. For the slower time-scale control, OLTCs and switched capacitors are coordinated by a single agent-based SAC algorithm using the global information with considering control behaviors of the inverters. Particularly, the two-level agents are trained concurrently with information exchange according to the reward signal calculated from the data-driven surrogate model. Comparative tests with different benchmark methods on IEEE 33- and 123-bus systems and 342-node low voltage distribution system demonstrate that the proposed method can effectively mitigate the fast voltage violations and achieve systematical coordination of different voltage regulation assets without the knowledge of accurate system model.