13 March 2019 in IEEE Transactions on Smart Grid
18 March 2024 in IEEE Transactions on Sustainable Energy
Inverter-based Volt-VAR control (IB-VVC) can be simplified as a single-period optimization problem. However, recently DRL methods formulate IB-VVC as a Markov decision process (MDP) and solve it as a multi-period optimization problem. It complicates the IB-VVC problem and degrades the performance of deep reinforcement learning (DRL) algorithms. To avoid this, this paper formulates the inverter-based VVC as a one-step MDP and designs a single-period DRL algorithm to solve the problem. It simplifies the DRL approach considerably and accelerates the convergence rate as well as the control performance. Since VVC has two aims: eliminating voltage violations and minimizing power loss, those two objectives have different profiles. Existing DRL methods use one critic neural network to approximate the two objectives together without considering their special property. It increases the approximate difficulty of the critic neural networks in the training process. To alleviate it, we design a two-critic approach. It approximates the two objective functions by two critic neural networks separately. It has a better approximation capability, thus accelerating the convergence rate and improving the control performance of the DRL method further. Based on the single-period two-critic DRL (TC) approach, we design two DRL algorithms: 1) TC-DDPG with deterministic policy and 2) TC-SAC with stochastic policy. Further, we extend the TC-DRL to multi-agent TC to show it scales well for multi-agent DRL algorithms. Simulations conducted on 33-bus and 69-bus test distribution networks demonstrate the superiority of the proposed approach in both single-agent DRL algorithms and multi-agent DRL algorithms. A two-critic deep reinforcement learning (TC-DRL) approach for inverter-based volt-var control (IB-VVC) in active distribution networks is proposed in this paper. Considering two objectives of VVC, minimizing power loss and eliminating voltage violations, have different mathematical properties, we utilize two critics to approximate two objectives separately, which reduces the learning difficulties of each critic. The TC-DRL approach cooperates well with many actor-critic DRL algorithms for the centralized IB-VVC problems, and two centralized DRL algorithms were designed as examples. For decentralized IB-VVC, we extend the approach to a multi-agent TC-DRL approach and further simplify the multi-agent DRL approach with all agents sharing the same centralized two-critic. Extensive simulation experiments show that the proposed two centralized TC-DRL algorithms require fewer iteration times and return better results than the recent DRL algorithms, and the multi-agent TC-DRL algorithms work well for decentralized IB-VVC problems with different limited real-time measurement conditions.
3 September 2024 in IEEE Transactions on Sustainable Energy
A residual deep reinforcement learning (RDRL) based on an approximate-model-driven optimization approach is proposed for inverter-based volt-var control (IB-VVC) in active distribution networks. A modified Markov decision process is introduced to formulate the model-based and RDRL-based IB-VVC simultaneously, and then RDRL learns a residual action based on the action of the model-based approach with an approximate model. It inherits the control capability of the approximate-model-based optimization and enhances the policy optimization capability by residual policy learning. Since the approximate model acquired by operators is generally relatively reliable, the action solved by model-based optimization approaches is not far away from the optimal one. This allows RDRL to search for the residual action in a smaller residual action space, which further improves the approximation accuracy of the critic and reduces the search difficulties of the actor. Simulations demonstrate that RDRL improves the optimization performance considerably throughout the learning stage and verifies their three rationales for superior performance point-by-point on 69 and 141 bus balanced distribution networks.
17 March 2026 in IEEE Transactions on Smart Grid
The rapid growth of cloud-edge collaborative computing and its increasing electricity demand are forging strong interdependencies between computing and power markets. This paper proposes a market-driven joint trading strategy that coordinates computing services and electricity procurement within a cloud-edge collaborative system. To capture heterogeneous operational characteristics, we introduce a differentiated modeling paradigm, combining a task-queuing model for the cloud data center with an empirically-grounded nonlinear delay model for edge servers to characterize its performance saturation. These models are then embedded in a unified trading framework that explicitly characterizes the bidirectional interaction between computing and electricity markets, where the cloud acts as a price-making participant. The resulting large-scale, non-convex problem is transformed into a tractable mixed-integer second-order cone program by applying targeted convexification techniques—specifically, second-order cone relaxation for nonlinear delay constraints and special ordered sets of type 2 for bilinear market-clearing terms. Numerical case studies demonstrate that the differentiated modeling paradigm reduces the overall operational cost of the test system by 16.3%, including both electricity and delay costs. Furthermore, the solution methodology ensures high scalability and computational efficiency for large-scale instances while maintaining an optimality gap below 0.3%.
1 July 2026 in IEEE Transactions on Sustainable Energy