29 April 2024 in IEEE Transactions on Power Systems
In this paper, a deep reinforcement learning (DRL)-based electric vehicles (EVs) management strategy is proposed to achieve peak shaving and regulate the voltage violations in distribution networks. We present a new approach for modeling the EV user willingness considering the effect of multi-attribute attitudes and price incentives on user’ behaviors. The real-time optimal regulation problem is modeled as a bi-level optimization and then formulated as a constrained Markov decision process. To solve the problem, we first define an EV energy availability criterion for K-means clustering method to assess the EV scheduling potential for hierarchical dispatching, which facilitates meeting the user charging demand constraint. Then we present a bi-level safe soft actor critic algorithm with optimal convergence guarantee to solve the optimal sequential gaming problem between distribution system operator and charging stations aggregator. For each constraint, we employ individual neural networks to compute the cost Q-function and update independent Lagrange multiplier, which contributes to the setting of expected cost thresholds for practical application. In addition, we develop a data-enhanced method using Gaussian mixture model with a two-stage training process to improve the data efficiency and model robustness, as well as avoiding the constraint violations in the start-up process. Test results on the modified IEEE 33-bus and 118-bus systems with the real-world data are presented to validate the effectiveness of the proposed approach.
18 March 2024 in IEEE Transactions on Sustainable Energy
Inverter-based Volt-VAR control (IB-VVC) can be simplified as a single-period optimization problem. However, recently DRL methods formulate IB-VVC as a Markov decision process (MDP) and solve it as a multi-period optimization problem. It complicates the IB-VVC problem and degrades the performance of deep reinforcement learning (DRL) algorithms. To avoid this, this paper formulates the inverter-based VVC as a one-step MDP and designs a single-period DRL algorithm to solve the problem. It simplifies the DRL approach considerably and accelerates the convergence rate as well as the control performance. Since VVC has two aims: eliminating voltage violations and minimizing power loss, those two objectives have different profiles. Existing DRL methods use one critic neural network to approximate the two objectives together without considering their special property. It increases the approximate difficulty of the critic neural networks in the training process. To alleviate it, we design a two-critic approach. It approximates the two objective functions by two critic neural networks separately. It has a better approximation capability, thus accelerating the convergence rate and improving the control performance of the DRL method further. Based on the single-period two-critic DRL (TC) approach, we design two DRL algorithms: 1) TC-DDPG with deterministic policy and 2) TC-SAC with stochastic policy. Further, we extend the TC-DRL to multi-agent TC to show it scales well for multi-agent DRL algorithms. Simulations conducted on 33-bus and 69-bus test distribution networks demonstrate the superiority of the proposed approach in both single-agent DRL algorithms and multi-agent DRL algorithms. A two-critic deep reinforcement learning (TC-DRL) approach for inverter-based volt-var control (IB-VVC) in active distribution networks is proposed in this paper. Considering two objectives of VVC, minimizing power loss and eliminating voltage violations, have different mathematical properties, we utilize two critics to approximate two objectives separately, which reduces the learning difficulties of each critic. The TC-DRL approach cooperates well with many actor-critic DRL algorithms for the centralized IB-VVC problems, and two centralized DRL algorithms were designed as examples. For decentralized IB-VVC, we extend the approach to a multi-agent TC-DRL approach and further simplify the multi-agent DRL approach with all agents sharing the same centralized two-critic. Extensive simulation experiments show that the proposed two centralized TC-DRL algorithms require fewer iteration times and return better results than the recent DRL algorithms, and the multi-agent TC-DRL algorithms work well for decentralized IB-VVC problems with different limited real-time measurement conditions.
19 April 2021 in IEEE Transactions on Artificial Intelligence
Composite adaptive radial basis function neural network (RBFNN) control with a lattice distribution of hidden nodes has three inherent demerits: 1) the approximation domain of adaptive RBFNNs is difficult to be determined a priori ; 2) only a partial persistence of excitation (PE) condition can be guaranteed; 3) in general, the required number of hidden nodes of RBFNNs is enormous. This article proposes an adaptive feedforward RBFNN controller with an optimized distribution of hidden nodes to suitably address the above demerits. The distribution of the hidden nodes calculated by a K-means algorithm is optimally distributed along the desired state trajectory. The adaptive RBFNN satisfies the PE condition for the periodic reference trajectory. The weights of all hidden nodes will converge to the optimal values. This proposed method considerably reduces the number of hidden nodes, while achieving a better approximation ability. The proposed control scheme shares a similar rationality to that of the classical PID control in two special cases, which can thus be seen as an enhanced PID scheme with a better approximation ability. For the controller implemented by digital devices, the proposed method, for a manipulator with unknown dynamics, potentially achieves better control performance than model-based schemes with accurate dynamics. Simulation results demonstrate the effectiveness of the proposed scheme. This result provides a deeper insight into the coordination of the adaptive neural network control and the deterministic learning theory. Impact Statement —Adaptive RBFNN control learns to control a robot manipulator when both the structures and parameters of the target robot are unknown in advance. Unfortunately, current adaptive RBFNN controllers need a large-scale neural network to approximate the dynamics of the robot manipulator, and the learning performance cannot be guaranteed to converge. The proposed method in this article not only reduces the scale of neural networks to substantially alleviate the computational burden but also evidently achieves better learning performance. Simulation examples show that this method increases the control accuracy by more than nine times and reduces the scale of neural networks by 35 times as compared to the traditional lattice scheme. Intuitively, people usually believe that a model-based controller with an accurate dynamic model may achieve the best control performance. However, compared with the model-based controller with an accurate dynamic model, the proposed control scheme with an unknown dynamic model even further increases the control accuracy by 1.5 times. This technology provides a more straightforward path for engineers, who may be not experts in complicated control system analysis methods, to design an adaptive robotic controller to achieve enhanced performance.
25 July 2025 in Applied Energy
3 September 2024 in IEEE Transactions on Sustainable Energy
A residual deep reinforcement learning (RDRL) based on an approximate-model-driven optimization approach is proposed for inverter-based volt-var control (IB-VVC) in active distribution networks. A modified Markov decision process is introduced to formulate the model-based and RDRL-based IB-VVC simultaneously, and then RDRL learns a residual action based on the action of the model-based approach with an approximate model. It inherits the control capability of the approximate-model-based optimization and enhances the policy optimization capability by residual policy learning. Since the approximate model acquired by operators is generally relatively reliable, the action solved by model-based optimization approaches is not far away from the optimal one. This allows RDRL to search for the residual action in a smaller residual action space, which further improves the approximation accuracy of the critic and reduces the search difficulties of the actor. Simulations demonstrate that RDRL improves the optimization performance considerably throughout the learning stage and verifies their three rationales for superior performance point-by-point on 69 and 141 bus balanced distribution networks.
27 February 2026 in Applied Energy