24 June 2022 in IEEE Transactions on Smart Grid
The increasing penetration of distributed renewable energy resources causes voltage fluctuations in distribution networks. The controllable active and reactive power resources such as energy storage (ES) systems and electric vehicles (EVs) in active distribution networks play an important role in mitigating the voltage excursions. This paper proposes a two-timescale hybrid voltage control strategy based on a mixed-integer optimization method and multi-agent reinforcement learning (MARL) to reduce power loss and mitigate voltage violations. In the slow-timescale, the active and reactive power optimization problem involving capacitor banks (CBs), on-load tap changers (OLTC), and ES systems is formulated as a mixed-integer second-order cone programming problem. In the fast-timescale, the reactive power of smart inverters connected to solar photovoltaic systems and active power of EVs are adjusted to mitigate short-term voltage fluctuations with a MARL algorithm. Specifically, we propose an experience augmented multi-agent actor-critic (EA-MAAC) algorithm with an attention mechanism to learn high-quality control policies. The control policies are executed online in a decentralized manner. The proposed hybrid voltage control strategy is validated on an IEEE testing distribution feeder. The numerical results show that our proposed control strategy is not only sample-efficient and robust but also effective in mitigating voltage fluctuations.
6 April 2022 in IEEE Transactions on Smart Grid
The rapid adoption of electric vehicles (EVs) calls for the widespread installation of EV charging stations. To maximize the profitability of charging stations, intelligent controllers that provide both charging and electric grid services are in great need. However, it is challenging to determine the optimal charging schedule due to the uncertain arrival time and charging demands of EVs. In this paper, we propose a novel centralized allocation and decentralized execution (CADE) reinforcement learning (RL) framework to maximize the charging station’s profit. In the centralized allocation process, EVs are allocated to either the waiting or charging spots. In the decentralized execution process, each charger makes its own charging/discharging decision while learning the action-value functions from a shared replay memory. This CADE framework significantly improves the scalability and sample efficiency of the RL algorithm. Numerical results show that the proposed CADE framework is both computationally efficient and scalable, and significantly outperforms the baseline model predictive control (MPC). We also provide an in-depth analysis of the learned action-value function to explain the inner working of the reinforcement learning agent.
10 August 2022 in IEEE Transactions on Smart Grid
The high instantaneous discharging capability of battery energy storage systems (BESSs) make them ideal candidates for reducing peak loads in commercial buildings. An efficient online BESS control algorithm can be beneficial for reducing the monthly electricity bill of individual commercial buildings. Conventional model-based BESS control algorithms rely heavily on accurate long-horizon net load forecast, which is difficult to obtain. To address this problem, we develop a Lyapunov optimization-based online BESS control algorithm and derive its theoretical performance bound. Comprehensive numerical study results using real world commercial building smart meter data in southern California show that our proposed Lyapunov optimization-based online control algorithm with time-varying weighting parameters yields higher savings in electricity cost and requires less computation time in comparison to the state-of-the-art baseline algorithms.
20 July 2024 in Transportation Research Part D: Transport and Environment
13 November 2025 in IEEE Transactions on Automation Science and Engineering
The Electric Vehicle Routing Problem with Time Windows (EVRP-TW) poses significant challenges for sustainable logistics due to its tight coupling of spatial, temporal, and energy constraints. Classical optimization methods face trade-offs: exact solvers like CPLEX ensure optimality but require prohibitive runtimes, while metaheuristics like Variable Neighborhood Search struggle with feasibility under complex constraints. We propose TERRAN, a transformer-based reinforcement learning framework for real-time and scalable EVRP-TW optimization. TERRAN integrates three key components: (1) Future-Feasibility Pruning (FFP), which proactively eliminates energy-infeasible actions by verifying reachability to charging stations or depots before each move; (2) Staged Reward Scheduling, which progressively transitions from dense auxiliary signals to task-aligned rewards to guide the agent from achieving feasibility to minimizing cost; and (3) An End-to-End Transformer-Based RL Agent Tailored for EVRP-TW, which directly integrates EV-specific constraints—including battery consumption, charging decisions, and delivery time windows—into the policy network and decoding process, enabling unified, post-processing-free optimization across varying instance scales. Experiments on Solomon benchmark instances with 5–100 customers demonstrate that TERRAN achieves 100% feasibility across all problem scales. It matches CPLEX’s optimality on 5–customer instances, achieves up to 170,000× speedups with solutions within 1.5% of optimal on 15–customer instances (0.02 s vs. 3,500 s), and delivers feasible solutions for 100–customer instances in 0.47 s, where CPLEX fails on over 80% of cases within 1 hour. These results establish TERRAN as a practical and scalable solution for real-time electric vehicle routing in complex and constraint-rich environments.