For a battery swapping station (BSS) with solar generation, N charging bays, and an inventory of M batteries, we study the charging scheduling problem under random EV arrivals, renewable generation, and electricity prices. To minimize the expected weighted sum of charging cost (sum of electricity and battery degradation costs) and EV owners’ waiting cost, we formulate the problem as a Markov decision process with unknown state transition probability. Under a mild heavy-traffic assumption, we rigorously establish the optimality of the Less Demand First (LDF) priority rule under arbitrary system dynamics: batteries with less demand shall be charged first. The optimality result enables us to integrate the LDF rule into a state-of-the-art deep reinforcement learning (DRL) method, proximal policy optimization (PPO), reducing the dimensionality of its output from O(M+N) to O(1), without loss of optimality in the heavy-traffic scenario. Numerical results (on real-world data) demonstrate that the proposed LDF enhanced PPO approach significantly outperforms classical DRL methods and FCFS (first come, first served) priority rule based DRL counterparts.

We study the charging routing problem faced by a smart electric vehicle (EV) that looks for an EV charging station (EVCS) to fulfill its battery charging demand. Leveraging the real-time information from both the power and intelligent transportation systems, the EV seeks to minimize the sum of the travel cost (to a selected EVCS) and the charging cost (a weighted sum of electricity cost and waiting time cost at the EVCS). We formulate the problem as a Markov decision process with unknown dynamics of system uncertainties (in traffic conditions, charging prices, and waiting time). To mitigate the curse of dimensionality, we reformulate the deterministic charging routing problem (a mixed-integer program) as a two-level shortest-path (SP)-based optimization problem that can be solved in polynomial time. Its low dimensional solution is input into a state-of-the-art deep reinforcement learning (DRL) algorithm, the advantage actor–critic (A2C) method, to make efficient online routing decisions. Numerical results (on a real-world transportation network) demonstrate that the proposed SP-based A2C approach outperforms the classical A2C method and two alternative SP-based DRL methods.

In this article, we study the online pricing and charging scheduling problem for a public electric vehicle (EV) charging station under stochastic electricity prices and renewable generation. We formulate the sequential decision making problem as a partially observable Markov decision process with continuous state and action spaces, with an objective of profit or social welfare maximization. The joint pricing and charging problem is challenging due to the curse of dimensionality (in both the system state and action spaces) and the unknown dynamics of system uncertainties. We propose a novel laxity differentiated pricing (LDP) scheme to tradeoff between electricity cost (associated with EV charging) and opportunity cost (associated with parking infrastructure occupancy). Shown to be optimal under arbitrary system dynamics, the characterized deadline-differentiated threshold charging (DTC) policy is integrated into a model-free soft actor critic (SAC) algorithm to reduce the action dimensionality. Numerical results demonstrate that the proposed SAC + LDP + DTC approach significantly outperforms alternative methods with various pricing and charging schemes.

This work studies the joint electric vehicle (EV) routing and battery charging scheduling problem in a transportation network with multiple battery swapping stations (BSSs) under random EV swapping demands, renewable generation, and electricity prices. The joint scheduling problem is formulated as a decentralized partially observable Markov decision process (Dec-POMDP) with an objective to minimize the expected sum of the battery charging cost and the travel/waiting cost of EV owners. The formulated Dec-POMDP is hard to solve, due the curse of dimensionality and the unknown system dynamics. To tackle the challenges, we propose a new heterogeneous multiagent hypergraph attention actor-critic (HMA-HGAAC) framework, which integrates hypergraph attention (HGAT) networks to multiagent deep reinforcement learning (MADRL) to enhance the learning efficiency with a hypergraph where multiple nodes can be connected by a single hyperedge. Numerical experiments based on real-world data and a 180-node transportation network show that the proposed approach can save the system cost achieved by state-of-the-art benchmarks, independent proximal policy optimization (IPPO), multiagent proximal policy optimization (MAPPO), and heterogeneous multiagent graph attention proximal policy optimization (HMA-GAPPO), by 23.5%, 18.9%, and 13.3%, respectively.

For a serving system with multiple servers and a public queue, we study the scheduling of multiple tasks with deadlines, under random task arrivals and renewable energy generation. To minimize the weighted sum of the serving cost (associated with the energy consumption) and the delay cost (resulting from deferring the processing of tasks after their deadlines), we formulate the problem as a dynamic program with unknown transition probability. To mitigate the curse of dimensionality, we establish a partial priority rule, the earlier deadline and less demand first (ED-LDF): priority should be given to tasks with earlier deadline and less demand. In the heavy-traffic regime, the established ED-LDF characterization is proved to be optimal under arbitrary system dynamics. We propose a new, scalable ED-LDF-based proximal policy optimization (PPO) approach that integrates our (partial) optimal policy characterizations into the state-of-the-art deep reinforcement learning (DRL) algorithm. Numerical results demonstrate that the proposed ED-LDF-based PPO approach outperforms the classical PPO and three other priority rule-based PPO approaches.