Utility grids are undergoing several upgrades. Distributed generators that are supplied by intermittent renewable energy sources (RES) are being connected to the grids. As RES get cheaper, more customers are opting for peer-to-peer energy interchanges through the smart metering infrastructure. Consequently, power management in grid-tied RES-based microgrids has become a challenging task. This paper reviews the applications of reinforcement learning (RL) algorithms in managing power in grid-tide microgrids. Unlike other optimization methods such as numerical and soft computing techniques, RL does not require an accurate model of the optimization environment in order to arrive at an optimal solution. In this paper, various challenges associated with the control of power in grid-tied microgrids are described. The application of RL techniques in addressing those challenges is reviewed critically. This review identifies the need to improve and scale multi-agent RL methods to enable seamless distributed power dispatch among interconnected microgrids. Finally, the paper gives directions for future research, e.g., the hybridization of intrinsic and extrinsic reward schemes, the use of transfer learning to improve the learning outcomes of RL in complex power systems environments and the deployment of priority-based experience replay in post-disaster microgrid power flow control.

The high penetration of electric vehicles (EVs) will burden the existing power delivery infrastructure if their charging and discharging are not adequately coordinated. Dynamic pricing is a special form of demand response that can encourage EV owners to participate in scheduling programs. Therefore, EV charging and discharging scheduling and its dynamic pricing model are important fields of study. Many researchers have focused on artificial intelligence-based EV charging demand forecasting and scheduling models and suggested that artificial intelligence techniques perform better than conventional optimization methods such as linear, exponential, and multinomial logit models. However, only a few research studies focused on EV discharging scheduling (i.e., vehicle-to-grid, V2G) because the concept of EV discharging electricity back to the power grid is relatively new and evolving. Therefore, a review of existing EV charging and discharging-related studies is needed to understand the research gaps and to make some improvements in future studies. This paper reviews EV charging and discharging-related studies and classifies them into forecasting, scheduling, and pricing mechanisms. The paper determines the linkage between forecasting, scheduling, and pricing mechanism and identifies the research gaps in EV discharging scheduling and dynamic pricing models.

The use of photovoltaic (PV) arrays in smart grid systems is growing due to the increasing energy demand and greenhouse gas emissions. However, due to the intermittent nature of PV arrays, the Maximum Power Point Tracking (MPPT) algorithm is typically employed to optimize the system’s energy production. In the past, the conventional perturb and observe (P&O) method was proposed for solar PV MPPT control. While the P&O method can estimate the PV maximum power under uniform irradiation, it often exhibits sluggish tracking and unstable steady-state oscillations and fails to track the global maximum power point (GMPP) under partial shading conditions (PSCs). These problems have been addressed using deep reinforcement learning (RL) algorithms, such as the deep deterministic policy gradient (DDPG) algorithm. However, due to the DDPG’s intrinsic drawbacks, such as unstable training, Q-value overestimation, brittle convergence, and hyperparameter sensitivity, it often produces steady-state power oscillations near the GMPP under PSCs, resulting in power loss. This paper presents a soft actor-critic (SAC) algorithm, for solving solar PV MPPT control problems under PSCs. Unlike DDPG, which utilizes only one Q-network in the critic, SAC utilizes two Q-networks in the critic and maximum entropy policy in the reward function, which guarantees its training stability and improves its exploration and robustness in the presence of “estimation and model errors”. Despite its potential, the SAC-based MPPT approach has not been extensively explored or compared with DDPG to determine the superior method for PV MPPT control. This paper provides an adequate comparison between the performances of DDPG and SAC, including their optimal hyperparameter configurations, for PV MPPT control. To solve the MPPT control problem, the mathematical model of the boost converter and the solar PV system were developed. Then, a Markov Decision Process model was formulated, which represents the PV system’s behavior. For completeness in the comparison, the conventional P&O algorithm was also included. Simulation results show that SAC and DDPG algorithms achieved superior performance compared to the P&O method under PSCs, and constant and varying irradiance levels. It is shown that the SAC algorithm provides superior performance in achieving high tracking efficiency and zero power oscillations near the PV MPP and GMPP compared to the DDPG method.

EV drivers have experienced a charging inconvenience due to a limited number of charging facilities and mileage anxiety due to the limited driving distance for a single full charge. This paper developed a user-friendly online EV charging guidance algorithm to cope with the two aforementioned issues using multi-agent deep reinforcement learning. First, three models, i.e., the traffic network model, charging station model, and EV driver model, are established, respectively, considering the traffic condition, the potential competition of future charging demand at charging stations, and the drivers’ mileage anxiety. Second, the charging guidance process is modeled as a Markov decision process, and charging stations are taken as agents. The attentional multi-agent actor-critic algorithm based on the centralized training with decentralized execution framework is built. Finally, compared to the comparison algorithm, the performance does not diminish with the increase in the number of agents, indicating that the approach has the scalability to be applied to large-scale agent systems. The model still has the generalization in extreme scenarios such as traffic road and charger failures. The testing time within various numbers of charging stations is about 23ms per EV, which is sufficient to apply the proposed model to real-time decision-making and online recommendation.