Motivated by recent developments in batch Reinforcement Learning (RL), this paper contributes to the application of batch RL in energy management in microgrids. We tackle the challenge of finding a closed-loop control policy to optimally schedule the operation of a storage device, in order to maximize self-consumption of local photovoltaic production in a microgrid. In this work, the fitted Q-iteration algorithm, a standard batch RL technique, is used by an RL agent to construct a control policy. The proposed method is data-driven and uses a state-action value function to find an optimal scheduling plan for a battery. The battery’s charge and discharge efficiencies, and the nonlinearity in the microgrid due to the inverter’s efficiency are taken into account. The proposed approach has been tested by simulation in a residential setting using data from Belgian residential consumers. The developed framework is benchmarked with a model-based technique, and the simulation results show a performance gap of 19%. The simulation results provide insight for developing optimal policies in more realistically-scaled and interconnected microgrids and for including uncertainties in generation and consumption for which white-box models become inaccurate and/or infeasible.
The smart grid paradigm and the development of smart meters have led to the availability of large volumes of data. This data is expected to assist in power system planning/operation and the transition from passive to active electricity users. With recent advances in machine learning, this data can be used to learn system dynamics. This study explores two model‐free reinforcement learning (RL) techniques – policy iteration (PI) and fitted Q‐iteration (FQI) for scheduling the operation of flexibility providers – battery and heat pump in a residential microgrid. The proposed algorithms are data‐driven and can be easily generalised to fit the control of any flexibility provider without requiring expert knowledge to build a detailed model of the flexibility provider and/or microgrid. The algorithms are tested in multi‐agent collaborative and single‐agent stochastic microgrid settings – with the uncertainty due to lack of knowledge on future electricity consumption patterns and photovoltaic production. Simulation results show that PI outperforms FQI with a 7.2% increase in photovoltaic self‐consumption in the multi‐agent setting and a 3.7% increase in the single‐agent setting. Both RL algorithms perform better than a rule‐based controller, and compete with a model‐based optimal controller, and are thus, a valuable alternative to model‐ and rule‐based controllers.