Thermostatically Controlled Loads (TCLs) provide a source of demand flexibility, and are often considered a good source for Demand Response (DR) applications. Due to their heterogeneity, and as such a lack of dynamics models, Reinforcement Learning (RL) is often used to exploit this flexibility. Unfortunately, RL requires exploratory interaction with the TCL, resulting in a period of potential discomfort for the users. We present an approach to reduce this exploratory time by pretraining the RL-agent. Domain randomization is used to facilitate knowledge transfer. We evaluate the pre-training potential in a DR energy arbitrage scenario with an Electric Water Heater (EWH). Our experiments show that a priori knowledge about EWH dynamics can be used to initialize and improve the control policy. In our experiments, pre-training attributes to 8.8 % additional cost savings, compared to starting from scratch.
With the increasing share of renewable energy sources in the electricity grid comes the need to exploit the available flexibility at the demand side. Demand response programs seek to exploit the flexibility of consumers by motivating end-users to shift demand based on grid signals. An important source of flexibility available at residential consumers are Thermostatically Controlled Loads (TCLs). Additionally, recent advances within the reinforcement learning area have made it possible to apply this technique to a large range of problems. Driven by these promising examples, a Batch Reinforcement Learning (BRL) algorithm is applied to a TCL. An important property of the complex optimization problem studied here, is its partial observability. The main contribution of this paper is the application of BRL to a detailed building and heating system model, implemented in Modelica. A detailed TCL model allows to perform an in-depth analysis of the effects of partial observability on the performance of the chosen control strategy. Ultimately, this paper illustrates that Modelica can be used to provide a detailed environment for a BRL algorithm. At the end, the learned control policy has been compared with two other control policies. The obtained policy outperforms both and is economically feasible after a limited amount of training days.