In this paper, we study safe building HVAC control via batch reinforcement learning. Random exploration in building HVAC control is infeasible due to safety considerations. However, diverse states are necessary for RL algorithms to learn useful policies. To enable \emph{safety} during exploration, we propose guided exploration by adding a Gaussian noise to a hand-crafted rule-based controller. Adjusting the variance of the noise provides a tradeoff between the \emph{diversity} of the dataset and the \emph{safety}. We apply Conservative Q Learning (CQL) to learn a policy. CQL ensures that the trained policy stays within the policy distribution used to collect the dataset, thereby guarantees safety at deployment. To select the optimal policy during the offline training, we apply model-based performance evaluation. We use the widely adopted CityLearn testbed to evaluate the performance of our proposed method. Compared with a rule-based controller, our approach obtains $12\%\sim35\%$ reduction in ramping, $3\%\sim10\%$ reduction in 1-load factor, $3\%\sim8\%$ reduction in daily peak at deployment with less than $10\%$ performance degradation during the exploration. On the contrary, the performance degradation of the state-of-the-art online reinforcement learning algorithm during exploration is around $8\%\sim 18\%$ . It also fails to surpass the performance of the rule-based controller at deployment.

Internet-of-Things (IoT) enabled monitoring and control capabilities are enabling increasing numbers of household users with controllable loads to actively participate in smart grid energy management. Realizing an efficient real-time energy management system that takes advantage of these developments requires novel techniques for managing the increased complexity of the control action space in resolving multiple challenges such as the uncertainty in energy prices and renewable energy output along with the need to satisy physical grid constraints such as transformer capacity. Addressing these challenges, we develop a multi-household energy management framework for residential units connected to the same transformer and containing DERs such as PV, ESS and controllable loads. The goal of our framework is to schedule controllable household appliances and ESS such that the cost of procuring electricity from the utility over a horizon is minimized while physical grid constraints are satisfied at each scheduling step. Traditional energy management frameworks either perform global optimization to satisfy grid constraints but suffer from high computational complexity (for example Integer Program, Mixed Integer Programming frameworks and centralized reinforcement learning) or perform decentralized real-time energy management without satisfying global grid constraints (for example multi-agent reinforcement learning with no cooperation). In contrast, we propose a cooperative multi-agent reinforcement learning (MARL) framework that i) operates in real-time, and ii) performs explicit collaboration to satisfy global grid constraints. The novelty in our framework is two fold. Firstly, our framework trains multiple independent learners (IL) for each household in parallel using historical data and performs real-time inferencing of control actions using the most recent system state. Secondly, our framework contains a low complexity knapsack based cooperation agent which combines the outputs of ILs to minimize cost while satisfying grid constraints. Simulation results show that our cooperative MARL approach achieves significant cost improvement over centralized reinforcement learning and day-ahead planning baselines. Moreover, our approach strictly satisfies physical constraints with no apriori knowledge of system dynamics while the baseline approaches have occasional violations. We also measure the training and inference time by ranging the number of households from 1 to 25. Results show that our cooperative MARL approach scales best among various approaches.

Reinforcement Learning (RL) is a major branch of AI that enables agents to learn optimal decision making via interaction with the environment. Proximal Policy Optimization (PPO) is the state-of-the-art policy optimization based RL algorithm which achieves superior overall performance on various benchmarks. A PPO agent iteratively optimizes its policy - a function which chooses optimal actions approximated by a DNN, with each iteration consisting of two computationally intensive phases: Sample Generation - where agents inference on its policy and interact with the environment to collect data, and Model Update - where the policy is trained using the collected data. In this paper, we develop the first high-throughput PPO accelerator on CPU-FPGA heterogeneous platform. Our unified systolic-array based design accelerates both the inference and the training of the deep neural network used in a RL algorithm, and is generalizable to various MLP and CNN models across a wide range of RL applications. We develop novel optimizations to simultaneously reduce data access and computation latencies, specifically: (a) optimal data flow mapping to systolic array, (b) novel memory-blocked data layout to enable streaming stall-free data access in both forward and backward propagations, and, (c) a systolic array compute sharing technique to mitigate load imbalance in the training of two networks. We evaluate our design on widely used robotics and gaming benchmarks, achieving 1.4×–26× and 1.3×–2.7× improvements in throughput, respectively, when compared with state-of-the-art CPU/CPU-GPU implementations.

Energy storage systems (ESS) are effective solutions to reduce the cost of smart grid operations due to their ability to store and supply electricity on demand. Traditionally, ESS have been used to implement a plethora of cost reduction techniques for smart grid operations such as temporal supply-demand shifting, frequency regulation, voltage regulation etc. However, the stationary nature of ESS limits their flexibility, potentially leading to low utilization and risk being a stranded asset. In this work, we explore the use of Mobile ESS in implementing such cost reduction techniques in a grid consisting of multiple microgrids. We develop an algorithmic framework to assign multiple Mobile ESS to various microgrids of the smart grid across several days. Our framework attempts to maximize the cost reduction achieved due to Mobile ESS assignment minus the routing cost. We show that our algorithm is a polynomial time $1/e-$1/e-approximation for the NP-Hard problem of optimal assignment of MESS.

Demand Response (DR) is a widely used technique to minimize the peak to average consumption ratio during high demand periods. We consider the DR problem of achieving a given curtailment target for a set of consumers equipped with a set of discrete curtailment strategies over a given duration. An effective DR scheduling algorithm should minimize the curtailment error - the difference between the targeted and achieved curtailment values - to minimize costs to the utility provider and maintain system reliability. The availability of smart meters with fine-grained customer control capability can be leveraged to offer customers a dynamic range of curtailment strategies that are feasible for small durations within the overall DR event. Both the availability and achievable curtailment values of these strategies can vary dynamically through the DR event and thus the problem of achieving a target curtailment over the entire DR interval can be modeled as a dynamic strategy selection problem over multiple discrete sub-intervals. We argue that DR curtailment error minimizing algorithms should not be oblivious to customer curtailment behavior during sub-intervals as (expensive) demand peaks can be concentrated in a few sub-intervals while consumption is heavily curtailed during others in order to achieve the given target, which makes such solutions expensive for the utility. Thus in this paper, we formally develop the notion of Sustainable DR (SDR) as a solution that attempts to distribute the curtailment evenly across sub-intervals in the DR event. We formulate the SDR problem as an Integer Linear Program and provide a very fast $\sqrt{2}$-factor approximation algorithm. We then propose a Polynomial Time Approximation Scheme (PTAS) for approximating the SDR curtailment error to within an arbitrarily small factor of the optimal. We then develop a novel ILP formulation that solves the SDR problem while explicitly accounting for customer strategy switching overhead as a constraint. We perform experiments using real data acquired from the University of Southern California’s smart grid and show that our sustainable DR model achieves results with a very low absolute error of 0.001-0.05 kWh range.