This paper addresses the challenge of minimizing training time for the control of Heating, Ventilation, and Air-conditioning (HVAC) systems with online Reinforcement Learning (RL). This is done by developing a novel approach to Multi-Agent Reinforcement Learning (MARL) to HVAC systems. In this paper, the environment formed by the HVAC system is formulated as a Markov Game (MG) in a general sum setting. The MARL algorithm is designed in a decentralized structure, where only relevant states are shared between agents, and actions are shared in a sequence, which are sensible from a system’s point of view. The simulation environment is a domestic house located in Denmark and designed to resemble an average house. The heat source in the house is an air-to-water heat pump, and the HVAC system is an Underfloor Heating system (UFH). The house is subjected to weather changes from a data set collected in Copenhagen in 2006, spanning the entire year except for June, July, and August, where heat is not required. It is shown that: (1) When comparing Single Agent Reinforcement Learning (SARL) and MARL, training time can be reduced by 70% for a four temperature-zone UFH system, (2) the agent can learn and generalize over seasons, (3) the cost of heating can be reduced by 19% or the equivalent to 750 kWh of electric energy per year for an average Danish domestic house compared to a traditional control method, and (4) oscillations in the room temperature can be reduced by 40% when comparing the RL control methods with a traditional control method.

The smart water infrastructures laboratory is a research facility at Aalborg University, Denmark. The laboratory enables experimental research in control and management of water infrastructures in a realistic environment. The laboratory is designed as a modular system that can be configured to adapt the test-bed to the desired network. The water infrastructures recreated in this laboratory are district heating, drinking water supply, and waste water collection systems. This paper focuses on the first two types of infrastructure. In the scaled-down network the researchers can reproduce different scenarios that affect its management and validate new control strategies. This paper presents four study-cases where the laboratory is configured to represent specific water distribution and waste collection networks allowing the researcher to validate new management solutions in a safe environment. Thus, without the risk of affecting the consumers in a real network. The outcome of this research facilitates the sustainable deployment of new technology in real infrastructures.

Real-time control (RTC) of urban drainage networks (UDNs) is a complex task where transport flows are non-pressurized and therefore impose flow-dependent time delays in the system. Unfortunately, the installation of flow sensors is economically out of reach at most utilities, although knowing volumes and flows are essential to optimize system operation. In this article, we formulate joint parameter and state estimation based on level sensors deployed inside manholes and basins in the network. We describe the flow dynamics on the main pipelines by the level variations inside manholes, characterized by a system of coupled partial differential equations (PDEs). These dynamics are approximated with kinematic waves where the network model is established with the water levels being the system states. Moving horizon estimation (MHE) is developed where the states and parameters are obtained via the levels and estimated flow data, utilizing the topological layout of the network. The obtained model complexity is kept within practically achievable limits, suitable for nonlinear predictive control. The effectiveness of the control and estimation method is demonstrated on a high-fidelity model of a drainage network, acting as virtual reality. We use real rain and wastewater flow data and test the controller against the uncertainty in the disturbance forecasts.

The main energy consumers in sewer networks are the sewage pumps. Therefore, to minimize the energy consumption, it is essential that these pumps operate under satisfactory conditions. Knowledge about the efficiency of the pumps and their operating conditions can help the pump station management to operate the system optimal. In the search for innovative solutions that can help the sewer management with this information, we propose a method that provides information on the pump flows, the inflow to a sewage pit, and an online estimate of the efficiency of the pump. All these information are obtained without a flow sensor. We argued that the calculated flow values can be used by the sewer management to optimize the operation on the sewer pumps, and the efficiency estimate can be used for optimal scheduling of maintain procedures. The flow and the efficiency estimations are exemplified on a pumping station of the sewer network in Herning, Denmark.