We address the problem of learning robot control by model-free reinforcement learning (RL). We adopt the probabilistic model for model-free RL of Vlassis and Toussaint (Proceedings of the international conference on machine learning, Montreal, Canada, 2009), and we propose a Monte Carlo EM algorithm (MCEM) for control learning that searches directly in the space of controller parameters using information obtained from randomly generated robot trajectories. MCEM is related to, and generalizes, the PoWER algorithm of Kober and Peters (Proceedings of the neural information processing systems, 2009). In the finite-horizon case MCEM reduces precisely to PoWER, but MCEM can also handle the discounted infinite-horizon case. An interesting result is that the infinite-horizon case can be viewed as a ‘randomized’ version of the finite-horizon case, in the sense that the length of each sampled trajectory is a random draw from an appropriately constructed geometric distribution. We provide some preliminary experiments demonstrating the effects of fixed (PoWER) vs randomized (MCEM) horizon length in two simulated and one real robot control tasks.

Thermostats are widely used in temperature regulation of indoor spaces and have a direct impact on energy use and occupant thermal comfort. Existing guidelines make recommendations for properly selecting set points to reduce energy use, but there is little or no information regarding the actual achieved thermal comfort of the occupants. While dry-bulb air temperature measured at the thermostat location is sometimes a good proxy, there is less understanding of whether thermal comfort targets are actually met. In this direction, we have defined an experimental simulation protocol involving two office buildings; the buildings have contrasting geometrical and construction characteristics, as well as different building services systems for meeting heating and cooling demands. A parametric analysis is performed for combinations of controlled variables and boundary conditions. In all cases, occupant thermal comfort is estimated using the Fanger index, as defined in ISO 7730. The results of the parametric study suggest that simple bounds on the dry-bulb air temperature are not sufficient to ensure comfort, and in many cases, more detailed considerations taking into account building characteristics, as well as the types of building heating and cooling services are required. The implication is that the calculation or estimation of detailed comfort indices, or even the use of personalised comfort models, is key towards a more human-centric approach to building design and operation.

Over the last several years, a great amount of research work has been focused on the development of model predictive control techniques for the indoor climate control of buildings, but, despite the promising results, this technology is still not adopted by the industry. One of the main reasons for this is the increased cost associated with the development and calibration (or identification) of mathematical models of special structure used for predicting future states of the building. We propose a methodology to overcome this obstacle by replacing these hand-engineered mathematical models with a thermal simulation model of the building developed using detailed thermal simulation engines such as EnergyPlus. As designing better controllers requires interacting with the simulation model, a central part of our methodology is the control improvement (or optimisation) module, facilitating two simulation-based control improvement methodologies: one based in multi-criteria decision analysis methods and the other based on state-space identification of dynamical systems using Gaussian process models and reinforcement learning. We evaluate the proposed methodology in a set of simulation-based experiments using the thermal simulation model of a real building located in Portugal. Our results indicate that the proposed methodology could be a viable alternative to model predictive control-based supervisory control in buildings.

The realization of smart and energetically efficient buildings is contingent upon the successful implementation of two tasks that occur on distinct phases of the building life cycle: in the design and subsequent retrofitting phases, the selection and implementation of an effective energy concept, and, during the operation phase, the actuation of energy systems to ensure parsimonious energy use while retaining acceptable end-user thermal comfort. Operational efficiencies are achieved through the use of Building Energy Management Systems tasked to deliver core Sense, Think, Act (STA) functionalities: Sense, using sensing modalities installed in the building; Think, utilizing, typically a rule-based decision system; and Act, by sending actuation commands to controllable building elements. Providing the intelligence in this STA process can be a formidable task due to the complex interplay of many systems and occurrence of disturbances. In this article, an architectural and algorithmic framework is presented to provide streamlined implementation of this process. Important ingredients in this framework are: (S) a data access component capable of collecting and aggregating information from a number of heterogeneous sources (sensors, weather stations, weather forecasts); (T) a model-based optimization methodology to generate intelligent operational decisions; and (A) an assessment and actuation component. An illustrative application of the proposed methodology in an office building is provided

Deploying deep neural networks (DNNs) on microcontrollers (TinyML) is a common trend to process the increasing amount of sensor data generated at the edge, but in practice, resource and latency constraints make it difficult to find optimal DNN candidates. Neural architecture search (NAS) is an excellent approach to automate this search and can easily be combined with DNN compression techniques commonly used in TinyML. However, many NAS techniques are not only computationally expensive, especially hyperparameter optimization (HPO), but also often focus on optimizing only a single objective, e.g., maximizing accuracy, without considering additional objectives such as memory requirements or computational complexity of a DNN, which are key to making deployment at the edge feasible. In this article, we propose a novel NAS strategy for TinyML based on multi-objective Bayesian optimization (MOBOpt) and an ensemble of competing parametric policies trained using augmented random search (ARS) reinforcement learning (RL) agents. Our methodology aims at efficiently finding tradeoffs between a DNN’s predictive accuracy, memory requirements on a given target system, and computational complexity. Our experiments show that we consistently outperform existing MOBOpt approaches on different datasets and architectures such as ResNet-18 and MobileNetv3.