Reinforcement learning (RL) allows an agent to solve sequential decision-making problems by interacting with an environment in a trial-and-error fashion. When these environments are very complex, pure random exploration of possible solutions often fails, or is very sample inefficient, requiring an unreasonable amount of interaction with the environment. Hierarchical reinforcement learning (HRL) utilizes forms of temporal- and state-abstractions in order to tackle these challenges, while simultaneously paving the road for behavior reuse and increased interpretability of RL systems. In this survey paper we first introduce a selection of problem-specific approaches, which provided insight in how to utilize often handcrafted abstractions in specific task settings. We then introduce the Options framework, which provides a more generic approach, allowing abstractions to be discovered and learned semi-automatically. Afterwards we introduce the goal-conditional approach, which allows sub-behaviors to be embedded in a continuous space. In order to further advance the development of HRL agents, capable of simultaneously learning abstractions and how to use them, solely from interaction with complex high dimensional environments, we also identify a set of promising research directions.

IEEE 802.11 (Wi-Fi) is one of the technologies that provides high performance with a high density of connected devices to support emerging demanding services, such as virtual and augmented reality. However, in highly dense deployments, Wi-Fi performance is severely affected by interference. This problem is even worse in new standards, such as 802.11n/ac, where new features such as Channel Bonding (CB) are introduced to increase network capacity but at the cost of using wider spectrum channels. Finding the best channel assignment in dense deployments under dynamic environments with CB is challenging, given its combinatorial nature. Therefore, the use of analytical or system models to predict Wi-Fi performance after potential changes (e.g., dynamic channel selection with CB, and the deployment of new devices) are not suitable, due to either low accuracy or high computational cost. This paper presents a novel, data-driven approach to speed up this process, using a Graph Neural Network (GNN) model that exploits the information carried in the deployment’s topology and the intricate wireless interactions to predict Wi-Fi performance with high accuracy. The evaluation results show that preserving the graph structure in the learning process obtains a 64% increase versus a naive approach, and around 55% compared to other Machine Learning (ML) approaches when using all training features.

We assess the feasibility of a reusable neural architecture search agent aimed at amortizing the initial time-investment in building a good search strategy. We do this through the use of Reinforcement Learning, where an agent learns to iteratively select the best way to modify a given neural network architecture. This is achieved using a transformer-based agent design trained using the Ape-X algorithm. We consider both the NAS-Bench-101 and NAS-Bench-301 settings, and compare against various known strong baselines, such as local search and random search. While achieving competitive performance on both benchmarks, the amount of training required for the much larger NAS-Bench-301 is only marginally greater than NAS-Bench-101, illustrating the strong scaling properties of our agent. Our agent is able to achieve strong performance, but the choice of values for certain parameters are crucial to ensuring the succesful training of the agent. We provide some guidance for the selection of appropriate values for hyperparameters through a detailed description of our experimental setup and several ablation studies.

Wildfires are becoming increasingly devastating, and detecting them early is essential to containing them. Deep learning-based wildfire detection systems have increased in complexity dramatically in recent years, and in order to manage this added complexity, techniques have been proposed to automate the design of neural network architectures. Such techniques are usually referred to as Neural Architecture Search (NAS). This paper showcases the use of a reinforcement learning-based NAS agent to design a small neural network to perform active fire detection on multispectral satellite imagery. Specifically, we aim to automatically design a neural network that can determine if a single multispectral pixel is a part of a fire, and do so within the constraints of a Low Earth Orbit (LEO) nanosatellite with a limited power budget, to facilitate on-board processing of sensor data. A regression model that predicts the F1 score obtained by a particular architecture following quantization is used as a reward function. This model is trained on the classification performance statistics of a sample of neural network architectures. Besides the F1 score, we also include the total number of parameters in our reward function to limit the size of the designed model. Finally, we deployed the best neural network to the Google Coral Micro Dev Board and evaluated its inference latency and power consumption. This neural network consists of 1,716 parameters, takes on average 984μs to inference, and consumes around 800mW to perform inference. These results show that our approach can be applied to new problems.