16 March 2020 in IEEE Transactions on Automatic Control
We study the problem of estimating a random process from the observations collected by a network of sensors that operate under resource constraints. When the dynamics of the process and sensor observations are described by a state-space model and the resource are unlimited, the conventional Kalman filter provides the minimum mean square error (MMSE) estimates. However, at any given time, restrictions on the available communications bandwidth and computational capabilities and/or power impose a limitation on the number of network nodes, whose observations can be used to compute the estimates. We formulate the problem of selecting the most informative subset of the sensors as a combinatorial problem of maximizing a monotone set function under a uniform matroid constraint. For the MMSE estimation criterion, we show that the maximum elementwise curvature of the objective function satisfies a certain upper-bound constraint and is, therefore, weak submodular. Building upon the work of Mirzasoleiman et al. on submodular maximization, we develop an efficient randomized greedy algorithm for sensor selection and establish guarantees on the estimator's performance in this setting. Extensive simulation results demonstrate the efficacy of the randomized greedy algorithm compared to state-of-the-art greedy and semidefinite programming relaxation methods.
26 June 2023 in Proceedings of the AAAI Conference on Artificial Intelligence
Safe exploration is a common problem in reinforcement learning (RL) that aims to prevent agents from making disastrous decisions while exploring their environment. A family of approaches to this problem assume domain knowledge in the form of a (partial) model of this environment to decide upon the safety of an action. A so-called shield forces the RL agent to select only safe actions. However, for adoption in various applications, one must look beyond enforcing safety and also ensure the applicability of RL with good performance. We extend the applicability of shields via tight integration with state-of-the-art deep RL, and provide an extensive, empirical study in challenging, sparse-reward environments under partial observability. We show that a carefully integrated shield ensures safety and can improve the convergence rate and final performance of RL agents. We furthermore show that a shield can be used to bootstrap state-of-the-art RL agents: they remain safe after initial learning in a shielded setting, allowing us to disable a potentially too conservative shield eventually.
25 September 2017 in Formal Methods in System Design
Shield synthesis is an approach to enforce safety properties at runtime. A shield monitors the system and corrects any erroneous output values instantaneously. The shield deviates from the given outputs as little as it can and recovers to hand back control to the system as soon as possible. In the first part of this paper, we consider shield synthesis for reactive hardware systems. First, we define a general framework for solving the shield synthesis problem. Second, we discuss two concrete shield synthesis methods that automatically construct shields from a set of safety properties: (1) k-stabilizing shields, which guarantee recovery in a finite time. (2) Admissible shields, which attempt to work with the system to recover as soon as possible. Next, we discuss an extension of k-stabilizing and admissible shields, where erroneous output values of the reactive system are corrected while liveness properties of the system are preserved. Finally, we give experimental results for both synthesis methods. In the second part of the paper, we consider shielding a human operator instead of shielding a reactive system: the outputs to be corrected are not initiated by a system but by a human operator who works with an autonomous system. The challenge here lies in giving simple and intuitive explanations to the human for any interferences of the shield. We present results involving mission planning for unmanned aerial vehicles.
16 February 2021 in IEEE Transactions on Control of Network Systems
Urban air mobility (UAM) refers to air transportation services within an urban area, often in an on-demand fashion. We study air traffic management (ATM) for vehicles in a UAM fleet, while guaranteeing system safety requirements such as traffic separation. Existing ATM methods for unmanned aerial systems, such as UAS traffic management, utilize alternative approaches which do not provide strict safety guarantees. No established infrastructure exists for providing ATM at scale for UAM. We provide a decentralized, hierarchical approach for UAM ATM that allows for scalability to high traffic densities as well as providing theoretical guarantees of correctness with respect to user-provided safety specifications. Our main contributions are two-fold. First, we propose a novel UAM ATM architecture that divides the control authority between vertihubs that are each in charge of all UAM vehicles in their local airspace. Each vertihub also contains a number of vertiports that are in charge of UAM vehicle takeoffs and landings. The resulting architecture is decentralized and hierarchical, which not only enables scalability, but also robustness in the event of any individual vertihub or vertiport no longer being operational. Second, we provide a contract-based correct-by-construction reactive synthesis approach that provably guarantees safety properties with respect to user-provided specifications in linear temporal logic. We demonstrate the approach on large-volume UAM air traffic data.
12 June 2019 in IEEE Transactions on Automatic Control
We study the problem of synthesizing a policy that maximizes the entropy of a Markov decision process (MDP) subject to a temporal logic constraint. Such a policy minimizes the predictability of the paths it generates, or dually, maximizes the exploration of different paths in an MDP while ensuring the satisfaction of a temporal logic specification. We first show that the maximum entropy of an MDP can be finite, infinite or unbounded. We provide necessary and sufficient conditions under which the maximum entropy of an MDP is finite, infinite or unbounded. We then present an algorithm which is based on a convex optimization problem to synthesize a policy that maximizes the entropy of an MDP. We also show that maximizing the entropy of an MDP is equivalent to maximizing the entropy of the paths that reach a certain set of states in the MDP. Finally, we extend the algorithm to an MDP subject to a temporal logic specification. In numerical examples, we demonstrate the proposed method on different motion planning scenarios and illustrate the relation between the restrictions imposed on the paths by a specification, the maximum entropy, and the predictability of paths.