8 November 2023 in IEEE Transactions on Robotics
Estimating and reacting to disturbances is crucial for robust flight control of quadrotors. Existing estimators typically require significant tuning for a specific flight scenario or training with extensive ground-truth disturbance data to achieve satisfactory performance. In this article, we propose a neural moving horizon estimator (NeuroMHE) that can automatically tune its key parameters modeled by a neural network and adapt to different flight scenarios. We achieve this by deriving the analytical gradients of the MHE estimates with respect to the MHE weighting matrices, which enables a seamless embedding of the MHE as a learnable layer into the neural network for highly effective learning. Interestingly, we show that the gradients can be computed efficiently using a Kalman filter in a recursive form. Moreover, we develop a model-based policy gradient algorithm to train NeuroMHE directly from the quadrotor trajectory tracking error without needing the ground-truth disturbance data. The effectiveness of NeuroMHE is verified extensively via both numerical and physical experiments on quadrotors in various challenging flights. Notably, NeuroMHE outperforms a state-of-the-art neural network-based estimator, reducing force estimation errors by up to 76.7%, while using a portable neural network that has only 7.7% of the learnable parameters of the latter. The proposed method is general and can be applied to robust adaptive control of other robotic systems.
1 February 2024 in IEEE Transactions on Automatic Control
The convergence of policy gradient algorithms hinges on the optimization landscape of the underlying optimal control problem. Theoretical insights into these algorithms can often be acquired from analyzing those of linear quadratic control. However, most of the existing literature only considers the optimization landscape for static full-state or output feedback policies (controllers). We investigate the more challenging case of dynamic output-feedback policies for linear quadratic regulation (abbreviated as dLQR ), which is prevalent in practice but has a rather complicated optimization landscape. We first show how the dLQR cost varies with the coordinate transformation of the dynamic controller and then derive the optimal transformation for a given observable stabilizing controller. One of our core results is the uniqueness of the stationary point of dLQR when it is observable, which provides an optimality certificate for solving dynamic controllers using policy gradient methods. Moreover, we establish conditions under which dLQR and linear quadratic Gaussian control are equivalent, thus providing a unified viewpoint of optimal control of both deterministic and stochastic linear systems. These results further shed light on designing policy gradient algorithms for more general decision-making problems with partially observed information.
4 January 2022 in IEEE Transactions on Cybernetics
For the existing adaptive constrained robotic control algorithms, the demanding “feasibility conditions” on virtual controller is normally inevitable and the extra limits on constraining functions have to be imposed, making the corresponding approaches more demanding and less user friendly in control development. Here, we develop a new neuroadaptive constrained control strategy for uncertain robotic manipulators in the presence of position and velocity constraints. First, a novel unified mapping function (UMF) is constructed so that the restriction on constraining boundaries is removed and more kinds of constraining forms can be handled. Second, by integrating the UMF-based coordinate transformation with the “universal” approximation characteristic of neural networks over some compact set, the developed neuroadaptive control completely obviates the complicated yet undesired “feasibility conditions.” Furthermore, it is proven that all closed-loop signals are semiglobally bounded and the constraints are not violated. The effectiveness of the proposed control is validated via a two-link rigid robotic manipulator.
26 October 2023 in IEEE Transactions on Cybernetics
In recent times, significant advancements have been made in delving into the optimization landscape of policy gradient methods for achieving optimal control in linear time-invariant (LTI) systems. Compared with state-feedback control, output-feedback control is more prevalent since the underlying state of the system may not be fully observed in many practical settings. This article analyzes the optimization landscape inherent to policy gradient methods when applied to static output feedback (SOF) control in discrete-time LTI systems subject to quadratic cost. We begin by establishing crucial properties of the SOF cost, encompassing coercivity, $L$ -smoothness, and $M$ -Lipschitz continuous Hessian. Despite the absence of convexity, we leverage these properties to derive novel findings regarding convergence (and nearly dimension-free rate) to stationary points for three policy gradient methods, including the vanilla policy gradient method, the natural policy gradient method, and the Gauss–Newton method. Moreover, we provide proof that the vanilla policy gradient method exhibits linear convergence toward local minima when initialized near such minima. This article concludes by presenting numerical examples that validate our theoretical findings. These results not only characterize the performance of gradient descent for optimizing the SOF problem but also provide insights into the effectiveness of general policy gradient methods within the realm of reinforcement learning.
9 February 2018 in IEEE Transactions on Automatic Control
Aggregate load modeling is of fundamental importance for systematic analysis and design of various demand response strategies. Instead of keeping track of the trajectories of individual loads, the aggregate modeling problem focuses on characterizing the density evolution of the load population. Most existing models are only applicable to Thermostatically Controlled Loads (TCL) with first-order linear dynamics. This paper develops a unified aggregate modeling approach that can be used for general TCLs as well as deferrable loads. We propose a deterministic hybrid system model to describe individual load dynamics under demand response rules, and develop a general stochastic hybrid system (SHS) model to capture the population dynamics. We also derive a set of partial differential equations (PDE) that governs the probability density evolution of the SHS. Our results cannot be obtained using the exiting SHS tools in the literature as the proposed SHS model involves both random and deterministic switchings with general switching surfaces in multi-dimensional domains. The derived PDE model includes many existing aggregate load modeling results as special cases and can be
12 December 2024 in Aerospace Science and Technology