1 May 2020 in IEEE Signal Processing Magazine
PKParameswaran Kamalaruban
Fueled by recent advances in deep neural networks, reinforcement learning (RL) has been in the limelight because of many recent breakthroughs in artificial intelligence, including defeating humans in games (e.g., chess, Go, StarCraft), self-driving cars, smart-home automation, and service robots, among many others. Despite these remarkable achievements, many basic tasks can still elude a single RL agent. Examples abound, from multiplayer games, multirobots, cellular-antenna tilt control, traffic-control systems, and smart power grids to network management.
8 July 2020 in IEEE/ASME Transactions on Mechatronics
Fault diagnosis for rolling elements in rotating machinery persistently receives high research interest due to the said machinery's prevalence in a broad range of applications. State-of-the-art methods in such setups focus on effective identification of faults that usually involve a single component while rejecting noise from limited sources. This paper studies the data-based diagnosis of mixed faults coming from multiple components with an emphasis on model robustness against a wide spectrum of external perturbation. A dataset is collected on a rotor and bearing system by varying the levels and types of faults in both the rotor and bearing, which results in 48 machine health conditions. A duplet classifier is developed by combining two 1-D convolutional neural networks (CNNs) that are responsible for the diagnosis of rotor and bearing faults, respectively. Experimental results show that the proposed classifier can reliably identify the onset and nature of mixed faults. In addition, one-vs-all classifiers are built using the features generated by the developed 1-D CNNs as predictors to recognize previously unlearned fault types. The effectiveness of such classifiers is demonstrated using data collected from four new fault types. Finally, the robustness and ability to reject external perturbation of the duplet classification model are analyzed using kernel density estimation (KDE).
1 January 2020 in SIAM Journal on Optimization
In this paper, we study a class of stochastic optimization problems, referred to as the conditional stochastic optimization (CSO), in the form of $\min_{x \in \mathcal{X}} \mathbb{E}_{\xi}f_\xi({\mathbb{E}_{\eta|\xi}[g_\eta(x, \xi)]})$, which finds a wide spectrum of applications including portfolio selection, reinforcement learning, robust learning, and causal inference. Assuming availability of samples from the distribution $\mathbb{P}(\xi)$ and samples from the conditional distribution $\mathbb{P}(\eta|\xi)$, we establish the sample complexity of the sample average approximation (SAA) for CSO, under a variety of structural assumptions, such as Lipschitz continuity, smoothness, and error bound conditions. We show that the total sample complexity improves from $\mathcal{O}(d/\epsilon^4)$ to $\mathcal{O}(d/\epsilon^3)$ when assuming smoothness of the outer function, and further to $\mathcal{O}(1/\epsilon^2)$ when the empirical function satisfies the quadratic growth condition. We also establish the sample complexity of a modified SAA when $\xi$ and $\eta$ are independent. Several numerical experiments further support our theoretical findings.
8 January 2015 in Computational Optimization and Applications
30 July 2024 in SIAM Journal on Optimization
Natural policy gradient (NPG) methods, equipped with function approximation and entropy regularization, achieve impressive empirical success in reinforcement learning problems with large state-action spaces. However, their convergence properties and the impact of entropy regularization remain elusive in the function approximation regime. In this paper, we establish finite-time convergence analyses of entropy-regularized NPG with linear function approximation under softmax parameterization. In particular, we prove that entropy-regularized NPG with averaging satisfies the persistence of excitation condition, and achieves a fast convergence rate of [Formula: see text] up to a function approximation error in regularized Markov decision processes. This convergence result does not require any a priori assumptions on the policies. Furthermore, under mild regularity conditions on the concentrability coefficient and basis vectors, we prove that entropy-regularized NPG exhibits linear convergence up to the compatible function approximation error. Finally, we provide sample complexity results for sample-based NPG with entropy regularization.
26 June 2023 in SIAM Journal on Control and Optimization
This paper develops a novel control-theoretic framework to analyze the nonasymptotic convergence of Q-learning. We show that the dynamics of asynchronous Q-learning with a constant step size can be naturally formulated as a discrete-time stochastic affine switching system. In particular, for a given [Formula: see text]-function parameter, [Formula: see text], the greedy policy, [Formula: see text], in the Q-learning update plays the role of the switching policy, and is the key connection between the switching system and Q-learning. Then, the evolution of the Q-learning estimation error is over- and under-estimated by trajectories of two simpler dynamical systems. Based on these two systems, we derive a new finite-time error bound of asynchronous Q-learning when a constant step size is used. In addition, the new analysis sheds light on the overestimation phenomenon of Q-learning.