Vrushabh S. Donge

Vrushabh S. Donge

Auto-generated profile, based on publicly available information

Vrushabh S. Donge

Department of Electrical Engineering, University of Texas at Arlington, Arlington, TX, USA

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
4 July 2022 in IEEE Transactions on Neural Networks and Learning Systems
Verified
Journal Article
Closed access
BLBosen Lian
Bosen Lian
VDVrushabh S. Donge
Vrushabh S. Donge
FLFrank L. Lewis
Frank L. Lewis
TCTianyou Chai
Tianyou Chai
ADAli Davoudi
Ali Davoudi
This article proposes a data-driven inverse reinforcement learning (RL) control algorithm for nonzero-sum multiplayer games in linear continuous-time differential dynamical systems. The inverse RL problem in the games is solved by a learner reconstructing the unknown expert players’ cost functions from demonstrated expert’s optimal state and control input trajectories. The learner, thus, obtains the same control feedback gains and trajectories as the expert, only using data along system trajectories without knowing system dynamics. This article first proposes a model-based inverse RL policy iteration framework that has: 1) policy evaluation step for reconstructing cost matrices using Lyapunov functions; 2) state-reward weight improvement step using inverse optimal control (IOC); and 3) policy improvement step using optimal control. Based on the model-based policy iteration algorithm, this article further develops an online data-driven off-policy inverse RL algorithm without knowing any knowledge of system dynamics or expert control gains. Rigorous convergence and stability analysis of the algorithms are provided. It shows that the off-policy inverse RL algorithm guarantees unbiased solutions while probing noises are added to satisfy the persistence of excitation (PE) condition. Finally, two different simulation examples validate the effectiveness of the proposed algorithms.

29 September 2022 in IEEE Transactions on Control of Network Systems
Verified
Journal Article
Closed access
VDVrushabh S. Donge
Vrushabh S. Donge
BLBosen Lian
Bosen Lian
FLFrank L. Lewis
Frank L. Lewis
ADAli Davoudi
Ali Davoudi
This work investigates inverse reinforcement learning (RL) for multiagent systems (MAS) defined by Graphical Apprentice Games. These games are solved by a learner MAS finding the unknown cost functions of an expert MAS from its demonstrated behavior. We begin by developing a model-based inverse RL algorithm including two update loops:1) an inner-loop optimal control update and 2) an outer-loop inverse optimal control (IOC) update. We then introduce a model-free inverse RL algorithm that uses online behaviors of the expert and learner MAS without knowing their dynamics. The optimal control and IOC are solved as subproblems in both proposed inverse RL algorithms. The reward functions that the learner MAS finds are proven to be both stabilizing and nonunique. Simulated case studies validate the effectiveness of the proposed inverse RL algorithms.

31 October 2022 in IEEE Transactions on Neural Networks and Learning Systems
Verified
Journal Article
Closed access
BLBosen Lian
Bosen Lian
VDVrushabh S. Donge
Vrushabh S. Donge
WXWenqian Xue
Wenqian Xue
FLFrank L. Lewis
Frank L. Lewis
ADAli Davoudi
Ali Davoudi
This article studies a distributed minmax strategy for multiplayer games and develops reinforcement learning (RL) algorithms to solve it. The proposed minmax strategy is distributed, in the sense that it finds each player’s optimal control policy without knowing all the other players’ policies. Each player obtains its distributed control policy by solving a distributed algebraic Riccati equation in a multiplayer noncooperative game. This policy is found against the worst policies of all the other players. We guarantee the existence of distributed minmax solutions and study their $\mathcal{L}_2$ and asymptotic stabilities. Under mild conditions, the resulting minmax control policies are shown to improve robust gain and phase margins of multiplayer systems compared to the standard linear–quadratic regulator controller. Distributed minmax solutions are found using both model-based policy iteration and data-driven off-policy RL algorithms. Simulation examples verify the proposed formulation and its computational efficiency over the nondistributed Nash solutions.

31 October 2023 in IEEE Transactions on Cybernetics
Verified
Journal Article
Closed access
VDVrushabh S. Donge
Vrushabh S. Donge
BLBosen Lian
Bosen Lian
FLFrank L. Lewis
Frank L. Lewis
ADAli Davoudi
Ali Davoudi
This article proposes a data-efficient model-free reinforcement learning (RL) algorithm using Koopman operators for complex nonlinear systems. A high-dimensional data-driven optimal control of the nonlinear system is developed by lifting it into the linear system model. We use a data-driven model-based RL framework to derive an off-policy Bellman equation. Building upon this equation, we deduce the data-efficient RL algorithm, which does not need a Koopman-built linear system model. This algorithm preserves dynamic information while reducing the required data for optimal control learning. Numerical and theoretical analyses of the Koopman eigenfunctions for dataset truncation are discussed in the proposed model-free data-efficient RL algorithm. We validate our framework on the excitation control of the power system.

14 May 2024 in IEEE Transactions on Control of Network Systems
Verified
Journal Article
Closed access
VDVrushabh S. Donge
Vrushabh S. Donge
BLBosen Lian
Bosen Lian
FLFrank L. Lewis
Frank L. Lewis
ADAli Davoudi
Ali Davoudi
We address the reward-shaping problem of large-scale multiagent systems (MASs) using inverse reinforcement learning (RL). The learning MAS does not have prior knowledge of the cost function of the target MAS, and aims to reconstruct it based on the target's demonstrations. We propose a scalable model-free inverse RL (IRL) algorithm for large-scale MAS, where dynamic mode decomposition (DMD) extracts dynamic modes and builds a projection matrix. This significantly reduces the data required while retaining the system's essential dynamic information. The proofs of the algorithm's convergence, stability, and non-uniqueness of the state-reward weight are presented. The efficacy of our method is validated with a large-scale consensus network, by comparing the required data sizes and computational time for reward-shaping with and without DMD.

20 March 2023 in IEEE Transactions on Control of Network Systems
Verified
Journal Article
Closed access
VDVrushabh S. Donge
Vrushabh S. Donge
BLBosen Lian
Bosen Lian
FLFrank L. Lewis
Frank L. Lewis
ADAli Davoudi
Ali Davoudi
This work applies the decomposition principle to discrete-time reinforcement learning to solve the optimal control problems for a network of subsystems. The control design is defined as a linear quadratic regulator graphical problem, where the performance function couples subsystems' dynamics. We first present a model-free discrete-time reinforcement learning algorithm based on online behaviors without using system dynamics. This could become a prohibitively-long learning process for larger networks. To remedy this issue, we develop an efficient model-free reinforcement learning algorithm based on dynamic mode decomposition. This decomposition method reduces the size of the measured data while the dynamic information of the original network is still retained. This algorithm is then implemented online. The proposed methodology is validated using examples of a consensus network and a power system network.