Johannes Ultsch

Mr. Johannes Ultsch

orcid

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
26 October 2021 in Actuators
Verified
Journal Article
Open access
PDF file
JMJonas Mirwald
Jonas Mirwald
JUJohannes Ultsch
Johannes Ultsch
RCRicardo De Castro
Ricardo De Castro
Jonathan BrembeckJonathan Brembeck
Jonathan Brembeck
Traffic congestion and the occurrence of traffic accidents are problems that can be mitigated by applying cooperative adaptive cruise control (CACC). In this work, we used deep reinforcement learning for CACC and assessed its potential to outperform model-based methods. The trade-off between distance-error minimization and energy consumption minimization whilst still ensuring operational safety was investigated. Alongside a string stability condition, robustness against burst errors in communication also was incorporated, and the effect of preview information was assessed. The controllers were trained using the proximal policy optimization algorithm. A validation by comparison with a model-based controller was performed. The performance of the trained controllers was verified with respect to the mean energy consumption and the root mean squared distance error. In our evaluation scenarios, the learning-based controllers reduced energy consumption in comparison to the model-based controller by 17.9% on average.

12 August 2024 in Applied Sciences
Verified
Journal Article
Open access
PDF file
JUJohannes Ultsch
Johannes Ultsch
APAndreas Pfeiffer
Andreas Pfeiffer
JRJulian Ruggaber
Julian Ruggaber
TKTobias Paul Hermann Kamp
Tobias Paul Hermann Kamp
Jonathan BrembeckJonathan Brembeck
Jonathan Brembeck
JTJakub Tobolář
Jakub Tobolář
In vertical vehicle dynamics control, semi-active dampers are used to enhance ride comfort and road-holding with only minor additional energy expenses. However, a complex control problem arises from the combined effects of (1) the constrained semi-active damper characteristic, (2) the opposing control objectives of improving ride comfort and road-holding, and (3) the additionally coupled vertical dynamic system. This work presents the application of Reinforcement Learning to the vertical dynamics control problem of a real street vehicle to address these issues. We discuss the entire Reinforcement Learning-based controller design process, which started with deriving a sufficiently accurate training model representing the vehicle behavior. The obtained model was then used to train a Reinforcement Learning agent, which offered improved vehicle ride qualities. After that, we verified the trained agent in a full-vehicle simulation setup before the agent was deployed in the real vehicle. Quantitative and qualitative real-world tests highlight the increased performance of the trained agent in comparison to a benchmark controller. Tests on a real-world four-post test rig showed that the trained RL-based controller was able to outperform an offline-optimized benchmark controller on road-like excitations, improving the comfort criterion by about 2.5% and the road-holding criterion by about 2.0% on average.

8 March 2023 in Applied Sciences
Verified
Journal Article
Open access
PDF file
KAKenan Ahmic
Kenan Ahmic
JUJohannes Ultsch
Johannes Ultsch
Jonathan BrembeckJonathan Brembeck
Jonathan Brembeck
CWChristoph Winter
Christoph Winter
Reinforcement learning-based controllers for safety-critical applications, such as autonomous driving, are typically trained in simulation, where a vehicle model is provided during the learning process. However, an inaccurate parameterization of the vehicle model used for training heavily influences the performance of the reinforcement learning agent during execution. This inaccuracy is either caused by changes due to environmental influences or by falsely estimated vehicle parameters. In this work, we present our approach of combining dynamics randomization with reinforcement learning to overcome this issue for a path-following control task of an autonomous and over-actuated robotic vehicle. We train three independent agents, where each agent experiences randomization for a different vehicle dynamics parameter, i.e., the mass, the yaw inertia, and the road-tire friction. We randomize the parameters uniformly within predefined ranges to enable the agents to learn an equally robust control behavior for all possible parameter values. Finally, in a simulation study, we compare the performance of the agents trained with dynamics randomization to the performance of an agent trained with the nominal parameter values. Simulation results demonstrate that the former agents obtain a higher level of robustness against model uncertainties and varying environmental conditions than the latter agent trained with nominal vehicle parameter values.

12 November 2021 in Actuators
Verified
Journal Article
Open access
PDF file
JUJohannes Ultsch
Johannes Ultsch
JRJulian Ruggaber
Julian Ruggaber
APAndreas Pfeiffer
Andreas Pfeiffer
CSChristina Schreppel
Christina Schreppel
JTJakub Tobolář
Jakub Tobolář
DBDaniel Baumgartner
Daniel Baumgartner
High-level modeling languages facilitate system modeling and the development of control systems. This is mainly achieved by the automated handling of differential algebraic equations which describe the dynamics of the modeled systems across different physical domains. A wide selection of model libraries provides additional support to the modeling process. Nevertheless, deployment on embedded targets poses a challenge and usually requires manual modification and reimplementation of the control system. The novel proposed eFMI Standard (Functional Mock-up Interface for embedded systems) introduces a workflow and an automated toolchain to simplify the deployment of model-based control systems on embedded targets. This contribution describes the application and verification of the eFMI workflow using a vertical dynamics control problem with an automotive application as an example. The workflow is exemplified by a control system design process which is supported by the a-causal, multi-physical, high-level modeling language Modelica. In this process, the eFMI toolchain is applied to a model-based controller for semi-active dampers and demonstrated using an eFMI-based nonlinear prediction model within a nonlinear Kalman filter. The generated code was successfully tested in different validation steps on the dedicated embedded system. Additionally, tests with a low-volume production electronic control unit (ECU) in a series-produced car demonstrated the correct execution of the controller code under real-world conditions. The novelty of our approach is that it automatically derives an embedded software solution from a high-level multi-physical model with standardized eFMI methodology and tooling. We present one of the first full application scenarios (covering all aspects ranging from multi-physical modeling up to embedded target deployment) of the new eFMI tooling.

25 November 2025 in IEEE Access
Verified
Journal Article
Open access
JUJohannes Ultsch
Johannes Ultsch
KAKenan Ahmic
Kenan Ahmic
Jonathan BrembeckJonathan Brembeck
Jonathan Brembeck
Deep reinforcement learning (DRL) has become increasingly popular for solving sequential decision-making problems in control applications, particularly for multi-physical systems such as vehicle and robotic motion control. The application of DRL to control applications is especially appealing, as control engineering is strongly conducted with the utilization of simulation models. Leveraging these preexisting simulation models for DRL agent training provides a promising and effective strategy. Although DRL has gained popularity in controlling complex multi-physical systems, existing DRL frameworks still lack support for the entire training process of DRL agents utilizing preexisting simulation models. In this work, we present a structured workflow accompanied by its implementation, which utilizes the Functional Mock-up Interface (FMI) to leverage the use of preexisting simulation models for DRL training. We introduce the concept of scenarios as a set of environment parameters and formalize the concept of evaluation metrics. In addition, we extended the scenario concept to uncertain scenarios, to account for environmental uncertainties during training. Our proposed workflow is demonstrated on a path following control problem in automated driving, where a residual policy is learned to enhance the performance of a baseline controller. Using our workflow and toolchain implementation, we successfully trained an agent that achieved significant reductions in tracking errors, with maximum lateral displacement errors reduced by up to 93.4% and velocity tracking errors decreased by up to 41.3%. Our results demonstrates the effectiveness of our approach in leveraging preexisting simulation models for DRL training, paving the way for more efficient and effective control applications.

18 November 2025
Verified
Conference Paper
Closed access
KAKenan Ahmic
Kenan Ahmic
JUJohannes Ultsch
Johannes Ultsch
Jonathan BrembeckJonathan Brembeck
Jonathan Brembeck
DBDarius Burschka
Darius Burschka
A major issue with learning-based control methods is their lack of generalization in unseen parts of the state space. They tend to exhibit inefficient control performances or even complete failure in situations that were not encountered during the training process. This limitation is highly problematic in safety-critical applications, such as autonomous driving, where reliability and safety are essential. To address this issue, we propose a prediction-based safeguarding and performance monitoring framework for learning-based vehicle motion controllers. The proposed framework ensures a fail-safe execution during runtime and additionally intervenes if the predicted performance of the controller is safe but inefficient due to poor generalization capabilities. More specifically, in our framework, the behavior of the learning-based controller is predicted for a finite time horizon. If predictions are unsafe, a fallback controller is activated that ensures vehicle safety. If predictions are safe, the fallback controller's behavior is also predicted and compared to that of the learning-based controller. Afterwards, the more efficient controller is applied to the vehicle, which prevents the learning-based controller from executing safe but inefficient control behaviors. Simulation results for path following control applications demonstrate that the proposed framework is able to ensure vehicle safety at all times and that it is more efficient compared to the sole use of the fallback controller.