Advances in artificial intelligence (AI) have led to its application in many areas of everyday life. In the context of control engineering, reinforcement learning (RL) represents a particularly promising approach as it is centred around the idea of allowing an agent to freely interact with its environment to find an optimal strategy. One of the challenges professionals face when training and deploying RL agents is that the latter often have to run on dedicated embedded devices. This could be to integrate them into an existing toolchain or to satisfy certain performance criteria like real-time constraints. Conventional RL libraries, however, cannot be easily utilised in conjunction with that kind of hardware. In this paper, we present a framework named LExCI, the Learning and Experiencing Cycle Interface, which bridges this gap and provides end-users with a free and open-source tool for training agents on embedded systems using the open-source library RLlib. Its operability is demonstrated with two state-of-the-art RL-algorithms and a rapid control prototyping system.

The increasing requirements to further reduce pollutant emissions, particularly with regard to the upcoming Euro 7 (EU7) legislation, cause further technical and economic challenges for the development of internal combustion engines. All the emission reduction technologies lead to an increasing complexity not only of the hardware, but also of the control functions to be deployed in engine control units (ECUs). Virtualization has become a necessity in the development process in order to be able to handle the increasing complexity. The virtual development and calibration of ECUs using hardware-in-the-loop (HiL) systems with accurate engine models is an effective method to achieve cost and quality targets. In particular, the selection of the best-practice engine model to fulfil accuracy and time targets is essential to success. In this context, this paper presents a physically- and chemically-based stochastic reactor model (SRM) with tabulated chemistry for the prediction of engine raw emissions for real-time (RT) applications. First, an efficient approach for a time-optimal parametrization of the models in steady-state conditions is developed. The co-simulation of both engine model domains is then established via a functional mock-up interface (FMI) and deployed to a simulation platform. Finally, the proposed RT platform demonstrates its prediction and extrapolation capabilities in transient driving scenarios. A comparative evaluation with engine test dynamometer and vehicle measurement data from worldwide harmonized light vehicles test cycle (WLTC) and real driving emissions (RDE) tests depicts the accuracy of the platform in terms of fuel consumption (within 4% deviation in the WLTC cycle) as well as NOx and soot emissions (both within 20%).

Developing powertrain control functions is time-consuming and resource-intensive, often leading to sub-optimal solutions. Reinforcement Learning (RL) allows agents to perform complex control tasks with minimal human involvement, but is often confined to simulations due to testing costs and safety concerns. To effectively apply RL in embedded powertrain control, agents must be able to handle real-world scenarios, particularly through direct interaction with real actuators and control systems. Therefore, this research applies Transfer Learning (TL) and X-in-the-Loop (XiL) simulations to develop agents that can seamlessly transition and perform robustly in real-world environments. For transient exhaust gas re-circulation control of an internal combustion engine, the process begins with a computationally inexpensive Model-in-the-Loop (MiL) simulation to select a suitable algorithm, fine-tune hyperparameters, and conduct preliminary training. In the next step, pre-trained agents are transferred to an advanced Hardware-in-the-Loop (HiL) system with real hardware using TL for further training. Compared to agents trained entirely on HiL systems, transferred agents required significantly less real-world training time (up to 5.9 times shorter) while outperforming the series production Engine Control Unit (ECU). The results highlight that for real-world effectiveness, integrating actual hardware into training is essential, reward fine-tuning plays a critical role in optimizing these interactions, and the maturity of the policy significantly influences both training duration and overall performance.