Reinforcement learning (RL) agents have proven to be highly effective in many control domains; however, despite their generally promising performance, they may still encounter difficulties in certain challenging scenarios. To address this limitation, we propose a two-stage learning framework, where an initial pre-trained policy is extended with a second set of networks, aiming to correct the base agent's knowledge and preserve its behavior. Both modules are trained with the conventional RL loop, while the second learning system is also faced with an increased number of training samples from those problematic scenarios to provide more exploration possibilities across them. We evaluated the effectiveness of the proposed framework on various environments from the robotic and vehicle control domains. The results showed that the final agent demonstrates improved performance and can successfully complete a broader range of scenarios compared to the baseline agents, while also maintaining the control quality.
Although today's reinforcement learning-based control agents operate effectively in ideal situations, their performance can significantly decrease under changing conditions. This paper addresses this limitation by proposing an experience-based online adaptation framework. It is designed to enable agents to adjust their policies to differing domains and scenarios by leveraging the information embedded in past state-action transitions. The method extracts these extra features via a history-based adaptor module and introduces them as supplementary input for the agents. It also contains a state encoder network, improving the representation capabilities and supporting the adaptation process. Furthermore, the introduced modules can replace hard-to-obtain information during deployment (e.g., sensor or domain-specific data), which facilitates real-world applications. To emphasize the contribution of our work, we evaluate the solution within highly dynamic environments where the changing physical parameters necessitate diverse control strategies. Our approach significantly improves conventional algorithms and results in comparable performance with agents receiving privileged information.
During the development of vehicle control algorithms, effective real-world validation is crucial. Model vehicle platforms provide a cost-effective and accessible method for such testing. The open-source F1Tenth project is a popular choice, but its reliance on lidar sensors limits certain applications. To enable more universal environmental perception, integrating a stereo camera system could be advantageous, although existing software packages do not yet support this functionality. Therefore, our research focuses on developing a modular software architecture for the F1Tenth platform, incorporating real-time stereo vision-based environment perception, robust state representation, and clear actuator interfaces. The system simplifies the integration and testing of control algorithms, while minimizing the simulation-to-reality gap. The framework’s operation is demonstrated through a real-world control problem. Environmental sensing, representation, and the control method combine classical and deep learning techniques to ensure real-time performance and robust operation. Our platform facilitates real-world testing and is suitable for validating research projects.