During the fourth industrial revolution, agile production methods have gained increasing importance to meet the growing demand for product individualization. Conventional process monitoring systems, which predominantly rely on static, statistically based approaches, are insufficient for the requirements of flexible manufacturing environments. Although numerous research initiatives have proposed machine learning (ML) based solutions for agile process monitoring, widespread adoption in industrial practice has not yet been achieved. This is partly due to a lack of system comparability and insufficient validation under realistic production conditions.