Machine learning (ML) is increasingly used to support or automate decision processes in critical decision systems such as self driving cars or systems for medical diagnosis. These systems require decisions in which human lives are at stake and the decisions should therefore be well founded and very reliable. This need for reliability contrasts with the black-box nature of many ML models, making it difficult to ensure that they always behave as intended. In face of the high stakes involved, the resulting uncertainty is a significant challenge. Explainable artificial intelligence (XAI) addresses the issue by making black-box models more interpretable, often to increase user trust. However, many current XAI applications focus more on transparency and usability than on enhancing safety of ML applications. In this work, we therefore conduct a systematic literature review to examine how XAI can be leveraged to increase safety of ML applications in critical decision systems. We strive to find out for what purposes XAI is currently used in critical decision systems, what are the most common XAI techniques in critical decision systems and how XAI can be harnessed to increase safety of ML applications in critical decision systems. Using the SPAR-4-SLR protocol, we are able to answer these questions and provide a foundational resource for researchers and practitioners seeking to mitigate risks of ML applications. Essentially, we identify promising approaches of XAI which go beyond increasing trust to actively ensure correctness of decisions. Our findings propose a three-layered framework to enhance safety of ML in critical decision systems by means of XAI. The approach consists of Reliability, Validation and Verification. Furthermore, we point out gaps in research and propose future directions of XAI research for enhancing safety of ML applications in critical decision systems.

Visual inspection is a crucial quality assurance process across many manufacturing industries. While many companies now employ machine learning-based systems, they face a significant challenge, particularly in safety-critical domains. The outcomes of these systems are often complex and difficult to comprehend, making them less reliable and trustworthy. To address this challenge, we build on our previously proposed R4VR-framework and provide practical, step-by-step guidelines that enable the safe and efficient implementation of machine learning in visual inspection tasks, even when starting from scratch. The framework leverages three complementary safety mechanisms—uncertainty detection, explainability, and model diversity—to enhance both accuracy and system safety while minimizing manual effort. Using the example of steel surface inspection, we demonstrate how a self-accelerating process of data collection where model performance improves while manual effort decreases progressively can arise. Based on that, we create a system with various safety mechanisms where less than 0.1% of images are classified wrongly and remain undetected. We provide concrete recommendations and an open-source code base to facilitate reproducibility and adaptation to diverse industrial contexts.

Objective: Machine learning is increasingly integrated into safety-critical domains such as medical applications. In this context, regulatory frameworks require the assessment and minimization of risks associated with incorrect model predictions. However, classical evaluation methods often focus on quantifying error frequencies, which do not reflect the heterogeneous impact of different types of errors. To address this deficiency, we elaborate an approach for assessing machine learning–based multiclass classification models that conforms to regulatory requirements in the field of medical devices. Additionally, we aim to demonstrate its applicability in a concrete application scenario as an exemplary reference. Methods: Herein, we provide weighted balanced accuracy W B A n as a risk-based approach for assessing machine learning models for multiclass classification. W B A n is systematically based on regulatory requirements for medical devices. It incorporates core properties such as the representation of risks as a multiplicative combination of likelihood and severity and the relationship between development and real-world scenarios. We demonstrate the application of W B A n using X-ray based detection of lung diseases as a reference scenario. Results: Our analysis shows that W B A n achieves a risk-based assessment in the lung disease scenario as a reference. W B A n systematically operationalizes the definition of risk from corresponding regulatory requirements and allows the assessment to be adapted according to risk levels that have to be elaborated in the context of the risk-management process. Conclusion: Although this study uses only a single use case to demonstrate the practical applicability of our approach, it shows the impact of a risk-based approach when assessing the performance of machine learning multiclass classification models in medical applications. Without such an approach, regulatory requirements cannot be implemented in a fully comprehensive way. Consequently, the clinical impact cannot be adequately assessed when applying the model in real-world scenarios.

Visual inspection is a crucial quality assurance process across many manufacturing industries. While many companies now employ AI-based systems, they face a significant challenge, particularly in safety-critical domains. The outcomes of these systems are often complex and difficult to comprehend, making them less reliable and trustworthy. To address this challenge, we propose a human-in-the-loop framework that enables the safe and efficient implementation of machine learning in visual inspection tasks, even when starting from scratch. Our framework leverages three complementary safety mechanisms—uncertainty detection, explainability, and model diversity—to enhance both accuracy and system safety while minimizing manual effort. Using the example of steel surface inspection, we demonstrate how a self-accelerating process of data collection can arise, where model performance improves while manual effort decreases progressively. Based on that, we create a system with various safety mechanisms where every wrong prediction is identified automatically. We provide concrete recommendations and an open-source code base to facilitate reproducibility and adaptation to diverse industrial contexts.