Objectives: To assess the interpretability and acceptance of Shapley values for making artificial intelligence/machine learning (AI/ML) tools more transparent, interpretable, and useful to clinicians. Materials and Methods: Structured assessments were conducted with 30 clinicians (15 providers; 15 nurses; 8 assessments per clinician) to evaluate their ability to understand interventional Shapley Additive exPlanations (SHAP) values, a type of Shapley value that provides individualized variable importance scores and ascertain their perspective on SHAP value utility for the use of an AI/ML sepsis diagnostic. Participants were shown the diagnostic interface for real clinical scenarios with de-identified patient data with and without SHAP values. The primary outcomes were clinician ability to correctly interpret SHAP values and clinician self-reported improvement in their understanding of how the AI/ML algorithm produced its result. Results: Participants correctly interpreted SHAP values in 235 of 240 assessments (98%; CI, 95%-99%) and reported SHAP values improved their understanding of how the algorithm produced its result in every case (240/240; 100%; CI, 99%-100%). Participants were unanimous (30/30) in preferring the interface with SHAP values over the interface without. Discussion: Clinician participants strongly preferred the device interface with SHAP values, were unanimous in reporting SHAP values improved their understanding of the AI/ML diagnostic, and scored nearly perfectly when asked to interpret SHAP values. Conclusion: These results suggest health care providers value transparency into AI/ML algorithms designed for clinical use, and that Shapley values are a useful approach to providing that transparency, which in turn may improve tool adoption and clinical utility.

Background: Early and accurate risk stratification of patients suspected of serious infection is essential for improving outcomes, but existing diagnostic and predictive tools have limited accuracy. The objective was to compare the performance of an FDA-authorized AI diagnostic test, the Sepsis ImmunoScore, against widely available biomarkers and clinical tools for diagnosis of sepsis and prediction of in-hospital mortality and intensive care unit (ICU) admission. Methods: This multicenter observational study included 6027 adult patients suspected of infection across 7 U.S. hospital sites. The Sepsis ImmunoScore’s predictive performance was compared to the sequential organ failure assessment (SOFA) score, procalcitonin (PCT), C-reactive protein (CRP), Systemic Inflammatory Response Syndrome (SIRS) score, National Early Warning Score (NEWS), and quick SOFA (qSOFA). Primary outcomes included sepsis as defined by Sepsis-3 criteria, in-hospital mortality, and ICU admission. Predictive accuracy was assessed using area under the receiver operating characteristic curve (AUC), and 95% confidence intervals were generated and hypothesis testing conducted using the bootstrap method. Results: The Sepsis ImmunoScore demonstrated statistically significant superior performance across all outcomes. For sepsis prediction, the Sepsis ImmunoScore achieved an AUC of 0.82, compared to SOFA (0.72), procalcitonin (PCT) (0.70), C-reactive protein (CRP) (0.61), SIRS (0.59), NEWS (0.69), and qSOFA (0.67). For in-hospital mortality prediction, the Sepsis ImmunoScore achieved an AUC of 0.80, outperforming SOFA (0.72), PCT (0.67), CRP (0.58), SIRS (0.60), NEWS (0.72), and qSOFA (0.69). For ICU admission, the Sepsis ImmunoScore reached an AUC of 0.74, superior to SOFA (0.63), PCT (0.64), CRP (0.54), SIRS (0.60), NEWS (0.70), and qSOFA (0.65). All differences between the Sepsis ImmunoScore and comparators were statistically significant. Conclusions: The Sepsis ImmunoScore significantly improved predictive accuracy for sepsis, in-hospital mortality, and ICU admission compared to six conventional clinical scores and biomarkers. This AI-based tool may enhance risk stratification and clinical decision-making, potentially leading to more timely sepsis interventions and improved outcomes.

BACKGROUND Early identification of patients at risk for sepsis, mortality, and clinical deterioration is essential for improving outcomes, but existing diagnostic and predictive tools have limited accuracy. The objective was to evaluate the performance of an FDA-authorized AI tool, the Sepsis ImmunoScore, compared to widely available biomarkers and clinical tools for diagnosis of sepsis and prediction of in-hospital mortality and intensive care unit (ICU) admission. METHODS This multicenter observational study included 6,027 adult patients suspected of infection across 7 U.S. hospital sites. The Sepsis ImmunoScore’s predictive performance was compared to the sequential organ failure assessment (SOFA) score, procalcitonin (PCT), C-reactive protein (CRP), Systemic Inflammatory Response Syndrome (SIRS) score, National Early Warning Score (NEWS), and quick SOFA (qSOFA). Primary outcomes included sepsis as defined by Sepsis-3 criteria, in-hospital mortality, and ICU admission. Predictive accuracy was assessed using area under the receiver operating characteristic curve (AUC), and 95% confidence intervals were generated and hypothesis testing conducted using the bootstrap method. RESULTS The Sepsis ImmunoScore demonstrated statistically significant superior performance across all outcomes. For sepsis prediction, the Sepsis ImmunoScore achieved an AUC of 0.82, compared to SOFA (0.72), procalcitonin (PCT) (0.70),C-reactive protein (CRP) (0.61), SIRS (0.59), NEWS (0.69), and qSOFA (0.67). For in-hospital mortality prediction, the Sepsis ImmunoScore achieved an AUC of 0.80, outperforming SOFA (0.72), PCT (0.67), CRP (0.58), SIRS (0.60), NEWS (0.72), and qSOFA (0.69). For ICU admission, the Sepsis ImmunoScore reached an AUC of 0.74, superior to SOFA (0.63), PCT (0.64), CRP (0.54), SIRS (0.60), NEWS (0.70), and qSOFA (0.65). All differences between the Sepsis ImmunoScore and comparators were statistically significant. CONCLUSIONS The Sepsis ImmunoScore significantly improved predictive accuracy for sepsis, in-hospital mortality, and ICU admission compared to six conventional clinical scores and biomarkers. This AI-based tool may enhance risk stratification and clinical decision-making, potentially leading to more timely sepsis interventions and improved outcomes. Question How does the FDA-authorized Sepsis ImmunoScore compare to conventional sepsis tools at diagnosing and predicting sepsis, clinical deterioration, and in-hospital mortality? Findings In a multicenter observational cohort of 6,027 patients with suspected infection, the Sepsis ImmunoScore demonstrated statistically superior performance compared to PCT, CRP, SOFA, qSOFA, SIRS, and NEWS in predicting all outcomes: sepsis diagnosis, ICU admission, and in-hospital mortality. Meaning Because the Sepsis ImmunoScore outperforms existing sepsis diagnostics, it could potentially enhance risk stratification and clinical decision-making for patients with suspected infection, enabling more appropriate and timely interventions.