While Artificial Intelligence (AI) promises significant enhancements for Clinical Decision Support Systems (CDSSs), the opacity of many AI models remains a major barrier to clinical adoption, primarily due to interpretability and trust challenges. Explainable AI (XAI) seeks to bridge this gap by making model reasoning understandable to clinicians, but technical XAI solutions have too often failed to address real-world clinician needs, workflow integration, and usability concerns. This study synthesizes persistent challenges in applying XAI to CDSS—including mismatched explanation methods, suboptimal interface designs, and insufficient evaluation practices—and proposes a structured, user-centered framework to guide more effective and trustworthy XAI-CDSS development. Drawing on a comprehensive literature review, we detail a three-phase framework encompassing user-centered XAI method selection, interface co-design, and iterative evaluation and refinement. We demonstrate its application through a retrospective case study analysis of a published XAI-CDSS for sepsis care. Our synthesis highlights the importance of aligning XAI with clinical workflows, supporting calibrated trust, and deploying robust evaluation methodologies that capture real-world clinician–AI interaction patterns, such as negotiation. The case analysis shows how the framework can systematically identify and address user-centric gaps, leading to better workflow integration, tailored explanations, and more usable interfaces. We conclude that achieving trustworthy and clinically useful XAI-CDSS requires a fundamentally user-centered approach; our framework offers actionable guidance for creating explainable, usable, and trusted AI systems in healthcare.

Machine learning models have the potential to transform healthcare by enabling the construction of decision support systems. However, a major challenge is the lack of transparency and accountability, as many models do not provide understandable explanations for their recommendations. Explainable Artificial Intelligence (XAI) methods aim to address this challenge by constructing and communicating explanations of how a model works and why it produces a particular output. This can help users evaluate the system and build trust in it, where appropriate. In this paper, we propose a method for explaining the relative rankings of predictions made by an XGBoost model, which involves understanding and comparing multiple predictions together. Our method uses counterfactual examples to show how changing the feature values of an entity can affect its position within the ranking defined by the model. Unlike traditional counterfactual explanations, which aim to find feature value changes that would result in a different predicted class label by meeting a fixed threshold, the proposed approach is unique in that it aims to identify changes that would bring the predictions in line with a dynamic threshold determined by other data items. We demonstrate the effectiveness of our approach in a healthcare triage problem. Our framework for counterfactual explanation provides a powerful tool for understanding the relationships between feature values and model rankings and can help promote transparency and accountability in healthcare decision-making and decision support.