XAI refers to the techniques and methods for building AI applications which assist end users to interpret output and predictions of AI models. Black box AI applications in high-stakes decision-making situations, such as medical domain have increased the demand for transparency and explainability since wrong predictions may have severe consequences. Model explainability and interpretability are vital successful deployment of AI models in healthcare practices. AI applications' underlying reasoning needs to be transparent to clinicians in order to gain their trust. This paper presents a systematic review of XAI aspects and challenges in the healthcare domain. The primary goals of this study are to review various XAI methods, their challenges, and related machine learning models in healthcare. The methods are discussed under six categories: Features-oriented methods, global methods, concept models, surrogate models, local pixel-based methods, and human-centric methods. Most importantly, the paper explores XAI role in healthcare problems to clarify its necessity in safety-critical applications. The paper intends to establish a comprehensive understanding of XAI-related applications in the healthcare field by reviewing the related experimental results. To facilitate future research for filling research gaps, the importance of XAI models from different viewpoints and their limitations are investigated.

Background: This study introduces the Adaptive Graph Fusion Network, an interpretable graph-based learning framework developed for large-scale prediction of intensive care outcomes. The proposed model dynamically constructs patient similarity networks through a density-aware kernel that adjusts neighborhood size based on local data distribution, thereby representing both frequent and rare clinical patterns. Methods: To characterize physiological evolution over time, the framework integrates a short-horizon convolutional encoder that captures acute variations in vital signs and laboratory results with a long-horizon recurrent memory unit that models gradual temporal trends. The approach was trained and internally validated on the publicly available eICU Collaborative Research Database, which includes more than 200,000 admissions from 208 hospitals across the United States. Results: The model achieved a mean area under the receiver operating characteristic curve of 0.91 across six critical outcomes, with in-hospital mortality reaching 0.96, outperforming logistic regression, temporal long short-term memory networks, and calibrated Transformer-based architectures. Feature attribution analysis using SHAP and temporal contribution mapping identified lactate trajectories, creatinine fluctuations, and vasopressor administration as dominant determinants of risk, consistent with established clinical understanding while revealing additional temporal dependencies overlooked by existing scoring systems. Conclusions: These findings demonstrate that adaptive graph construction combined with multi-horizon temporal reasoning improves predictive reliability and interpretability in heterogeneous intensive care populations, offering a transparent and reproducible foundation for future research in clinical machine learning.