Artificial intelligence and deep learning are powerful tools for extracting knowledge from large datasets, particularly in healthcare. However, their black-box nature raises interpretability concerns, especially in high-stakes applications. Existing eXplainable Artificial Intelligence methods often focus solely on visualization or rule-based explanations, limiting interpretability's depth and clarity. This work proposes a novel explainable AI method specifically designed for medical image analysis, integrating statistical, visual, and rule-based explanations to improve transparency in deep learning models. Statistical features are derived from deep features extracted using a custom Mobilenetv2 model. A two-step feature selection method - zero-based filtering with mutual importance selection - ranks and refines these features. Decision tree and RuleFit models are employed to classify data and extract human-readable rules. Additionally, a novel statistical feature map overlay visualization generates heatmap-like representations of three key statistical measures (mean, skewness, and entropy), providing both localized and quantifiable visual explanations of model decisions. The proposed method has been validated on five medical imaging datasets - COVID-19 radiography, ultrasound breast cancer, brain tumor magnetic resonance imaging, lung and colon cancer histopathological, and glaucoma images - with results confirmed by medical experts, demonstrating its effectiveness in enhancing interpretability for medical image classification tasks.