Interpretability is a critical requirement for machine learning models in healthcare, as clinicians need to understand how the input features are processed to make informed decisions about patient care. Traditional preprocessing methods often fail to capture subtle differences between patient groups, particularly in datasets with overlapping or highly similar classes. To address these challenges, we propose Pairwise Difference Filter (PDF), a novel pre-processing method that leverages pairwise differences between samples of opposite classes to identify the most influential features. PDF focuses on pairs of patients with the smallest overall differences but significant differences in specific features, enabling the identification of clinically meaningful biomarkers. By enhancing the interpretability of machine learning models, PDF supports medical decision-making and improves the transparency of predictive models in healthcare. Experimental results with three different on a COVID-19 severity classification dataset, MUSIC (a dataset for predicting outcomes in patients with several degrees of heart failure) and Wine Toy Dataset demonstrate that PDF achieves competitive performance while providing interpretable feature rankings that align with clinical knowledge.