Glaucoma is a progressive optic nerve disease and a leading cause of irreversible blindness worldwide. Early and accurate detection is critical to prevent vision loss, yet traditional diagnostic methods such as optical coherence tomography and visual field tests face challenges in accessibility, cost, and consistency, especially in under-resourced areas. This study evaluates the clinical applicability and robustness of three machine learning models for automated glaucoma detection: a convolutional neural network, a deep neural network, and an automated ensemble approach. The models were trained and validated on retinal fundus images and tested on an independent dataset to assess their ability to generalize across different patient populations. Data preprocessing included resizing, normalization, and feature extraction to ensure consistency. Among the models, the deep neural network demonstrated the highest generalizability with stable performance across datasets, while the convolutional neural network showed moderate but consistent results. The ensemble model exhibited overfitting, which limited its practical use. These findings highlight the importance of proper evaluation frameworks, including external validation, to ensure the reliability of artificial intelligence tools for clinical use. The study provides insights into the development of scalable, effective diagnostic solutions that align with regulatory guidelines, addressing the critical need for accessible glaucoma detection tools in diverse healthcare settings.

Background: High in-domain accuracy in healthcare machine learning (ML) models does not guarantee reliable clinical performance, especially when training and validation protocols are insufficiently robust. This paper presents a standardized framework for training and validating ML models intended for classifying medical conditions, emphasizing the need for clinically relevant evaluation metrics and external validation. Methods: We apply this framework to a case study in knee osteoarthritis grading, demonstrating how overfitting, data leakage, and inadequate validation can lead to deceptively high accuracy that fails to translate into clinical reliability. In addition to conventional metrics, we introduce composite clinical measures that better capture real-world utility. Results: Our findings show that models with strong in-domain performance may underperform on external datasets, and that composite metrics provide a more nuanced assessment of clinical applicability. Conclusions: Standardized training and validation protocols, together with clinically oriented evaluation, are essential for developing ML models that are both statistically robust and clinically reliable across a range of medical classification tasks.

This study presents a clinical utility-driven machine learning framework for retinal Optical Coherence Tomography classification, addressing challenges posed by manual interpretation variability and dataset heterogeneity. The methodology integrates biomimetic data partitioning, deep biomarker extraction via pretrained VGG16 networks, and automated model selection optimized for clinical decision-making. Stratified data curation preserved pathological distributions across training, validation, and testing subsets, while SMOTE optimization mitigated class imbalance. Cross-pathology testing evaluated generalizability on anatomically distinct retinal conditions excluded from training, assessing the framework’s robustness to unseen pathologies. Clinical utility metrics prioritized alignment with ophthalmological imperatives, emphasizing negative predictive value to minimize false negatives and enhance diagnostic reliability. The framework advances AI-driven Optical Coherence Tomography diagnostics by harmonizing computational performance with patient-centered outcomes, enabling standardized disease detection across diverse clinical datasets through robust feature generalization.

This study evaluated the generalization and reliability of machine learning models for multiclass classification of retinal pathologies using a diverse set of images representing eight disease categories. Images were aggregated from two public datasets and divided into training, validation, and test sets, with an additional independent dataset used for external validation. Multiple modeling approaches were compared, including classical machine learning algorithms, convolutional neural networks with and without data augmentation, and a deep neural network using pre-trained feature extraction. Analysis of learning dynamics revealed that classical models and unaugmented convolutional neural networks exhibited overfitting and poor generalization, while models with data augmentation and the deep neural network showed healthy, parallel convergence of training and validation performance. Only the deep neural network demonstrated a consistent, monotonic decrease in accuracy, F1-score, and recall from training through external validation, indicating robust generalization. These results underscore the necessity of evaluating learning dynamics (not just summary metrics) to ensure model reliability and patient safety. Typically, model performance is expected to decrease gradually as data becomes less familiar. Therefore, models that do not exhibit these healthy learning dynamics, or that show unexpected improvements in performance on subsequent datasets, should not be considered for clinical application, as such patterns may indicate methodological flaws or data leakage rather than true generalization.

Selecting machine learning algorithms for clinical deployment demands comprehensive evaluation beyond conventional performance metrics. While automated frameworks simplify model generation, identifying algorithms suitable for real-world medical applications requires systematic assessment of learning dynamics, generalization stability, and cross-subset reliability. This study addresses prediabetes prediction through a multi-stage evaluation comparing automated machine learning frameworks, neural networks, gradient boosting implementations (XGBoost, CatBoost, LightGBM), and specialized imbalance-handling techniques. A questionnaire-based dataset with a substantial class imbalance was analyzed through progressive evaluation stages: aggregate performance metrics, learning curve analysis, minority class detection capability, and cross-subset generalization stability. Linear Discriminant Analysis achieved maximum validation metrics in automated screening but exhibited flat learning curves, indicating an exhausted learning capacity. XGBoost demonstrated optimal convergence dynamics with the highest validation performance (0.749 AUC), yet suffered substantial validation-to-test degradation (5.9 percentage points). CatBoost, despite inferior validation performance (0.696 accuracy), exhibited exceptional cross-subset stability with minimal performance decline (0.2 percentage points) while achieving a comparable test accuracy (0.694). CatBoost was selected for deployment based on its superior generalization stability, demonstrating that a multi-dimensional evaluation spanning aggregate metrics, learning dynamics, and cross-subset stability is essential for identifying clinically deployable models, as validation performance alone provides insufficient evidence for real-world applicability.