Classifiers trained on high-dimensional data, such as genetic datasets, often encounter situations where the number of features exceeds the number of objects. In these cases, classifiers typically rely on a small subset of features. For a robust algorithm, this subset should remain relatively stable with minor changes in the training data, such as the replacement of a few samples. While the stability of feature selection is a common focus in studies of feature selection algorithms, it is less emphasized in classifier evaluation, where only metrics such as accuracy are commonly used. We investigate the importance of feature selection stability through an empirical study of four classifiers (logistic regression, support vector machine, convex and piecewise Linear, and Random Forest) on seven high dimensional, publicly available, gene datasets. We measure the stability of feature selection using Lustgarten, Nogueira and Jaccard Index measures. We employed our own cross-validation procedure that guarantees a difference of exactly p objects between any two training sets which allows us to control the level of disturbance in the data. Our results show the existence of a drop in feature selection stability when we increase disturbance in the data in all 28 experiment configurations (seven datasets and four classifiers). The relationship is not linear, but resembles more of a hyperbolic pattern. In the evaluation of the tested classifiers, logistic regression demonstrated the highest stability. It was followed in order by support vector machine, convex and piecewise linear, with Random Forest exhibiting the lowest stability among them. This work provides evidence that all tested classifiers are very sensitive to even small changes in learning data in terms of features used by the model, while showing almost no sensitivity in terms of accuracy. The data and codes to reproduce the results in the article are available publicly on GitHub: https://github.com/tlukaszuk/feature-selection-stability-in-classifier-evaluation.

In the rapidly evolving field of multi-omics data analysis, understanding the stability of feature selection is critical for reliable biomarker discovery and clinical applications. This study investigates the stability of feature-selection methods across various cancer types by utilizing 15 datasets from The Cancer Genome Atlas (TCGA). We employed classifiers with embedded feature selection, including Support Vector Machines (SVM), Logistic Regression (LR), and Lasso regression, each incorporating L1 regularization. Through a comprehensive evaluation using five-fold cross-validation, we measured feature-selection stability and assessed the accuracy of predictions regarding TP53 mutations, a known indicator of poor clinical outcomes in cancer patients. All three classifiers demonstrated optimal feature-selection stability, measured by the Nogueira metric, with higher regularization (fewer selected features), while lower regularization generally resulted in decreased stability across all omics layers. Our findings indicate differences in feature stability across the various omics layers; mirna consistently exhibited the highest stability across classifiers, while the mutation and rna layers were generally less stable, particularly with lower regularization. This work highlights the importance of careful feature selection and validation in high-dimensional datasets to enhance the robustness and reliability of multi-omics analyses.

In complex diseases such as chronic kidney disease (CKD), the risk of clinical complications is determined by interactions between phenotypic and genotypic factors. However, clinical epidemiological studies rarely attempt to analyse the combined effect of large numbers of phenotype and genotype features. We have recently shown that the relaxed linear separability (RLS) model of feature selection can address such complex issues. Here, it is applied to identify risk factors for inflammation in CKD. The RLS model was applied in 225 CKD stage 5 patients sampled in conjunction with dialysis initiation. Fifty-seven anthropometric or biochemical measurements and 79 genetic polymorphisms were entered into the model. The model was asked to identify phenotypes and genotypes that, when combined, could separate inflamed from non-inflamed patients. Inflammation was defined as a high-sensitivity C-reactive protein concentration above the median (5 mg/L). Among the 60 genotypic and phenotypic features predicting inflammation, 31 were genetic. Among the 10 strongest predictors of inflammation, 8 were single nucleotide polymorphisms located in the NAMPT, CIITA, BMP2 and PIK3CB genes, whereas fibrinogen and bone mineral density were the only phenotypic biomarkers. These results indicate a larger involvement of hereditary factors in inflammation than might have been expected and suggest that inclusion of genotype features in risk assessment studies is critical. The RLS model demonstrates that inflammation in CKD is determined by an extensive panel of factors and may prove to be a suitable tool that could enable a much-needed multifactorial approach as opposed to the commonly utilized single-factor analysis.

Identification of risk factors in patients with a particular disease can be analyzed in clinical data sets by using feature selection procedures of pattern recognition and data mining methods. The applicability of the relaxed linear separability (RLS) method of feature subset selection was checked for high-dimensional and mixed type (genetic and phenotypic) clinical data of patients with end-stage renal disease. The RLS method allowed for substantial reduction of the dimensionality through omitting redundant features while maintaining the linear separability of data sets of patients with high and low levels of an inflammatory biomarker. The synergy between genetic and phenotypic features in differentiation between these two subgroups was demonstrated.