Predicting corporate bankruptcy is a key task in financial risk management, and selecting a machine learning model with superior generalization performance is crucial for prediction accuracy. This study evaluates the effectiveness of k-fold cross-validation as a model selection strategy for random forest and XGBoost classifiers using a publicly available dataset of Taiwanese listed companies. We employ a nested cross-validation framework to assess the relationship between cross-validation (CV) and out-of-sample (OOS) performance on 40 different train/test data partitions. On average, we find k-fold cross-validation to be a valid selection technique when applied within a model class; however, k-fold cross-validation may fail for specific train/test splits. We find that 67% of model selection regret variability is explained by the particular train/test split, highlighting an irreducible uncertainty real world practitioners must contend with. Our study extensively explores hyperparameter tuning for both classifiers and highlights key insights. Additionally, we investigate practical implementation choices in k-fold cross-validation—such as the value of k or prediction strategies. We conclude that k-fold cross-validation is effective for model selection within a model class and on average, but it can be unreliable in specific cases or when comparing models from different classes—this latter issue warranting further investigation.

Firm failure risk is an important topic for managers, credit institutions, investors and policymakers. In recent years, machine learning models have been increasingly used in this field. Although they often achieve high predictive performance, they typically lack interpretability. Existing literature on machine learning model interpretability focuses on individual variable importance with limited practical guidance for firm management. We use a large firm-level dataset of Romanian companies (360.249 observations) to train various XGBoost models and to examine their interpretability. We use a wide range of explanatory variables: firm size, industry classification, financial statement items scaled by total assets and 44 financial ratios. We use AUROC as performance measure, while firm failure is proxied by the transition from positive to negative equity. We calculate variable importance using the permutation feature importance method. Our results show that variable importance in high-dimensional settings can be unstable, as there are considerable redundancies and compensation effects between explanatory variables. Furthermore, individual variable changes have limited effects on predicted probabilities of failure, hence having little practical relevance. We illustrate that using coherent scenario-based modifications to financial statements has substantially larger and economically meaningful effects on predicted failure risk. Compared to variable importance ranking and score assignment, scenario-based analysis provides clearer interpretations and more relevant insights for management. Our contribution aims to shift the focus from individual variable importance to economically meaningful insights derived from machine learning models.