The performance of prediction models can be assessed using a variety of methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic [ROC] curve), and goodness-of-fit statistics for calibration. Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision–analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions. We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration, we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n = 544 for model development, n = 273 for external validation). We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.

Evaluating Random Forests for Survival Analysis Using Prediction Error Curves

The effect of bystander interventions on long-term functional outcomes among survivors of out-of-hospital cardiac arrest has not been extensively studied. We linked nationwide data on out-of-hospital cardiac arrests in Denmark to functional outcome data and reported the 1-year risks of anoxic brain damage or nursing home admission and of death from any cause among patients who survived to day 30 after an out-of-hospital cardiac arrest. We analyzed risks according to whether bystander cardiopulmonary resuscitation (CPR) or defibrillation was performed and evaluated temporal changes in bystander interventions and outcomes. Among the 2855 patients who were 30-day survivors of an out-of-hospital cardiac arrest during the period from 2001 through 2012, a total of 10.5% had brain damage or were admitted to a nursing home and 9.7% died during the 1-year follow-up period. During the study period, among the 2084 patients who had cardiac arrests that were not witnessed by emergency medical services (EMS) personnel, the rate of bystander CPR increased from 66.7% to 80.6% (P<0.001), the rate of bystander defibrillation increased from 2.1% to 16.8% (P<0.001), the rate of brain damage or nursing home admission decreased from 10.0% to 7.6% (P<0.001), and all-cause mortality decreased from 18.0% to 7.9% (P=0.002). In adjusted analyses, bystander CPR was associated with a risk of brain damage or nursing home admission that was significantly lower than that associated with no bystander resuscitation (hazard ratio, 0.62; 95% confidence interval [CI], 0.47 to 0.82), as well as a lower risk of death from any cause (hazard ratio, 0.70; 95% CI, 0.50 to 0.99) and a lower risk of the composite end point of brain damage, nursing home admission, or death (hazard ratio, 0.67; 95% CI, 0.53 to 0.84). The risks of these outcomes were even lower among patients who received bystander defibrillation as compared with no bystander resuscitation. In our study, we found that bystander CPR and defibrillation were associated with risks of brain damage or nursing home admission and of death from any cause that were significantly lower than those associated with no bystander resuscitation. (Funded by TrygFonden and the Danish Heart Foundation.)

We introduce a new approach to competing risks using random forests. Our method is fully non-parametric and can be used for selecting event-specific variables and for estimating the cumulative incidence function. We show that the method is highly effective for both prediction and variable selection in high-dimensional problems and in settings such as HIV/AIDS that involve many competing risks.