Multinomial Logistic Regression (MLR) has been advocated for developing clinical prediction models that distinguish between three or more unordered outcomes. We present a full‐factorial simulation study to examine the predictive performance of MLR models in relation to the relative size of outcome categories, number of predictors and the number of events per variable. It is shown that MLR estimated by Maximum Likelihood yields overfitted prediction models in small to medium sized data. In most cases, the calibration and overall predictive performance of the multinomial prediction model is improved by using penalized MLR. Our simulation study also highlights the importance of events per variable in the multinomial context as well as the total sample size. As expected, our study demonstrates the need for optimism correction of the predictive performance measures when developing the multinomial logistic prediction model. We recommend the use of penalized MLR when prediction models are developed in small data sets or in medium sized data sets with a small total sample size (ie, when the sizes of the outcome categories are balanced). Finally, we present a case study in which we illustrate the development and validation of penalized and unpenalized multinomial prediction models for predicting malignancy of ovarian cancer.

Many randomized trials evaluate an intervention effect on time‐to‐event outcomes. Individual participant data (IPD) from such trials can be obtained and combined in a so‐called IPD meta‐analysis (IPD‐MA), to summarize the overall intervention effect. We performed a narrative literature review to provide an overview of methods for conducting an IPD‐MA of randomized intervention studies with a time‐to‐event outcome. We focused on identifying good methodological practice for modeling frailty of trial participants across trials, modeling heterogeneity of intervention effects, choosing appropriate association measures, dealing with (trial differences in) censoring and follow‐up times, and addressing time‐varying intervention effects and effect modification (interactions).We discuss how to achieve this using parametric and semi‐parametric methods, and describe how to implement these in a one‐stage or two‐stage IPD‐MA framework. We recommend exploring heterogeneity of the effect(s) through interaction and non‐linear effects. Random effects should be applied to account for residual heterogeneity of the intervention effect. We provide further recommendations, many of which specific to IPD‐MA of time‐to‐event data from randomized trials examining an intervention effect.We illustrate several key methods in a real IPD‐MA, where IPD of 1225 participants from 5 randomized clinical trials were combined to compare the effects of Carbamazepine and Valproate on the incidence of epileptic seizures.

Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor‐outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal‐external cross‐validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold‐out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta‐analysis of calibration and discrimination performance in each hold‐out cluster shows that trade‐offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.

Objective: To externally validate various prognostic models and scoring rules for predicting short term mortality in patients admitted to hospital for covid-19. Design: Two stage individual participant data meta-analysis. Setting: Secondary and tertiary care. Participants: 46 914 patients across 18 countries, admitted to a hospital with polymerase chain reaction confirmed covid-19 from November 2019 to April 2021. Data sources: Multiple (clustered) cohorts in Brazil, Belgium, China, Czech Republic, Egypt, France, Iran, Israel, Italy, Mexico, Netherlands, Portugal, Russia, Saudi Arabia, Spain, Sweden, United Kingdom, and United States previously identified by a living systematic review of covid-19 prediction models published in The BMJ , and through PROSPERO, reference checking, and expert knowledge. Model selection and eligibility criteria: Prognostic models identified by the living systematic review and through contacting experts. A priori models were excluded that had a high risk of bias in the participant domain of PROBAST (prediction model study risk of bias assessment tool) or for which the applicability was deemed poor. Methods: Eight prognostic models with diverse predictors were identified and validated. A two stage individual participant data meta-analysis was performed of the estimated model concordance (C) statistic, calibration slope, calibration-in-the-large, and observed to expected ratio (O:E) across the included clusters. Main outcome measures: 30 day mortality or in-hospital mortality. Results: Datasets included 27 clusters from 18 different countries and contained data on 46 914patients. The pooled estimates ranged from 0.67 to 0.80 (C statistic), 0.22 to 1.22 (calibration slope), and 0.18 to 2.59 (O:E ratio) and were prone to substantial between study heterogeneity. The 4C Mortality Score by Knight et al (pooled C statistic 0.80, 95% confidence interval 0.75 to 0.84, 95% prediction interval 0.72 to 0.86) and clinical model by Wang et al (0.77, 0.73 to 0.80, 0.63 to 0.87) had the highest discriminative ability. On average, 29% fewer deaths were observed than predicted by the 4C Mortality Score (pooled O:E 0.71, 95% confidence interval 0.45 to 1.11, 95% prediction interval 0.21 to 2.39), 35% fewer than predicted by the Wang clinical model (0.65, 0.52 to 0.82, 0.23 to 1.89), and 4% fewer than predicted by Xie et al’s model (0.96, 0.59 to 1.55, 0.21 to 4.28). Conclusion: The prognostic value of the included models varied greatly between the data sources. Although the Knight 4C Mortality Score and Wang clinical model appeared most promising, recalibration (intercept and slope updates) is needed before implementation in routine care.

Over the past few years, evidence synthesis has become essential to investigate and improve the generalizability of medical research findings. This strategy often involves a meta-analysis to formally summarize quantities of interest, such as relative treatment effect estimates. The use of meta-analysis methods is, however, less straightforward in prognosis research because substantial variation exists in research objectives, analysis methods and the level of reported evidence. We present a gentle overview of statistical methods that can be used to summarize data of prognostic factor and prognostic model studies. We discuss how aggregate data, individual participant data, or a combination thereof can be combined through meta-analysis methods. Recent examples are provided throughout to illustrate the various methods.

Missing data are a recurrent problem in all types of medical and epidemiological studies, regardless of the rigour with which they are designed and executed. Although causes of missing data can greatly vary, their presence is often addressed by omitting study participants with incomplete records from the analysis or by adopting multiple imputation methods. ¹ These strategies generally assume that missing values can be recovered from the observed data without introducing bias in study results (that is, data is missing at random; MAR). Unfortunately, problems arise when the missing data process is informative, i.e. both the probability of data availability and the value of missing observations are affected by unobservable or unmeasured factors. In this situation, data are missing not at random (MNAR). ² For instance, consider a health survey in which human immunodeficiency virus (HIV) testing is conducted to determine HIV prevalence in a population. The willingness to participate in a survey is rarely completely random, and often depends on observed (e.g. age, gender) or even unobserved (e.g. level of trust in the health care system, historical information on HIV status) factors. Missing values can therefore arise for key survey variables (e.g. HIV test result) when individuals refuse to participate or do not report some information. One option to address these missing values is to adopt traditional imputation methods and to recover incomplete variables (e.g. HIV status) from other observed survey variables (e.g. age, gender, sexual orientation). Although this approach is reasonable when the measured variables are strongly related to the missing data, validity is questionable when variables that affect the survey participation and the incomplete variables are not measured. In such a situation, it becomes important to adopt more advanced imputation methods that explicitly account for the causes of missing data. ^{3, 4}