Background: Prediction models inform many medical decisions, but their performance often deteriorates over time. Several discrete-time update strategies have been proposed in the literature, including model recalibration and revision. However, these strategies have not been compared in the dynamic updating setting. Methods: We used post-lung transplant survival data during 2010-2015 and compared the Brier Score (BS), discrimination, and calibration of the following update strategies: (1) never update, (2) update using the closed testing procedure proposed in the literature, (3) always recalibrate the intercept, (4) always recalibrate the intercept and slope, and (5) always refit/revise the model. In each case, we explored update intervals of every 1, 2, 4, and 8 quarters. We also examined how the performance of the update strategies changed as the amount of old data included in the update (i.e., sliding window length) increased. Results: All methods of updating the model led to meaningful improvement in BS relative to never updating. More frequent updating yielded better BS, discrimination, and calibration, regardless of update strategy. Recalibration strategies led to more consistent improvements and less variability over time compared to the other updating strategies. Using longer sliding windows did not substantially impact the recalibration strategies, but did improve the discrimination and calibration of the closed testing procedure and model revision strategies. Conclusions: Model updating leads to improved BS, with more frequent updating performing better than less frequent updating. Model recalibration strategies appeared to be the least sensitive to the update interval and sliding window length.

Prediction modeling for clinical decision making is of great importance and needed to be updated frequently with the changes of patient population and clinical practice. Existing methods are either done in an ad hoc fashion, such as model recalibration or focus on studying the relationship between predictors and outcome and less so for the purpose of prediction. In this article, we propose a dynamic logistic state space model to continuously update the parameters whenever new information becomes available. The proposed model allows for both time‐varying and time‐invariant coefficients. The varying coefficients are modeled using smoothing splines to account for their smooth trends over time. The smoothing parameters are objectively chosen by maximum likelihood. The model is updated using batch data accumulated at prespecified time intervals, which allows for better approximation of the underlying binomial density function. In the simulation, we show that the new model has significantly higher prediction accuracy compared to existing methods. We apply the method to predict 1 year survival after lung transplantation using the United Network for Organ Sharing data.

Rationale: In the United States, donor lungs are allocated to transplant candidates on the basis of lung allocation scores (LAS). However, additional factors beyond the LAS can impact who is transplanted, including listing and donor–organ acceptance practices. These factors can result in differential selection, undermining the objectivity of lung allocation. Yet their impact on the lung transplant pathway has been underexplored. Objectives: We sought to systematically examine sources of differential selection in lung transplantation via qualitative methods. Methods: We conducted semistructured qualitative interviews with lung transplant surgeons and pulmonologists in the United States between June 2019 and June 2020 to understand clinician perspectives on differential selection in lung transplantation and the LAS. Results: A total of 51 respondents (30 surgeons and 21 pulmonologists) identified many sources of differential selection arising throughout the pathway from referral to transplantation. We synthesized these sources into a conceptual model with five themes: 1) transplant center’s degree of risk tolerance and accountability; 2) successfulness and fairness of the LAS; 3) donor–organ availability and regional competition; 4) patient health versus program health; and 5) access to care versus responsible stewardship of organs. Conclusions: Our conceptual model demonstrates how differential selection can arise throughout lung transplantation and facilitates the further study of such selection. As new organ allocation models are developed, differential selection should be considered carefully to ensure that these models are more equitable.

Background Prediction models inform decisions in many areas of medicine. Most models are fitted once and then applied to new (future) patients, despite the fact that model coefficients can vary over time due to changes in patients' clinical characteristics and disease risk. However, the optimal method to detect changes in model parameters has not been rigorously assessed. Methods We simulated data, informed by post–lung transplant mortality data and tested the following two approaches for detecting model change: (1) the “Direct Approach,” it compares coefficients of the model refit on recent data to those at baseline; and (2) “Calibration Regression,” it fits a logistic regression model of the log-odds of the observed outcomes versus the linear predictor from the baseline model (i.e., the log-odds of the predicted probabilities obtained from the baseline model) and tests whether the intercept and slope differ from 0 and 1, respectively. Four scenarios were simulated using logistic regression for binary outcomes as follows: (1) we fixed all model parameters, (2) we varied the outcome prevalence between 0.1 and 0.2, (3) we varied the coefficient of one of the ten predictors between 0.2 and 0.4, and (4) we varied the outcome prevalence and coefficient of one predictor simultaneously. Results Calibration regression tended to detect changes sooner than the Direct Approach, with better performance (e.g., larger proportion of true claims). When the sample size was large, both methods performed well. When two parameters changed simultaneously, neither method performed well. Conclusion Neither change detection method examined here proved optimal under all circumstances. However, our results suggest that if one is interested in detecting a change in overall incidence of an outcome (e.g., intercept), the Calibration Regression method may be superior to the Direct Approach. Conversely, if one is interested in detecting a change in other model covariates (e.g., slope), the Direct Approach may be superior.