BACKGROUND AND OBJECTIVES: Emergency departments (EDs) in the United States are overcrowded and nearing a breaking point. Alongside ever-increasing demand, one of the leading causes of ED overcrowding is the boarding of hospitalized patients in the ED as they await bed placement. We sought to develop a model for early prediction of hospitalizations, thus enabling an earlier start for the placement process and shorter boarding times. METHODS: We conducted a retrospective cohort analysis of all visits to the Boston Children’s Hospital ED from July 1, 2014 to June 30, 2015. We used 50% of the data for model derivation and the remaining 50% for validation. We built the predictive model by using a mixed method approach, running a logistic regression model on results generated by a naive Bayes classifier. We performed sensitivity analyses to evaluate the impact of the model on overall resource utilization. RESULTS: Our analysis comprised 59 033 patient visits, of which 11 975 were hospitalized (cases) and 47 058 were discharged (controls). Using data available within the first 30 minutes from presentation, our model identified 73.4% of the hospitalizations with 90% specificity and 35.4% of hospitalizations with 99.5% specificity (area under the curve = 0.91). Applying this model in a real-time setting could potentially save the ED 5917 hours per year or 30 minutes per hospitalization. CONCLUSIONS: This approach can accurately predict patient hospitalization early in the ED encounter by using data commonly available in most electronic medical records. Such early identification can be used to advance patient placement processes and shorten ED boarding times.
Introduction One of the factors contributing to ED crowding is the lengthy delay in transferring an admitted patient from the ED to an inpatient department (ie, boarding time). An earlier start of the admission process using an automatic hospitalisation prediction model could potentially shorten these delays and reduce crowding. Methods Clinical, operational and demographic data were retrospectively collected on 80 880 visits to the ED of Rambam Health Care Campus in Haifa, Israel, from January 2011 to January 2012. Using these data, a logistic regression model was developed to predict patient disposition (hospitalisation vs discharge) at three progressive time points throughout the ED visit: within the first 10 min, within an hour and within 2 hours. The algorithm was trained on 50% of the data (n=40 440) and tested on the remaining 50%. Results During the study time period, 58 197 visits ended in discharge and 22 683 in hospitalisation. Within 1 hour of presentation, our model was able to predict hospitalisation with a specificity of 90%, sensitivity of 94% and an AUCof 0.97. Early clinical decisions such as testing for calcium levels were found to be highly predictive of hospitalisations. In the Rambam ED, the use of such a prediction system would have the potential to save more than 250 patient hours per day. Conclusions Data collected by EDs in electronic medical records can be used within a progressive modelling framework to predict patient flow and improve clinical operations. This approach relies on commonly available data and can be applied across different healthcare settings.
Several approaches exist today for developing predictive models across multiple clinical sites, yet there is a lack of comparative data on their performance, especially within the context of EHR-based prediction models. We set out to provide a framework for prediction across healthcare settings. As a case study, we examined an ED disposition prediction model across three geographically and demographically diverse sites. We conducted a 1-year retrospective study, including all visits in which the outcome was either discharge-to-home or hospitalization. Four modeling approaches were compared: a ready-made model trained at one site and validated at other sites, a centralized uniform model incorporating data from all sites, multiple site-specific models, and a hybrid approach of a ready-made model re-calibrated using site-specific data. Predictions were performed using XGBoost. The study included 288,962 visits with an overall admission rate of 16.8% (7.9–26.9%). Some risk factors for admission were prominent across all sites (e.g., high-acuity triage emergency severity index score, high prior admissions rate), while others were prominent at only some sites (multiple lab tests ordered at the pediatric sites, early use of ECG at the adult site). The XGBoost model achieved its best performance using the uniform and site-specific approaches (AUC = 0.9–0.93), followed by the calibrated-model approach (AUC = 0.87–0.92), and the ready-made approach (AUC = 0.62–0.85). Our results show that site-specific customization is a key driver of predictive model performance.
Growing evidence has shown that applying machine learning models to large clinical data sources may exceed clinician performance in suicide risk stratification. However, many existing prediction models either suffer from "temporal bias" (a bias that stems from using case-control sampling) or require training on all available patient visit data. Here, we adopt a "landmark model" framework that aligns with clinical practice for prediction of suicide-related behaviors (SRBs) using a large electronic health record database. Using the landmark approach, we developed models for SRB prediction (regularized Cox regression and random survival forest) that establish a time-point (e.g., clinical visit) from which predictions are made over user-specified prediction windows using historical information up to that point. We applied this approach to cohorts from three clinical settings: general outpatient, psychiatric emergency department, and psychiatric inpatients, for varying prediction windows and lengths of historical data. Models achieved high discriminative performance (area under the Receiver Operating Characteristic curve 0.74-0.93 for the Cox model) across different prediction windows and settings, even with relatively short periods of historical data. In short, we developed accurate, dynamic SRB risk prediction models with the landmark approach that reduce bias and enhance the reliability and portability of suicide risk prediction models.
Clinical risk prediction models powered by electronic health records (EHRs) are becoming increasingly widespread in clinical practice. With suicide-related mortality rates rising in recent years, it is becoming increasingly urgent to understand, predict, and prevent suicidal behavior. Here, we compare the predictive value of structured and unstructured EHR data for predicting suicide risk. We find that Naive Bayes Classifier (NBC) and Random Forest (RF) models trained on structured EHR data perform better than those based on unstructured EHR data. An NBC model trained on both structured and unstructured data yields similar performance (AUC = 0.743) to an NBC model trained on structured data alone (0.742, p = 0.668), while an RF model trained on both data types yields significantly better results (AUC = 0.903) than an RF model trained on structured data alone (0.887, p < 0.001), likely due to the RF model’s ability to capture interactions between the two data types. To investigate these interactions, we propose and implement a general framework for identifying specific structured-unstructured feature pairs whose interactions differ between case and non-case cohorts, and thus have the potential to improve predictive performance and increase understanding of clinical risk. We find that such feature pairs tend to capture heterogeneous pairs of general concepts, rather than homogeneous pairs of specific concepts. These findings and this framework can be used to improve current and future EHR-based clinical modeling efforts.