Machine learning (ML) and artificial intelligence (AI) algorithms have the potential to derive insights from clinical data and improve patient outcomes. However, these highly complex systems are sensitive to changes in the environment and liable to performance decay. Even after their successful integration into clinical practice, ML/AI algorithms should be continuously monitored and updated to ensure their long-term safety and effectiveness. To bring AI into maturity in clinical care, we advocate for the creation of hospital units responsible for quality assurance and improvement of these algorithms, which we refer to as “AI-QI” units. We discuss how tools that have long been used in hospital quality assurance and quality improvement can be adapted to monitor static ML algorithms. On the other hand, procedures for continual model updating are still nascent. We highlight key considerations when choosing between existing methods and opportunities for methodological innovation.

Background: Accurate projections of procedural case durations are complex but critical to the planning of perioperative staffing, operating room resources, and patient communication. Nonlinear prediction models using machine learning methods may provide opportunities for hospitals to improve upon current estimates of procedure duration. Objective: The aim of this study was to determine whether a machine learning algorithm scalable across multiple centers could make estimations of case duration within a tolerance limit because there are substantial resources required for operating room functioning that relate to case duration. Methods: Deep learning, gradient boosting, and ensemble machine learning models were generated using perioperative data available at 3 distinct time points: the time of scheduling, the time of patient arrival to the operating or procedure room (primary model), and the time of surgical incision or procedure start. The primary outcome was procedure duration, defined by the time between the arrival and the departure of the patient from the procedure room. Model performance was assessed by mean absolute error (MAE), the proportion of predictions falling within 20% of the actual duration, and other standard metrics. Performance was compared with a baseline method of historical means within a linear regression model. Model features driving predictions were assessed using Shapley additive explanations values and permutation feature importance. Results: A total of 1,177,893 procedures from 13 academic and private hospitals between 2016 and 2019 were used. Across all procedures, the median procedure duration was 94 (IQR 50-167) minutes. In estimating the procedure duration, the gradient boosting machine was the best-performing model, demonstrating an MAE of 34 (SD 47) minutes, with 46% of the predictions falling within 20% of the actual duration in the test data set. This represented a statistically and clinically significant improvement in predictions compared with a baseline linear regression model (MAE 43 min; P<.001; 39% of the predictions falling within 20% of the actual duration). The most important features in model training were historical procedure duration by surgeon, the word “free” within the procedure text, and the time of day. Conclusions: Nonlinear models using machine learning techniques may be used to generate high-performing, automatable, explainable, and scalable prediction models for procedure duration.

Opal is the first published example of a full-stack platform infrastructure for an implementation science designed for ML in anesthesia that solves the problem of leveraging ML for clinical decision support. Users interact with a secure online Opal web application to select a desired operating room (OR) case cohort for data extraction, visualize datasets with built-in graphing techniques, and run in-client ML or extract data for external use. Opal was used to obtain data from 29,004 unique OR cases from a single academic institution for pre-operative prediction of post-operative acute kidney injury (AKI) based on creatinine KDIGO criteria using predictors which included pre-operative demographic, past medical history, medications, and flowsheet information. To demonstrate utility with unsupervised learning, Opal was also used to extract intra-operative flowsheet data from 2995 unique OR cases and patients were clustered using PCA analysis and k-means clustering. A gradient boosting machine model was developed using an 80/20 train to test ratio and yielded an area under the receiver operating curve (ROC-AUC) of 0.85 with 95% CI [0.80–0.90]. At the default probability decision threshold of 0.5, the model sensitivity was 0.9 and the specificity was 0.8. K-means clustering was performed to partition the cases into two clusters and for hypothesis generation of potential groups of outcomes related to intraoperative vitals. Opal’s design has created streamlined ML functionality for researchers and clinicians in the perioperative setting and opens the door for many future clinical applications, including data mining, clinical simulation, high-frequency prediction, and quality improvement.

Background: Unplanned postoperative intensive care unit admissions (UIAs) are rare events that cause significant challenges to perioperative workflow. We describe the development of a machine-learning derived model to predict UIAs using only widely used preoperative variables. Methods: This was a 3-year retrospective review of all adult surgeries under the General, Vascular, and Thoracic surgical services with anticipated length of greater than 180 minutes at a single institution. A UIA was defined as any post-operative patient recovering in the post-anesthesia care unit (PACU) requiring direct transfer to the intensive care unit (ICU) for higher level of care. We developed our prediction model with a gradient-boosting decision tree algorithm (XGBoost). The model incorporated sixteen generalizable predictor variables that were derived from the demographics and surgical booking details. Validation and evaluation were performed with 10-fold cross validation, and model performance was evaluated using the area under the receiver operating characteristic (ROC) curve, sensitivity, specificity, and likelihood ratio. Results: A total of 4658 patients were included for analysis. The incidence of UIAs was 2.3%. With 10-fold cross validation, the area under the ROC curve was 0.80 (95% CI 0.74–0.86). Two decision thresholds were used, which achieved the best specificity of 94% (95% CI 92–96%), best positive likelihood ratio of 4.22 (95% CI 0.99–8.79), and best sensitivity of 82% (95% CI 58–100%). Conclusions: Our machine learning-derived model is a reliable tool for the perioperative clinician to predict a rare outcome in high-risk patients using only preoperative variables. Future studies will include prospective validation of this model at other institutions and real-time incorporation for improvement in perioperative workflow.

Acute kidney injury (AKI) is a major postoperative complication that lacks established intraoperative predictors. Our objective was to develop a prediction model using preoperative and high-frequency intraoperative data for postoperative AKI. In this retrospective cohort study, we evaluated 77,428 operative cases at a single academic center between 2016 and 2022. A total of 11,212 cases with serum creatinine (sCr) data were included in the analysis. Then, 8519 cases were randomly assigned to the training set and the remainder to the validation set. Fourteen preoperative and twenty intraoperative variables were evaluated using elastic net followed by hierarchical group least absolute shrinkage and selection operator (LASSO) regression. The training set was 56% male and had a median [IQR] age of 62 (51–72) and a 6% AKI rate. Retained model variables were preoperative sCr values, the number of minutes meeting cutoffs for urine output, heart rate, perfusion index intraoperatively, and the total estimated blood loss. The area under the receiver operator characteristic curve was 0.81 (95% CI, 0.77–0.85). At a score threshold of 0.767, specificity was 77% and sensitivity was 74%. A web application that calculates the model score is available online. Our findings demonstrate the utility of intraoperative time series data for prediction problems, including a new potential use of the perfusion index. Further research is needed to evaluate the model in clinical settings.

Background.: Early prediction of whether a liver allograft will be utilized for transplantation may allow better resource deployment during donor management and improve organ allocation. The national donor management goals (DMG) registry contains critical care data collected during donor management. We developed a machine learning model to predict transplantation of a liver graft based on data from the DMG registry. Methods.: Several machine learning classifiers were trained to predict transplantation of a liver graft. We utilized 127 variables available in the DMG dataset. We included data from potential deceased organ donors between April 2012 and January 2019. The outcome was defined as liver recovery for transplantation in the operating room. The prediction was made based on data available 12–18 h after the time of authorization for transplantation. The data were randomly separated into training (60%), validation (20%), and test sets (20%). We compared the performance of our models to the Liver Discard Risk Index. Results.: Of 13 629 donors in the dataset, 9255 (68%) livers were recovered and transplanted, 1519 recovered but used for research or discarded, 2855 were not recovered. The optimized gradient boosting machine classifier achieved an area under the curve of the receiver operator characteristic of 0.84 on the test set, outperforming all other classifiers. Conclusions.: This model predicts successful liver recovery for transplantation in the operating room, using data available early during donor management. It performs favorably when compared to existing models. It may provide real-time decision support during organ donor management and transplant logistics.