Medical researchers and clinicians have shown much interest in developing machine learning (ML) algorithms to detect/predict surgical site infections (SSIs). However, little is known about the overall performance of ML algorithms in predicting SSIs and how to improve the algorithm's robustness. We conducted a systematic review and meta-analysis to summarize the performance of ML algorithms in SSIs case detection and prediction and to describe the impact of using unstructured and textual data in the development of ML algorithms. MEDLINE, EMBASE, CINAHL, CENTRAL and Web of Science were searched from inception to March 25, 2021. Study characteristics and algorithm development information were extracted. Performance statistics (e.g., sensitivity, area under the receiver operating characteristic curve [AUC]) were pooled using a random effect model. Stratified analysis was applied to different study characteristic levels. Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Diagnostic Test Accuracy Studies (PRISMA-DTA) was followed. Of 945 articles identified, 108 algorithms from 32 articles were included in this review. The overall pooled estimate of the SSI incidence rate was 3.67%, 95% CI: 3.58–3.76. Mixed-use of structured and textual data-based algorithms (pooled estimates of sensitivity 0.83, 95% CI: 0.78–0.87, specificity 0.92, 95% CI: 0.86–0.95, AUC 0.92, 95% CI: 0.89–0.94) outperformed algorithms solely based on structured data (sensitivity 0.56, 95% CI:0.43–0.69, specificity 0.95, 95% CI:0.91–0.97, AUC = 0.90, 95% CI: 0.87–0.92). ML algorithms developed with structured and textual data provided optimal performance. External validation of ML algorithms is needed to translate current knowledge into clinical practice.

Background: Population based surveillance of surgical site infections (SSIs) requires precise case-finding strategies. We sought to develop and validate machine learning models to automate the process of complex (deep incisional/organ space) SSIs case detection. Methods: This retrospective cohort study included adult patients (age ≥ 18 years) admitted to Calgary, Canada acute care hospitals who underwent primary total elective hip (THA) or knee (TKA) arthroplasty between Jan 1st, 2013 and Aug 31st, 2020. True SSI conditions were judged by the Alberta Health Services Infection Prevention and Control (IPC) program staff. Using the IPC cases as labels, we developed and validated nine XGBoost models to identify deep incisional SSIs, organ space SSIs and complex SSIs using administrative data, electronic medical records (EMR) free text data, and both. The performance of machine learning models was assessed by sensitivity, specificity, positive predictive value, negative predictive value, F1 score, the area under the receiver operating characteristic curve (ROC AUC) and the area under the precision–recall curve (PR AUC). In addition, a bootstrap 95% confidence interval (95% CI) was calculated. Results: There were 22,059 unique patients with 27,360 hospital admissions resulting in 88,351 days of hospital stay. This included 16,561 (60.5%) TKA and 10,799 (39.5%) THA procedures. There were 235 ascertained SSIs. Of them, 77 (32.8%) were superficial incisional SSIs, 57 (24.3%) were deep incisional SSIs, and 101 (42.9%) were organ space SSIs. The incidence rates were 0.37 for superficial incisional SSIs, 0.21 for deep incisional SSIs, 0.37 for organ space and 0.58 for complex SSIs per 100 surgical procedures, respectively. The optimal XGBoost models using administrative data and text data combined achieved a ROC AUC of 0.906 (95% CI 0.835–0.978), PR AUC of 0.637 (95% CI 0.528–0.746), and F1 score of 0.79 (0.67–0.90). Conclusions: Our findings suggest machine learning models derived from administrative data and EMR text data achieved high performance and can be used to automate the detection of complex SSIs.

Objective To evaluate the validity of COVID-19 International Classification of Diseases, 10th Revision (ICD-10) codes and their combinations. Design Retrospective cohort study. Setting Acute care hospitals and emergency departments (EDs) in Alberta, Canada. Participants Patients who were admitted to hospital or presented to an ED in Alberta, as captured by local administrative databases between 1 March 2020 and 28 February 2021, who had a positive COVID-19 test and/or a COVID-19-related ICD-10 code. Main outcome measures The sensitivity, positive predictive value (PPV) and 95% CIs for ICD-10 codes were computed. Stratified analysis on age group, sex, symptomatic status, mechanical ventilation, hospital type, patient intensive care unit (ICU) admission, discharge status and season of pandemic were conducted. Results Two overlapping subsets of the study population were considered: those who had a positive COVID-19 test (cohort A, for estimating sensitivity) and those who had a COVID-19-related ICD-10 code (cohort B, for estimating PPV). Cohort A included 17 979 ED patients and 6477 inpatients while cohort B included 33 675 ED patients and 18 746 inpatients. Of inpatients, 9.5% in cohort A and 8.1% in cohort B received mechanical ventilation. Over 13% of inpatients were admitted to ICU. The length of hospital stay was 6 days (IQR: 3–14) for cohort A and 8 days (IQR: 3–19) for cohort B. In-hospital mortality was 15.9% and 38.8% for cohort A and B, respectively. The sensitivity for ICD-10 code U07.1 (COVID-19, virus identified) was 82.5% (81.8%–83.2%) with a PPV of 93.1% (92.6%–93.6%). The combination of U07.1 and U07.3 (multisystem inflammatory syndrome associated with COVID-19) had a sensitivity of 82.5% (81.9%–83.2%) and PPV of 92.9% (92.4%–93.4%). Conclusions In Alberta, ICD-10 COVID-19 codes (U07.1 and U07.3) were coded well with high validity. This indicates administrative data can be used for COVID-19 research and pandemic management purposes.

Background: Inpatient falls are a substantial concern for health care providers and are associated with negative outcomes for patients. Automated detection of falls using machine learning (ML) algorithms may aid in improving patient safety and reducing the occurrence of falls. Objective: This study aims to develop and evaluate an ML algorithm for inpatient fall detection using multidisciplinary progress record notes and a pretrained Bidirectional Encoder Representation from Transformers (BERT) language model. Methods: A cohort of 4323 adult patients admitted to 3 acute care hospitals in Calgary, Alberta, Canada from 2016 to 2021 were randomly sampled. Trained reviewers determined falls from patient charts, which were linked to electronic medical records and administrative data. The BERT-based language model was pretrained on clinical notes, and a fall detection algorithm was developed based on a neural network binary classification architecture. Results: To address various use scenarios, we developed 3 different Alberta hospital notes-specific BERT models: a high sensitivity model (sensitivity 97.7, IQR 87.7-99.9), a high positive predictive value model (positive predictive value 85.7, IQR 57.2-98.2), and the high F1-score model (F1=64.4). Our proposed method outperformed 3 classical ML algorithms and an International Classification of Diseases code–based algorithm for fall detection, showing its potential for improved performance in diverse clinical settings. Conclusions: The developed algorithm provides an automated and accurate method for inpatient fall detection using multidisciplinary progress record notes and a pretrained BERT language model. This method could be implemented in clinical practice to improve patient safety and reduce the occurrence of falls in hospitals.

Background In medical research, the effectiveness of machine learning algorithms depends heavily on the accuracy of labeled data. This study aimed to assess inter-rater reliability (IRR) in a retrospective electronic medical chart review to create high quality labeled data on comorbidities and adverse events (AEs). Methods Six registered nurses with diverse clinical backgrounds reviewed patient charts, extracted data on 20 predefined comorbidities and 18 AEs. All reviewers underwent four iterative rounds of training aimed to enhance accuracy and foster consensus. Periodic monitoring was conducted at the beginning, middle, and end of the testing phase to ensure data quality. Weighted Kappa coefficients were calculated with their associated 95% confidence intervals (CIs). Results Seventy patient charts were reviewed. The overall agreement, measured by Conger's Kappa, was 0.80 (95% CI: 0.78-0.82). IRR scores remained consistently high (ranging from 0.70 to 0.87) throughout each phase. Conclusion Our study suggests the detailed manual for chart review and structured training regimen resulted in a consistently high level of agreement among our reviewers during the chart review process. This establishes a robust foundation for generating high-quality labeled data, thereby enhancing the potential for developing accurate machine learning algorithms.

Background: Measurement of care quality and safety mainly relies on abstracted administrative data. However, it is well studied that administrative data-based adverse event (AE) detection methods are suboptimal due to lack of clinical information. Electronic medical records (EMR) have been widely implemented and contain detailed and comprehensive information regarding all aspects of patient care, offering a valuable complement to administrative data. Harnessing the rich clinical data in EMRs offers a unique opportunity to improve detection, identify possible risk factors of AE and enhance surveillance. However, the methodological tools for detection of AEs within EMR need to be developed and validated. The objectives of this study are to develop EMR-based AE algorithms from hospital EMR data and assess AE algorithm’s validity in Canadian EMR data. Methods: Patient EMR structured and text data from acute care hospitals in Calgary, Alberta, Canada will be linked with discharge abstract data (DAD) between 2010 and 2020 (n~1.5 million). AE algorithms development. First, a comprehensive list of AEs will be generated through a systematic literature review and expert recommendations. Second, these AEs will be mapped to EMR free texts using Natural Language Processing (NLP) technologies. Finally, an expert panel will assess the clinical relevance of the developed NLP algorithms. AE algorithms validation: We will test the newly developed AE algorithms on 10,000 randomly selected EMRs between 2010 to 2020 from Calgary, Alberta. Trained reviewers will review the selected 10,000 EMR charts to identify AEs that had occurred during hospitalization. Performance indicators (e.g., sensitivity, specificity, positive predictive value, negative predictive value, F₁ score, etc.) of the developed AE algorithms will be assessed using chart review data as the reference standard. Discussion: The results of this project can be widely implemented in EMR based healthcare system to accurately and timely detect in-hospital AEs.