Background: Machine learning algorithms are currently used in a wide array of clinical domains to produce models that can predict clinical risk events. Most models are developed and evaluated with retrospective data, very few are evaluated in a clinical workflow, and even fewer report performances in different hospitals. In this study, we provide detailed evaluations of clinical risk prediction models in live clinical workflows for three different use cases in three different hospitals. Objective: The main objective of this study was to evaluate clinical risk prediction models in live clinical workflows and compare their performance in these setting with their performance when using retrospective data. We also aimed at generalizing the results by applying our investigation to three different use cases in three different hospitals. Methods: We trained clinical risk prediction models for three use cases (ie, delirium, sepsis, and acute kidney injury) in three different hospitals with retrospective data. We used machine learning and, specifically, deep learning to train models that were based on the Transformer model. The models were trained using a calibration tool that is common for all hospitals and use cases. The models had a common design but were calibrated using each hospital’s specific data. The models were deployed in these three hospitals and used in daily clinical practice. The predictions made by these models were logged and correlated with the diagnosis at discharge. We compared their performance with evaluations on retrospective data and conducted cross-hospital evaluations. Results: The performance of the prediction models with data from live clinical workflows was similar to the performance with retrospective data. The average value of the area under the receiver operating characteristic curve (AUROC) decreased slightly by 0.6 percentage points (from 94.8% to 94.2% at discharge). The cross-hospital evaluations exhibited severely reduced performance: the average AUROC decreased by 8 percentage points (from 94.2% to 86.3% at discharge), which indicates the importance of model calibration with data from the deployment hospital. Conclusions: Calibrating the prediction model with data from different deployment hospitals led to good performance in live settings. The performance degradation in the cross-hospital evaluation identified limitations in developing a generic model for different hospitals. Designing a generic process for model development to generate specialized prediction models for each hospital guarantees model performance in different hospitals.

Induction of hypoxia-inducible-factor-1α (HIF-1α) pathway and HIF-target genes allow adaptation to hypoxia and are associated with reduced incidence of acute mountain sickness (AMS). Little is known about HIF-pathways in conjunction with inflammation or exercise stimuli under acute hypobaric hypoxia in non-acclimatized individuals. We therefore tested the hypotheses that (1) both hypoxic and inflammatory stimuli induce hypoxic-inflammatory signaling pathways in vitro, (2) similar results are seen in vivo under hypobaric hypoxia, and (3) induction of HIF-dependent genes is associated with AMS in 11 volunteers. In vitro, peripheral blood mononuclear cells (PBMCs) were incubated under hypoxic (10%/5% O₂) or inflammatory (CD3/CD28) conditions. In vivo, Interleukin 1β (IL-1β), C-X-C Chemokine receptor type 4 (CXCR-4), and C-C Chemokine receptor type 2 (CCR-2) mRNA expression, cytokines and receptors were analyzed under normoxia (520 m above sea level (a.s.l.)), hypobaric hypoxia (3883 m a.s.l.) before/after exercise, and after 24 h under hypobaric hypoxia. In vitro, isolated hypoxic (p = 0.004) or inflammatory (p = 0.006) stimuli induced IL-1β mRNA expression. CCR-2 mRNA expression increased under hypoxia (p = 0.005); CXCR-4 mRNA expression remained unchanged. In vivo, cytokines, receptors, and IL-1β, CCR-2 and CXCR-4 mRNA expression increased under hypobaric hypoxia after 24 h (all p ≤ 0.05). Of note, proinflammatory IL-1β and CXCR-4 mRNA expression changes were associated with symptoms of AMS. Thus, hypoxic-inflammatory pathways are differentially regulated, as combined hypoxic and exercise stimulus was stronger in vivo than isolated hypoxic or inflammatory stimulation in vitro.

Machine learning (ML) algorithms are now widely used in predicting acute events for clinical applications. While most of such prediction applications are developed to predict the risk of a particular acute event at one hospital, few efforts have been made in extending the developed solutions to other events or to different hospitals. We provide a scalable solution to extend the process of clinical risk prediction model development of multiple diseases and their deployment in different Electronic Health Records (EHR) systems. We defined a generic process for clinical risk prediction model development. A calibration tool has been created to automate the model generation process. We applied the model calibration process at four hospitals, and generated risk prediction models for delirium, sepsis and acute kidney injury (AKI) respectively at each of these hospitals. The delirium risk prediction models have on average an area under the receiver-operating characteristic curve (AUROC) of 0.82 at admission and 0.95 at discharge on the test datasets of the four hospitals. The sepsis models have on average an AUROC of 0.88 and 0.95, and the AKI models have on average an AUROC of 0.85 and 0.92, at the day of admission and discharge respectively. The scalability discussed in this paper is based on building common data representations (syntactic interoperability) between EHRs stored in different hospitals. Semantic interoperability, a more challenging requirement that different EHRs share the same meaning of data, e.g. a same lab coding system, is not mandated with our approach. Our study describes a method to develop and deploy clinical risk prediction models in a scalable way. We demonstrate its feasibility by developing risk prediction models for three diseases across four hospitals.

Normobaric (NH) and hypobaric hypoxia (HH) are associated with acute mountain sickness (AMS) and cognitive dysfunction. Only few variables, like heart-rate-variability, are correlated with AMS. However, prediction of AMS remains difficult. We therefore designed an expedition-study with healthy volunteers in NH/HH to investigate additional non-invasive hemodynamic variables associated with AMS. Eleven healthy subjects were examined in NH (FiO2 13.1%; equivalent of 3.883 m a.s.l; duration 4 h) and HH (3.883 m a.s.l.; duration 24 h) before and after an exercise of 120 min. Changes in parameters of electrical cardiometry (cardiac index (CI), left-ventricular ejection time (LVET), stroke volume (SV), index of contractility (ICON)), near-infrared spectroscopy (cerebral oxygenation, rScO2), Lake-Louise-Score (LLS) and cognitive function tests were assessed. One-Way-ANOVA, Wilcoxon matched-pairs test, Spearman’s-correlation-analysis and Student’s t-test were performed. HH increased heart rate (HR), mean arterial pressure (MAP) and CI and decreased LVET, SV and ICON, whereas NH increased HR and decreased LVET. In both NH and HH cerebral oxygenation decreased and LLS increased significantly. After 24 h in HH, 6 of 11 subjects (54.6%) developed AMS. LLS remained increased until 24 h in HH, whereas cognitive function remained unaltered. In HH, HR and LLS were inversely correlated (r = − 0.692; p < 0.05). More importantly, the rScO2-decrease after exercise in NH significantly correlated with LLS after 24 h in HH (r = − 0.971; p < 0.01) and rScO2 correlated significantly with HR (r = 0.802; p < 0.01), CI (r = 0.682; p < 0.05) and SV (r = 0.709; p < 0.05) after exercise in HH. Both acute NH and HH altered hemodynamic and cerebral oxygenation and induced AMS. Subjects, who adapted their CI had higher rScO2 and lower LLS. Furthermore, rScO2 after exercise under normobaric conditions was associated with AMS at high altitudes.

Background: In recent years, machine learning (ML)–based models have been widely used in clinical domains to predict clinical risk events. However, in production, the performances of such models heavily rely on changes in the system and data. The dynamic nature of the system environment, characterized by continuous changes, has significant implications for prediction models, leading to performance degradation and reduced clinical efficacy. Thus, monitoring model shifts and evaluating their impact on prediction models are of utmost importance. Objective: This study aimed to assess the impact of a model shift on ML-based prediction models by evaluating 3 different use cases—delirium, sepsis, and acute kidney injury (AKI)—from 2 hospitals (M and H) with different patient populations and investigate potential model deterioration during the COVID-19 pandemic period. Methods: We trained prediction models using retrospective data from earlier years and examined the presence of a model shift using data from more recent years. We used the area under the receiver operating characteristic curve (AUROC) to evaluate model performance and analyzed the calibration curves over time. We also assessed the influence on clinical decisions by evaluating the alert rate, the rates of over- and underdiagnosis, and the decision curve. Results: The 2 data sets used in this study contained 189,775 and 180,976 medical cases for hospitals M and H, respectively. Statistical analyses (Z test) revealed no significant difference (P>.05) between the AUROCs from the different years for all use cases and hospitals. For example, in hospital M, AKI did not show a significant difference between 2020 (AUROC=0.898) and 2021 (AUROC=0.907, Z=–1.171, P=.242). Similar results were observed in both hospitals and for all use cases (sepsis and delirium) when comparing all the different years. However, when evaluating the calibration curves at the 2 hospitals, model shifts were observed for the delirium and sepsis use cases but not for AKI. Additionally, to investigate the clinical utility of our models, we performed decision curve analysis (DCA) and compared the results across the different years. A pairwise nonparametric statistical comparison showed no differences in the net benefit at the probability thresholds of interest (P>.05). The comprehensive evaluations performed in this study ensured robust model performance of all the investigated models across the years. Moreover, neither performance deteriorations nor alert surges were observed during the COVID-19 pandemic period. Conclusions: Clinical risk prediction models were affected by the dynamic and continuous evolution of clinical practices and workflows. The performance of the models evaluated in this study appeared stable when assessed using AUROCs, showing no significant variations over the years. Additional model shift investigations suggested that a calibration shift was present for certain use cases (delirium and sepsis). However, these changes did not have any impact on the clinical utility of the models based on DCA. Consequently, it is crucial to closely monitor data changes and detect possible model shifts, along with their potential influence on clinical decision-making.