Many previous studies claim to have developed machine learning models that diagnose COVID-19 from blood tests. However, we hypothesize that changes in the underlying distribution of the data, so called domain shifts, affect the predictive performance and reliability and are a reason for the failure of such machine learning models in clinical application. Domain shifts can be caused, e.g., by changes in the disease prevalence (spreading or tested population), by refined RT-PCR testing procedures (way of taking samples, laboratory procedures), or by virus mutations. Therefore, machine learning models for diagnosing COVID-19 or other diseases may not be reliable and degrade in performance over time. We investigate whether domain shifts are present in COVID-19 datasets and how they affect machine learning methods. We further set out to estimate the mortality risk based on routinely acquired blood tests in a hospital setting throughout pandemics and under domain shifts. We reveal domain shifts by evaluating the models on a large-scale dataset with different assessment strategies, such as temporal validation. We present the novel finding that domain shifts strongly affect machine learning models for COVID-19 diagnosis and deteriorate their predictive performance and credibility. Therefore, frequent re-training and re-assessment are indispensable for robust models enabling clinical utility.

Objective: The diagnosis of COVID-19 is based on the detection of SARS-CoV-2 in respiratory secretions, blood, or stool. Currently, reverse transcription polymerase chain reaction (RT-PCR) is the most commonly used method to test for SARS-CoV-2. Methods: In this retrospective cohort analysis, we evaluated whether machine learning could exclude SARS-CoV-2 infection using routinely available laboratory values. A Random Forests algorithm with 28 unique features was trained to predict the RT-PCR results. Results: Out of 12,848 patients undergoing SARS-CoV-2 testing, routine blood tests were simultaneously performed in 1357 patients. The machine learning model could predict SARS-CoV-2 test results with an accuracy of 86% and an area under the receiver operating characteristic curve of 0.74. Conclusion: Machine learning methods can reliably predict a negative SARS-CoV-2 RT-PCR test result using standard blood tests.

Electronic health records (EHRs) have been successfully used in data science and machine learning projects. However, most of these data are collected for clinical use rather than for retrospective analysis. This means that researchers typically face many different issues when attempting to access and prepare the data for secondary use. We aimed to investigate how raw EHRs can be accessed and prepared in retrospective data science projects in a disciplined, effective, and efficient way. We report our experience and findings from a large-scale data science project analyzing routinely acquired retrospective data from the Kepler University Hospital in Linz, Austria. The project involved data collection from more than 150,000 patients over a period of 10 years. It included diverse data modalities, such as static demographic data, irregularly acquired laboratory test results, regularly sampled vital signs, and high-frequency physiological waveform signals. Raw medical data can be corrupted in many unexpected ways that demand thorough manual inspection and highly individualized data cleaning solutions. We present a general data preparation workflow, which was shaped in the course of our project and consists of the following 7 steps: obtain a rough overview of the available EHR data, define clinically meaningful labels for supervised learning, extract relevant data from the hospital’s data warehouses, match data extracted from different sources, deidentify them, detect errors and inconsistencies therein through a careful exploratory analysis, and implement a suitable data processing pipeline in actual code. Only few of the data preparation issues encountered in our project were addressed by generic medical data preprocessing tools that have been proposed recently. Instead, highly individualized solutions for the specific data used in one’s own research seem inevitable. We believe that the proposed workflow can serve as a guidance for practitioners, helping them to identify and address potential problems early and avoid some common pitfalls.

The first clinical impression of emergency patients conveys a myriad of information that has been incompletely elucidated. In this prospective, observational study, the value of the first clinical impression, assessed by 18 observations, to predict the need for timely medical attention, the need for hospital admission, and in-hospital mortality in 1506 adult patients presenting to the triage desk of an emergency department was determined. Machine learning models were used for statistical analysis. The first clinical impression could predict the need for timely medical attention [area under the receiver operating characteristic curve (AUC ROC), 0.73; p = 0.01] and hospital admission (AUC ROC, 0.8; p = 0.004), but not in-hospital mortality (AUC ROC, 0.72; p = 0.13). The five most important features informing the prediction models were age, ability to walk, admission by emergency medical services, lying on a stretcher, breathing pattern, and bringing a suitcase. The inability to walk at triage presentation was highly predictive of both the need for timely medical attention (p < 0.001) and the need for hospital admission (p < 0.001). In conclusion, the first clinical impression of emergency patients presenting to the triage desk can predict the need for timely medical attention and hospital admission. Important components of the first clinical impression were identified.

Background and importance Guidelines recommend that hospital emergency teams locally validate criteria for termination of cardiopulmonary resuscitation in patients with in-hospital cardiac arrest (IHCA). Objective To determine the value of a machine learning algorithm to predict failure to achieve return of spontaneous circulation (ROSC) and unfavourable functional outcome from IHCA using only data readily available at emergency team arrival. Design Retrospective cohort study. Setting and participants Adults who experienced an IHCA were attended to by the emergency team. Outcome measures and analysis Demographic and clinical data typically available at the arrival of the emergency team were extracted from the institutional IHCA database. In addition, outcome data including the Cerebral Performance Category (CPC) score count at hospital discharge were collected. A model selection procedure for random forests with a hyperparameter search was employed to develop two classification algorithms to predict failure to achieve ROSC and unfavourable (CPC 3–5) functional outcomes. Main results Six hundred thirty patients were included, of which 390 failed to achieve ROSC (61.9%). The final classification model to predict failure to achieve ROSC had an area under the receiver operating characteristic curve of 0.9 [95% confidence interval (CI), 0.89–0.9], a balanced accuracy of 0.77 (95% CI, 0.75–0.79), an F1-score of 0.78 (95% CI, 0.76–0.79), a positive predictive value of 0.88 (0.86–0.91), a negative predictive value of 0.61 (0.6–0.63), a sensitivity of 0.69 (0.66–0.72), and a specificity of 0.84 (0.8–0.88). Five hundred fifty-nine subjects experienced an unfavourable outcome (88.7%). The final classification model to predict unfavourable functional outcomes from IHCA at hospital discharge had an area under the receiver operating characteristic curve of 0.93 (95% CI, 0.92–0.93), a balanced accuracy of 0.59 (95% CI, 0.57–0.61), an F1-score of 0.94 (95% CI, 0.94–0.95), a positive predictive value of 0.91 (0.9–0.91), a negative predictive value of 0.57 (0.48–0.66), a sensitivity of 0.98 (0.97–0.99), and a specificity of 0.2 (0.16–0.24). Conclusion Using data readily available at emergency team arrival, machine learning algorithms had a high predictive power to forecast failure to achieve ROSC and unfavourable functional outcomes from IHCA while cardiopulmonary resuscitation was still ongoing; however, the positive predictive value of both models was not high enough to allow for early termination of resuscitation efforts.

Visually evoked potentials (VEPs) are widely used for diagnoses of different neurological diseases. Interestingly, there is limited research about the impact of the stimulus color onto the evoked response. Therefore, in our study we investigated the possibility of automatically classifying the stimulus color. The visual stimuli were selected to be red/black and green/black checkerboard patterns with equal light density. Both of these stimuli were presented in a random manner to nine subjects, while the electroencephalogram was recorded at the occipital lobe. After pre-processing and aligning the evoked potentials, an artificial neural network with one hidden layer was used to investigate the general possibility to automatically classify the stimulus color in three different settings. First, color classification with individually trained models, color classification with a common model, and color classification for each individual volunteer with a model trained on the data of the remaining subjects. With an average accuracy (ACC) of 0.83, the best results were achieved for the individually trained model. Also, the second (mean ACC = 0.76) and third experiments (mean ACC = 0.71) indicated a reasonable predictive accuracy across all subjects. Consequently, machine learning tools are able to appropriately classify stimuli colors based on VEPs. Although further studies are needed to improve the classification performance of our approach, this opens new fields of applications for VEPs.