Eric Karl Oermann

Eric Karl Oermann

Auto-generated profile, based on publicly available information

Eric Karl Oermann

Center for Data Science, New York University, New York, USA

Research Focus & Metrics

Academic Career Length

17Years (2010-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
6 November 2018 in PLOS Medicine
Verified
Journal Article
Open access
PDF file
JZJohn R Zech
John R Zech
MBMarcus A Badgeley
Marcus A Badgeley
MLManway Liu
Manway Liu
ACAnthony B Costa
Anthony B Costa
JTJoseph J Titano
Joseph J Titano
EOEric Karl Oermann
Eric Karl Oermann
There is interest in using convolutional neural networks (CNNs) to analyze medical imaging to provide computer-aided diagnosis (CAD). Recent work has suggested that image classification CNNs may not generalize to new data as well as previously believed. We assessed how well CNNs generalized across three hospital systems for a simulated pneumonia screening task. A cross-sectional design with multiple model training cohorts was used to evaluate model generalizability to external sites using split-sample validation. A total of 158,323 chest radiographs were drawn from three institutions: National Institutes of Health Clinical Center (NIH; 112,120 from 30,805 patients), Mount Sinai Hospital (MSH; 42,396 from 12,904 patients), and Indiana University Network for Patient Care (IU; 3,807 from 3,683 patients). These patient populations had an age mean (SD) of 46.9 years (16.6), 63.2 years (16.5), and 49.6 years (17) with a female percentage of 43.5%, 44.8%, and 57.3%, respectively. We assessed individual models using the area under the receiver operating characteristic curve (AUC) for radiographic findings consistent with pneumonia and compared performance on different test sets with DeLong’s test. The prevalence of pneumonia was high enough at MSH (34.2%) relative to NIH and IU (1.2% and 1.0%) that merely sorting by hospital system achieved an AUC of 0.861 (95% CI 0.855–0.866) on the joint MSH–NIH dataset. Models trained on data from either NIH or MSH had equivalent performance on IU (P values 0.580 and 0.273, respectively) and inferior performance on data from each other relative to an internal test set (i.e., new data from within the hospital system used for training data; P values both <0.001). The highest internal performance was achieved by combining training and test data from MSH and NIH (AUC 0.931, 95% CI 0.927–0.936), but this model demonstrated significantly lower external performance at IU (AUC 0.815, 95% CI 0.745–0.885, P = 0.001). To test the effect of pooling data from sites with disparate pneumonia prevalence, we used stratified subsampling to generate MSH–NIH cohorts that only differed in disease prevalence between training data sites. When both training data sites had the same pneumonia prevalence, the model performed consistently on external IU data (P = 0.88). When a 10-fold difference in pneumonia rate was introduced between sites, internal test performance improved compared to the balanced model (10× MSH risk P < 0.001; 10× NIH P = 0.002), but this outperformance failed to generalize to IU (MSH 10× P < 0.001; NIH 10× P = 0.027). CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs. The primary limitation of our approach and the available public data is that we cannot fully assess what other factors might be contributing to hospital system–specific biases. Pneumonia-screening CNNs achieved better internal than external performance in 3 out of 5 natural comparisons. When models were trained on pooled data from sites with different pneumonia prevalence, they performed better on new pooled data from these sites but not on external data. CNNs robustly identified hospital system and department within a hospital, which can have large differences in disease burden and may confound predictions.

7 June 2023 in Nature
Verified
Journal Article
Open access
PDF file
LJLavender Yao Jiang
Lavender Yao Jiang
XLXujin Chris Liu
Xujin Chris Liu
NNNima Pour Nejatian
Nima Pour Nejatian
MNMustafa Nasir-Moin
Mustafa Nasir-Moin
DWDuo Wang
Duo Wang
EOEric Karl Oermann
Eric Karl Oermann
Physicians make critical time-constrained decisions every day. Clinical predictive models can help physicians and administrators make decisions by forecasting clinical and operational events. Existing structured data-based clinical predictive models have limited use in everyday practice owing to complexity in data processing, as well as model development and deployment1–3. Here we show that unstructured clinical notes from the electronic health record can enable the training of clinical language models, which can be used as all-purpose clinical predictive engines with low-resistance development and deployment. Our approach leverages recent advances in natural language processing4,5to train a large language model for medical language (NYUTron) and subsequently fine-tune it across a wide range of clinical and operational predictive tasks. We evaluated our approach within our health system for five such tasks: 30-day all-cause readmission prediction, in-hospital mortality prediction, comorbidity index prediction, length of stay prediction, and insurance denial prediction. We show that NYUTron has an area under the curve (AUC) of 78.7–94.9%, with an improvement of 5.36–14.7% in the AUC compared with traditional models. We additionally demonstrate the benefits of pretraining with clinical text, the potential for increasing generalizability to different sites through fine-tuning and the full deployment of our system in a prospective, single-arm trial. These results show the potential for using clinical language models in medicine to read alongside physicians and provide guidance at the point of care.

13 August 2018 in Nature Medicine
Verified
Journal Article
Closed access
PDF file
JTJoseph J. Titano
Joseph J. Titano
MBMarcus Badgeley
Marcus Badgeley
JSJavin Schefflein
Javin Schefflein
MPMargaret Pain
Margaret Pain
ASAndres Su
Andres Su
EOEric K. Oermann
Eric K. Oermann

8 January 2025 in Nature Medicine
Verified
Journal Article
Open access
PDF file
DADaniel Alexander Alber
Daniel Alexander Alber
ZYZihao Yang
Zihao Yang
AAAnton Alyakin
Anton Alyakin
EYEunice Yang
Eunice Yang
SRSumedha Rai
Sumedha Rai
EOEric Karl Oermann
Eric Karl Oermann
The adoption of large language models (LLMs) in healthcare demands a careful analysis of their potential to spread false medical knowledge. Because LLMs ingest massive volumes of data from the open Internet during training, they are potentially exposed to unverified medical knowledge that may include deliberately planted misinformation. Here, we perform a threat assessment that simulates a data-poisoning attack against The Pile, a popular dataset used for LLM development. We find that replacement of just 0.001% of training tokens with medical misinformation results in harmful models more likely to propagate medical errors. Furthermore, we discover that corrupted models match the performance of their corruption-free counterparts on open-source benchmarks routinely used to evaluate medical LLMs. Using biomedical knowledge graphs to screen medical LLM outputs, we propose a harm mitigation strategy that captures 91.9% of harmful content (F1 = 85.7%). Our algorithm provides a unique method to validate stochastically generated LLM outputs against hard-coded relationships in knowledge graphs. In view of current calls for improved data provenance and transparent LLM development, we hope to raise awareness of emergent risks from LLMs trained indiscriminately on web-scraped data, particularly in healthcare where misinformation can potentially compromise patient safety.

13 March 2013 in Radiation Oncology
Verified
Journal Article
Open access
PDF file
LCLeonard N Chen
Leonard N Chen
SSSimeng Suy
Simeng Suy
SUSunghae Uhm
Sunghae Uhm
EOEric K Oermann
Eric K Oermann
AJAndrew W Ju
Andrew W Ju
SCSean Philip Collins
Sean Philip Collins
Background: Stereotactic body radiation therapy (SBRT) delivers fewer high-dose fractions of radiation which may be radiobiologically favorable to conventional low-dose fractions commonly used for prostate cancer radiotherapy. We report our early experience using SBRT for localized prostate cancer. Methods: Patients treated with SBRT from June 2008 to May 2010 at Georgetown University Hospital for localized prostate carcinoma, with or without the use of androgen deprivation therapy (ADT), were included in this retrospective review of data that was prospectively collected in an institutional database. Treatment was delivered using the CyberKnife® with doses of 35 Gy or 36.25 Gy in 5 fractions. Biochemical control was assessed using the Phoenix definition. Toxicities were recorded and scored using the CTCAE v.3. Quality of life was assessed before and after treatment using the Short Form-12 Health Survey (SF-12), the American Urological Association Symptom Score (AUA) and Sexual Health Inventory for Men (SHIM) questionnaires. Late urinary symptom flare was defined as an AUA score ≥ 15 with an increase of ≥ 5 points above baseline six months after the completion of SBRT. Results: One hundred patients (37 low-, 55 intermediate- and 8 high-risk according to the D’Amico classification) at a median age of 69 years (range, 48–90 years) received SBRT, with 11 patients receiving ADT. The median pre-treatment prostate-specific antigen (PSA) was 6.2 ng/ml (range, 1.9-31.6 ng/ml) and the median follow-up was 2.3 years (range, 1.4-3.5 years). At 2 years, median PSA decreased to 0.49 ng/ml (range, 0.1-1.9 ng/ml). Benign PSA bounce occurred in 31% of patients. There was one biochemical failure in a high-risk patient, yielding a two-year actuarial biochemical relapse free survival of 99%. The 2-year actuarial incidence rates of GI and GU toxicity ≥ grade 2 were 1% and 31%, respectively. A median baseline AUA symptom score of 8 significantly increased to 11 at 1 month (p = 0.001), however returned to baseline at 3 months (p = 0.60). Twenty one percent of patients experienced a late transient urinary symptom flare in the first two years following treatment. Of patients who were sexually potent prior to treatment, 79% maintained potency at 2 years post-treatment. Conclusions: SBRT for clinically localized prostate cancer was well tolerated, with an early biochemical response similar to other radiation therapy treatments. Benign PSA bounces were common. Late GI and GU toxicity rates were comparable to conventionally fractionated radiation therapy and brachytherapy. Late urinary symptom flares were observed but the majority resolved with conservative management. A high percentage of men who were potent prior to treatment remained potent two years following treatment.

11 December 2024 in Nature Medicine
Verified
Journal Article
Closed access
PDF file
JWJinzhuo Wang
Jinzhuo Wang
KWKai Wang
Kai Wang
YYYunfang Yu
Yunfang Yu
YLYuxing Lu
Yuxing Lu
WXWenchao Xiao
Wenchao Xiao
JQJia Qu
Jia Qu