John Caskey

John Caskey

Auto-generated profile, based on publicly available information

John Caskey

Department of Medicine, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
24 February 2025 in JMIR AI
Verified
Journal Article
Open access
YGYanjun Gao
Yanjun Gao
RLRuizhe Li
Ruizhe Li
ECEmma Croxford
Emma Croxford
JCJohn Caskey
John Caskey
BPBrian W Patterson
Brian W Patterson
MAMajid Afshar
Majid Afshar
Background: Electronic health records (EHRs) and routine documentation practices play a vital role in patients’ daily care, providing a holistic record of health, diagnoses, and treatment. However, complex and verbose EHR narratives can overwhelm health care providers, increasing the risk of diagnostic inaccuracies. While large language models (LLMs) have showcased their potential in diverse language tasks, their application in health care must prioritize the minimization of diagnostic errors and the prevention of patient harm. Integrating knowledge graphs (KGs) into LLMs offers a promising approach because structured knowledge from KGs could enhance LLMs’ diagnostic reasoning by providing contextually relevant medical information. Objective: This study introduces DR.KNOWS (Diagnostic Reasoning Knowledge Graph System), a model that integrates Unified Medical Language System–based KGs with LLMs to improve diagnostic predictions from EHR data by retrieving contextually relevant paths aligned with patient-specific information. Methods: DR.KNOWS combines a stack graph isomorphism network for node embedding with an attention-based path ranker to identify and rank knowledge paths relevant to a patient’s clinical context. We evaluated DR.KNOWS on 2 real-world EHR datasets from different geographic locations, comparing its performance to baseline models, including QuickUMLS and standard LLMs (Text-to-Text Transfer Transformer and ChatGPT). To assess diagnostic reasoning quality, we designed and implemented a human evaluation framework grounded in clinical safety metrics. Results: DR.KNOWS demonstrated notable improvements over baseline models, showing higher accuracy in extracting diagnostic concepts and enhanced diagnostic prediction metrics. Prompt-based fine-tuning of Text-to-Text Transfer Transformer with DR.KNOWS knowledge paths achieved the highest ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation–Longest Common Subsequence) and concept unique identifier F1-scores, highlighting the benefits of KG integration. Human evaluators found the diagnostic rationales of DR.KNOWS to be aligned strongly with correct clinical reasoning, indicating improved abstraction and reasoning. Recognized limitations include potential biases within the KG data, which we addressed by emphasizing case-specific path selection and proposing future bias-mitigation strategies. Conclusions: DR.KNOWS offers a robust approach for enhancing diagnostic accuracy and reasoning by integrating structured KG knowledge into LLM-based clinical workflows. Although further work is required to address KG biases and extend generalizability, DR.KNOWS represents progress toward trustworthy artificial intelligence–driven clinical decision support, with a human evaluation framework focused on diagnostic safety and alignment with clinical standards.

25 January 2023 in Journal of Biomedical Informatics
Verified
Journal Article
Closed access
YGYanjun Gao
Yanjun Gao
DDDmitriy Dligach
Dmitriy Dligach
TMTimothy Miller
Timothy Miller
JCJohn Caskey
John Caskey
BSBrihat Sharma
Brihat Sharma
MAMajid Afshar
Majid Afshar

22 April 2025 in medRxiv
Verified
Preprint
Open access
PDF file
ECEmma Croxford
Emma Croxford
YGYanjun Gao
Yanjun Gao
EFElliot First
Elliot First
NPNicholas Pellegrino
Nicholas Pellegrino
MSMiranda Schnier
Miranda Schnier
MAMajid Afshar
Majid Afshar
Electronic Health Records (EHRs) store vast amounts of clinical information that are difficult for healthcare providers to summarize and synthesize relevant details to their practice.12345 To reduce cognitive load on providers, generative AI with Large Language Models have emerged to automatically summarize patient records into clear, actionable insights and offload the cognitive burden for providers.678 However, LLM summaries need to be precise and free from errors, making evaluations on the quality of the summaries necessary.9101112 While human experts are the gold standard for evaluations, their involvement is time-consuming and costly.13 Therefore, we introduce and validate an automated method for evaluating real-world EHR multi-document summaries using an LLM as the evaluator, referred to as LLM-as-a-Judge. Benchmarking against the validated Provider Documentation Summarization Quality Instrument (PDSQI)-9 for human evaluation14, our LLM-as-a-Judge framework uses the PDSQI-9 rubric and demonstrated strong inter-rater reliability with human evaluators. GPT-o3-mini achieved the highest intraclass correlation coefficient of 0.818 (95% CI 0.772, 0.854), with a median score difference of 0 from human evaluators, and completes evaluations in just 22 seconds. Overall, the reasoning models excelled in inter-rater reliability, particularly in evaluations that require advanced reasoning and domain expertise, outperforming non-reasoning models, those trained on the task, and multi-agent workflows. Cross-task validation on the Problem Summarization15 task similarly confirmed high reliability. By automating high-quality evaluations, medical LLM-as-a-Judge offers a scalable, efficient solution to rapidly identify accurate and safe AI-generated summaries in healthcare settings.

18 July 2019 in Immunogenetics
Verified
Journal Article
Closed access
PDF file
JCJohn R. Caskey
John R. Caskey
RWRoger W. Wiseman
Roger W. Wiseman
JKJulie A. Karl
Julie A. Karl
DBDavid A. Baker
David A. Baker
TLTaylor Lee
Taylor Lee
DODavid O'Connor
David O'Connor

4 October 2023 in JAMIA Open
Verified
Journal Article
Open access
PDF file
GKGeorge K Karway
George K Karway
JKJay L Koyner
Jay L Koyner
JCJohn Caskey
John Caskey
ASAlexandra B Spicer
Alexandra B Spicer
KCKyle A Carey
Kyle A Carey
MCMatthew M Churpek
Matthew M Churpek
Objectives: To develop and externally validate machine learning models using structured and unstructured electronic health record data to predict postoperative acute kidney injury (AKI) across inpatient settings. Materials and Methods: Data for adult postoperative admissions to the Loyola University Medical Center (2009-2017) were used for model development and admissions to the University of Wisconsin-Madison (2009-2020) were used for validation. Structured features included demographics, vital signs, laboratory results, and nurse-documented scores. Unstructured text from clinical notes were converted into concept unique identifiers (CUIs) using the clinical Text Analysis and Knowledge Extraction System. The primary outcome was the development of Kidney Disease Improvement Global Outcomes stage 2 AKI within 7 days after leaving the operating room. We derived unimodal extreme gradient boosting machines (XGBoost) and elastic net logistic regression (GLMNET) models using structured-only data and multimodal models combining structured data with CUI features. Model comparison was performed using the receiver operating characteristic curve (AUROC), with Delong’s test for statistical differences. Results: The study cohort included 138 389 adult patient admissions (mean [SD] age 58 [16] years; 11 506 [8%] African-American; and 70 826 [51%] female) across the 2 sites. Of those, 2959 (2.1%) developed stage 2 AKI or higher. Across all data types, XGBoost outperformed GLMNET (mean AUROC 0.81 [95% confidence interval (CI), 0.80-0.82] vs 0.78 [95% CI, 0.77-0.79]). The multimodal XGBoost model incorporating CUIs parameterized as term frequency-inverse document frequency (TF-IDF) showed the highest discrimination performance (AUROC 0.82 [95% CI, 0.81-0.83]) over unimodal models (AUROC 0.79 [95% CI, 0.78-0.80]). Discussion: A multimodality approach with structured data and TF-IDF weighting of CUIs increased model performance over structured data-only models. Conclusion: These findings highlight the predictive power of CUIs when merged with structured data for clinical prediction models, which may improve the detection of postoperative AKI.

15 April 2025 in Clinical Journal of the American Society of Nephrology
Verified
Journal Article
Closed access
JKJay L. Koyner
Jay L. Koyner
JMJennie Martin
Jennie Martin
KCKyle A. Carey
Kyle A. Carey
JCJohn Caskey
John Caskey
DEDana P. Edelson
Dana P. Edelson
MCMatthew M. Churpek
Matthew M. Churpek
Key Points: We developed and validated a multimodal (structured and unstructured data) model to predict moderate to severe AKI using multicenter data. This multimodal AKI risk score accurately identifies patients who will develop stage 2 AKI over 2 days earlier than serum creatinine alone. The multimodal model performed better than a model based solely on structured data and performed similarly during temporal and site-based validation. Background: Prior models for the early identification of AKI have used structured data (e.g., vital signs and laboratory values). We aimed to develop and validate a deep learning model to predict moderate to severe AKI by combining structured data and information from unstructured notes. Methods: Adults (18 years or older) admitted to the University of Wisconsin (2009–2020) and the University of Chicago Medicine (2016–2022) were eligible for inclusion. Patients were excluded if they had no documented serum creatinine (SCr), ESKD, an admission SCr ≥3.0 mg/dl, developed ≥stage 2 AKI before reaching the wards or intensive care unit, or required dialysis (KRT) within the first 48 hours. Text from unstructured notes was mapped to standardized concept unique identifiers to create predictor variables, and structured data variables were also included. An intermediate fusion deep learning recurrent neural network architecture was used to predict ≥stage 2 AKI within the next 48 hours. This multimodal model was developed in the first 80% of the data and temporally validated in the next 20%. Results: There were 339,998 admissions in the derivation cohort and 84,581 in the validation cohort, with 12,748 (3%) developing ≥stage 2 AKI. Patients with ≥stage 2 AKI were older, more likely to be male, had higher baseline SCr, and were more commonly in the intensive care unit (P < 0.001 for all). The multimodal model outperformed a model based only on structured data for all outcomes, with an area under the receiver operating characteristic curve (95% confidence interval) of 0.88 (0.88 to 0.88) for predicting ≥stage 2 AKI and 0.93 (0.93 to 0.94) for receiving KRT. The area under the precision-recall-curve for ≥stage 2 AKI was 0.20. The results were similar during external validation. Conclusions: We developed and validated a multimodal deep learning model using structured and unstructured data that predicts the development of severe AKI across the hospital stay for earlier intervention.