24 February 2025 in JMIR AI
Background: Electronic health records (EHRs) and routine documentation practices play a vital role in patients’ daily care, providing a holistic record of health, diagnoses, and treatment. However, complex and verbose EHR narratives can overwhelm health care providers, increasing the risk of diagnostic inaccuracies. While large language models (LLMs) have showcased their potential in diverse language tasks, their application in health care must prioritize the minimization of diagnostic errors and the prevention of patient harm. Integrating knowledge graphs (KGs) into LLMs offers a promising approach because structured knowledge from KGs could enhance LLMs’ diagnostic reasoning by providing contextually relevant medical information. Objective: This study introduces DR.KNOWS (Diagnostic Reasoning Knowledge Graph System), a model that integrates Unified Medical Language System–based KGs with LLMs to improve diagnostic predictions from EHR data by retrieving contextually relevant paths aligned with patient-specific information. Methods: DR.KNOWS combines a stack graph isomorphism network for node embedding with an attention-based path ranker to identify and rank knowledge paths relevant to a patient’s clinical context. We evaluated DR.KNOWS on 2 real-world EHR datasets from different geographic locations, comparing its performance to baseline models, including QuickUMLS and standard LLMs (Text-to-Text Transfer Transformer and ChatGPT). To assess diagnostic reasoning quality, we designed and implemented a human evaluation framework grounded in clinical safety metrics. Results: DR.KNOWS demonstrated notable improvements over baseline models, showing higher accuracy in extracting diagnostic concepts and enhanced diagnostic prediction metrics. Prompt-based fine-tuning of Text-to-Text Transfer Transformer with DR.KNOWS knowledge paths achieved the highest ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation–Longest Common Subsequence) and concept unique identifier F1-scores, highlighting the benefits of KG integration. Human evaluators found the diagnostic rationales of DR.KNOWS to be aligned strongly with correct clinical reasoning, indicating improved abstraction and reasoning. Recognized limitations include potential biases within the KG data, which we addressed by emphasizing case-specific path selection and proposing future bias-mitigation strategies. Conclusions: DR.KNOWS offers a robust approach for enhancing diagnostic accuracy and reasoning by integrating structured KG knowledge into LLM-based clinical workflows. Although further work is required to address KG biases and extend generalizability, DR.KNOWS represents progress toward trustworthy artificial intelligence–driven clinical decision support, with a human evaluation framework focused on diagnostic safety and alignment with clinical standards.
25 January 2023 in Journal of Biomedical Informatics
18 July 2019 in Immunogenetics
4 October 2023 in JAMIA Open
Objectives: To develop and externally validate machine learning models using structured and unstructured electronic health record data to predict postoperative acute kidney injury (AKI) across inpatient settings. Materials and Methods: Data for adult postoperative admissions to the Loyola University Medical Center (2009-2017) were used for model development and admissions to the University of Wisconsin-Madison (2009-2020) were used for validation. Structured features included demographics, vital signs, laboratory results, and nurse-documented scores. Unstructured text from clinical notes were converted into concept unique identifiers (CUIs) using the clinical Text Analysis and Knowledge Extraction System. The primary outcome was the development of Kidney Disease Improvement Global Outcomes stage 2 AKI within 7 days after leaving the operating room. We derived unimodal extreme gradient boosting machines (XGBoost) and elastic net logistic regression (GLMNET) models using structured-only data and multimodal models combining structured data with CUI features. Model comparison was performed using the receiver operating characteristic curve (AUROC), with Delong’s test for statistical differences. Results: The study cohort included 138 389 adult patient admissions (mean [SD] age 58 [16] years; 11 506 [8%] African-American; and 70 826 [51%] female) across the 2 sites. Of those, 2959 (2.1%) developed stage 2 AKI or higher. Across all data types, XGBoost outperformed GLMNET (mean AUROC 0.81 [95% confidence interval (CI), 0.80-0.82] vs 0.78 [95% CI, 0.77-0.79]). The multimodal XGBoost model incorporating CUIs parameterized as term frequency-inverse document frequency (TF-IDF) showed the highest discrimination performance (AUROC 0.82 [95% CI, 0.81-0.83]) over unimodal models (AUROC 0.79 [95% CI, 0.78-0.80]). Discussion: A multimodality approach with structured data and TF-IDF weighting of CUIs increased model performance over structured data-only models. Conclusion: These findings highlight the predictive power of CUIs when merged with structured data for clinical prediction models, which may improve the detection of postoperative AKI.
15 April 2025 in Clinical Journal of the American Society of Nephrology
Key Points: We developed and validated a multimodal (structured and unstructured data) model to predict moderate to severe AKI using multicenter data. This multimodal AKI risk score accurately identifies patients who will develop stage 2 AKI over 2 days earlier than serum creatinine alone. The multimodal model performed better than a model based solely on structured data and performed similarly during temporal and site-based validation. Background: Prior models for the early identification of AKI have used structured data (e.g., vital signs and laboratory values). We aimed to develop and validate a deep learning model to predict moderate to severe AKI by combining structured data and information from unstructured notes. Methods: Adults (18 years or older) admitted to the University of Wisconsin (2009–2020) and the University of Chicago Medicine (2016–2022) were eligible for inclusion. Patients were excluded if they had no documented serum creatinine (SCr), ESKD, an admission SCr ≥3.0 mg/dl, developed ≥stage 2 AKI before reaching the wards or intensive care unit, or required dialysis (KRT) within the first 48 hours. Text from unstructured notes was mapped to standardized concept unique identifiers to create predictor variables, and structured data variables were also included. An intermediate fusion deep learning recurrent neural network architecture was used to predict ≥stage 2 AKI within the next 48 hours. This multimodal model was developed in the first 80% of the data and temporally validated in the next 20%. Results: There were 339,998 admissions in the derivation cohort and 84,581 in the validation cohort, with 12,748 (3%) developing ≥stage 2 AKI. Patients with ≥stage 2 AKI were older, more likely to be male, had higher baseline SCr, and were more commonly in the intensive care unit (P < 0.001 for all). The multimodal model outperformed a model based only on structured data for all outcomes, with an area under the receiver operating characteristic curve (95% confidence interval) of 0.88 (0.88 to 0.88) for predicting ≥stage 2 AKI and 0.93 (0.93 to 0.94) for receiving KRT. The area under the precision-recall-curve for ≥stage 2 AKI was 0.20. The results were similar during external validation. Conclusions: We developed and validated a multimodal deep learning model using structured and unstructured data that predicts the development of severe AKI across the hospital stay for earlier intervention.