7 March 2024 in Hepatology
Background: Large language models (LLMs) have significant capabilities in clinical information processing tasks. Commercially available LLMs, however, are not optimized for clinical uses and are prone to generating hallucinatory information. Retrieval-augmented generation (RAG) is an enterprise architecture that allows embedding of customized data into LLMs. This approach “specializes” the LLMs and is thought to reduce hallucinations. Methods: We developed “LiVersa,” a liver disease-specific LLM, by using our institution’s protected health information (PHI)-complaint text embedding and LLM platform, “Versa.” We conducted RAG on 30 publicly available American Association for the Study of Liver Diseases guidance documents to be incorporated into LiVersa. Results: We evaluated LiVersa’s performance by conducting two rounds of testing. First, we compared LiVersa’s outputs versus those of trainees from a previously published knowledge assessment. LiVersa answered all 10 questions correctly. Second, we asked 15 hepatologists to evaluate the outputs to ten hepatology topic questions generated by LiVersa, OpenAI’s ChatGPT 4, and Meta’s LLaMA 2. LiVersa’s outputs were more accurate but were rated less comprehensive and safe compared to those of ChatGPT 4. Discussion: In this demonstration, we built a disease-specific and PHI-compliant LLMs using RAG. While LiVersa demonstrated higher accuracy in answering questions related to hepatology – there were some deficiencies due to limitations set by the number of documents used for RAG. LiVersa will likely require further refinement before potential live deployment. The LiVersa prototype, however, is a proof of concept for utilizing RAG to customize LLMs for clinical use cases. Export
1 November 2020 in JCO Clinical Cancer Informatics
PURPOSEThe Electronic Medical Record Search Engine (EMERSE) is a software tool built to aid research spanning cohort discovery, population health, and data abstraction for clinical trials. EMERSE i...
8 October 2024 in Hepatology
Background: Diagnosis code classification is a common method for cohort identification in cirrhosis research, but it is often inaccurate and augmented by labor-intensive chart review. Natural language processing (NLP) using large language models (LLMs) is a potentially more accurate method. To assess LLMs’ potential for cirrhosis cohort identification, we compared code-based versus LLM-based classification with chart review as a “gold standard.” Methods: We extracted and conducted a limited chart review of 3,788 discharge summaries of cirrhosis admissions. We engineered zero-shot prompts using Generative Pre-trained Transformer (GPT)-4 to determine whether cirrhosis and its complications were active hospitalization problems. We calculated positive predictive values (PPVs) of LLM-based classification versus limited chart review, and PPVs of code-based versus LLM-based classification as a “silver standard” in all 3,788 summaries. Results: Versus gold standard chart review, code-based classification achieved PPVs of 82.2% for identifying cirrhosis, 41.7% hepatic encephalopathy, 72.8% ascites, 59.8% gastrointestinal bleeding, and 48.8% spontaneous bacterial peritonitis. Compared to chart review, GPT-4 achieved 87.8-98.8% accuracies for identifying cirrhosis and its complications. Using LLM as a silver standard, code-based classification achieved PPVs of 79.8% for identifying cirrhosis, 53.9% hepatic encephalopathy, 55.3% ascites, 67.6% gastrointestinal bleeding, and 65.5% spontaneous bacterial peritonitis. Conclusions: LLM-based classification was highly accurate versus manual chart review in identifying cirrhosis and its complications – this allowed us to assess the performance of code-based classification at scale using LLMs as a silver standard. These results suggest LLMs could augment or replace code-based cohort classification and raise questions regarding the necessity of chart review. Export
27 June 2024 in Hepatology
An abstract is unavailable. This article is available as a PDF only.