6 March 2024 in Journal of the American Medical Informatics Association
Objective: This study evaluates an AI assistant developed using OpenAI’s GPT-4 for interpreting pharmacogenomic (PGx) testing results, aiming to improve decision-making and knowledge sharing in clinical genetics and to enhance patient care with equitable access. Materials and Methods: The AI assistant employs retrieval-augmented generation (RAG), which combines retrieval and generative techniques, by harnessing a knowledge base (KB) that comprises data from the Clinical Pharmacogenetics Implementation Consortium (CPIC). It uses context-aware GPT-4 to generate tailored responses to user queries from this KB, further refined through prompt engineering and guardrails. Results: Evaluated against a specialized PGx question catalog, the AI assistant showed high efficacy in addressing user queries. Compared with OpenAI’s ChatGPT 3.5, it demonstrated better performance, especially in provider-specific queries requiring specialized data and citations. Key areas for improvement include enhancing accuracy, relevancy, and representative language in responses. Discussion: The integration of context-aware GPT-4 with RAG significantly enhanced the AI assistant’s utility. RAG’s ability to incorporate domain-specific CPIC data, including recent literature, proved beneficial. Challenges persist, such as the need for specialized genetic/PGx models to improve accuracy and relevancy and addressing ethical, regulatory, and safety concerns. Conclusion: This study underscores generative AI’s potential for transforming healthcare provider support and patient accessibility to complex pharmacogenomic information. While careful implementation of large language models like GPT-4 is necessary, it is clear that they can substantially improve understanding of pharmacogenomic data. With further development, these tools could augment healthcare expertise, provider productivity, and the delivery of equitable, patient-centered healthcare services.
28 April 2021 in Journal of Biomedical Informatics
27 February 2024 in medRxiv
Objective This study evaluates an AI assistant developed using OpenAI’s GPT-4 for interpreting pharmacogenomic (PGx) testing results, aiming to improve decision-making and knowledge sharing in clinical genetics, and to enhance patient care with equitable access. Methods The AI assistant employs Retrieval Augmented Generation (RAG) combining retrieval and generative techniques. It employs a Knowledge Base (KB) comprising Clinical Pharmacogenetics Implementation Consortium (CPIC) data, with context-aware GPT-4 generating tailored responses to user queries from this KB, refined through prompt engineering and guardrails. Results Evaluated against a specialized PGx question catalog, the AI assistant showed high efficacy in addressing user queries. Compared with OpenAI’s ChatGPT 3.5, it demonstrated better performance, especially in provider-specific queries requiring specialized data and citations. Key areas for improvement include enhancing accuracy, relevancy, and representative language in responses. Discussion The integration of context-aware GPT-4 with RAG significantly enhanced the AI assistant’s utility. RAG’s ability to incorporate domain-specific CPIC data, including recent literature, proved beneficial. Challenges persist, such as the need for specialized genetic/PGx models to improve accuracy and relevancy and addressing ethical, regulatory, and safety concerns. Conclusion This study underscores generative AI’s potential for transforming healthcare provider support and patient accessibility to complex pharmacogenomic information. While careful implementation of large language models like GPT-4 is necessary, it is clear that they can substantially improve understanding of pharmacogenomic data. With further development, these tools could augment healthcare expertise, provider productivity, and the delivery of equitable, patient-centered healthcare services.
19 February 2026 in medRxiv
BACKGROUND: Genetic variant curation, an important step in the implementation of Genomic Medicine, requires literature-guided comparison of variant prevalence in affected individuals versus healthy controls. This evidence is categorized as the PS4 evidence code by the AMP/ACMG variant interpretation guidelines and its manual extraction is a major bottleneck in clinical variant curation. This study aimed to evaluate whether reasoning-capable large language models (LLMs) can support guideline-constrained PS4 evidence extraction from literature. METHODS: We benchmarked five LLMs for publication-level variant detection and PS4-eligible proband counting under ACMG/AMP and ClinGen Variant Curation Expert Panel (VCEP) guidance using an expert-curated truth-set. We assembled an expert-curated truth-set of 281 publication-variant pairs from 275 peer-reviewed publications (58 genes and 128 variants). Five LLMs spanning frontier-scale, reasoning-optimized, and efficiency-oriented classes (Gemini 2.5 Pro, GPT-5, o3, o4-mini, and Claude Sonnet 4) were evaluated against this truth-set using identical inputs, a unified prompt template, and a schema-constrained JSON output format on two tasks: (1) determining whether a prespecified variant was correctly identified and (2) counting independent PS4-eligible probands under applicable guidance. Primary outcomes were Task 1 accuracy and Task 2 exact-count concordance (model PS4 count equals truth-set count). We also assessed prompt sensitivity, error modes, and output variability across models. RESULTS: Models were able to detect the presence of a variant in a publication with high accuracy (93.6-97.9%). For PS4 case counting, exact-count concordance was highest for Gemini 2.5 Pro (91.1%) and GPT-5 (90.0%), followed by o3 (86.5%), o4-mini (79.4%), and Claude Sonnet 4 (73.0%). Most counting errors resulted from an inability of a model to correctly apply guidelines, including evaluating phenotype and family structure. Prompt refinements improved concordance for most models but reduced performance for Claude Sonnet 4, indicating model specific prompting may be warranted. CONCLUSIONS: Reasoning-capable LLMs can support automation of guideline-based PS4 evidence extraction, achieving high concordance with expert curation, but performance is model- and prompt-dependent and failures concentrate in applying guidelines. Our findings support a hybrid workflow for clinical use in which LLM outputs accelerate evidence extraction with expert escalation.
1 February 2021 in bioRxiv
Structured representation of clinical genetic results is necessary for advancing precision medicine. The Electronic Medical Records and Genomics (eMERGE) Network’s Phase III program initially used a commercially developed XML message format for standardized and structured representation of genetic results for electronic health record (EHR) integration. In a desire to move towards a standard representation, the network created a new standardized format based upon Health Level Seven Fast Healthcare Interoperability Resources (HL7 FHIR), to represent clinical genomics results. These new standards improve the utility of HL7 FHIR as an international healthcare interoperability standard for management of genetic data from patients. This work advances the establishment of standards that are being designed for broad adoption in the current health information technology landscape.