24 October 2024 in NEJM AI
Hospital quality measures are a vital component of a learning health system, yet they can be costly to report, statistically underpowered, and inconsistent due to poor interrater reliability. Large language models (LLMs) have recently demonstrated impressive performance on health care–related tasks and offer a promising way to provide accurate abstraction of complete charts at scale. To evaluate this approach, we deployed an LLM-based system that ingests Fast Healthcare Interoperability Resources data and outputs a completed Severe Sepsis and Septic Shock Management Bundle (SEP-1) abstraction. We tested the system on a sample of 100 manual SEP-1 abstractions that University of California San Diego Health reported to the Centers for Medicare & Medicaid Services in 2022. The LLM system achieved agreement with manual abstractors on the measure category assignment in 90 of the abstractions (90%; κ=0.82; 95% confidence interval, 0.71 to 0.92). Expert review of the 10 discordant cases identified four that were mistakes introduced by manual abstraction. This pilot study suggests that LLMs using interoperable electronic health record data may perform accurate abstractions for complex quality measures. (Funded by the National Institute of Allergy and Infectious Diseases [1R42AI177108–1] and others.)
18 December 2017 in Journal of Medical Internet Research
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet
5 November 2020 in JACEP Open
Objective The coronavirus disease 2019 pandemic has inspired new innovations in diagnosing, treating, and dispositioning patients during high census conditions with constrained resources. Our objective is to describe first experiences of physician interaction with a novel artificial intelligence (AI) algorithm designed to enhance physician abilities to identify ground‐glass opacities and consolidation on chest radiographs. Methods During the first wave of the pandemic, we deployed a previously developed and validated deep‐learning AI algorithm for assisted interpretation of chest radiographs for use by physicians at an academic health system in Southern California. The algorithm overlays radiographs with “heat” maps that indicate pneumonia probability alongside standard chest radiographs at the point of care. Physicians were surveyed in real time regarding ease of use and impact on clinical decisionmaking. Results Of the 5125 total visits and 1960 chest radiographs obtained in the emergency department (ED) during the study period, 1855 were analyzed by the algorithm. Among these, emergency physicians were surveyed for their experiences on 202 radiographs. Overall, 86% either strongly agreed or somewhat agreed that the intervention was easy to use in their workflow. Of the respondents, 20% reported that the algorithm impacted clinical decisionmaking. Conclusions To our knowledge, this is the first published literature evaluating the impact of medical imaging AI on clinical decisionmaking in the emergency department setting. Urgent deployment of a previously validated AI algorithm clinically was easy to use and was found to have an impact on clinical decision making during the predicted surge period of a global pandemic.
2 May 2024 in Journal of Medical Internet Research
This study explores the potential of using large language models to assist content analysis by conducting a case study to identify adverse events (AEs) in social media posts. The case study compares ChatGPT’s performance with human annotators’ in detecting AEs associated with delta-8-tetrahydrocannabinol, a cannabis-derived product. Using the identical instructions given to human annotators, ChatGPT closely approximated human results, with a high degree of agreement noted: 94.4% (9436/10,000) for any AE detection (Fleiss κ=0.95) and 99.3% (9931/10,000) for serious AEs (κ=0.96). These findings suggest that ChatGPT has the potential to replicate human annotation accurately and efficiently. The study recognizes possible limitations, including concerns about the generalizability due to ChatGPT’s training data, and prompts further research with different models, data sources, and content analysis tasks. The study highlights the promise of large language models for enhancing the efficiency of biomedical research.
26 August 2021 in JMIR Research Protocols
Background: Patient–physician communication during clinical encounters is essential to ensure quality of care. Many studies have attempted to improve patient–physician communication. Incorporating patient priorities into agenda setting and medical decision-making are fundamental to patient-centered communication. Efficient and scalable approaches are needed to empower patients to speak up and prepare physicians to respond. Leveraging electronic health records (EHRs) in engaging patients and health care teams has the potential to enhance the integration of patient priorities in clinical encounters. A systematic approach to eliciting and documenting patient priorities before encounters could facilitate effective communication in such encounters. Objective: In this paper, we report the design and implementation of a set of EHR tools built into clinical workflows for facilitating patient–physician joint agenda setting and the documentation of patient concerns in the EHRs for ambulatory encounters. Methods: We engaged health information technology leaders and users in three health care systems for developing and implementing a set of EHR tools. The goal of these tools is to standardize the elicitation of patient priorities by using a previsit “patient important issue” questionnaire distributed through the patient portal to the EHR. We built additional EHR documentation tools to facilitate patient–staff communication when the staff records the vital signs and the reason for the visit in the EHR while in the examination room, with a simple transmission method for physicians to incorporate patient concerns in EHR notes. Results: The study is ongoing. The anticipated completion date for survey data collection is November 2021. A total of 34,037 primary care patients from three health systems (n=26,441; n=5136; and n=2460 separately recruited from each system) used the previsit patient important issue questionnaire in 2020. The adoption of the digital previsit questionnaire during the COVID-19 pandemic was much higher in one health care system because it expanded the use of the questionnaire from physicians participating in trials to all primary care providers midway through the year. It also required the use of this previsit questionnaire for eCheck-ins, which are required for telehealth encounters. Physicians and staff suggested anecdotally that this questionnaire helped patient–clinician communication, particularly during the COVID-19 pandemic. Conclusions: EHR tools have the potential to facilitate the integration of patient priorities into agenda setting and documentation in real-world primary care practices. Early results suggest the feasibility and acceptability of such digital tools in three health systems. EHR tools can support patient engagement and clinicians’ work during in-person and telehealth visits. They could potentially exert a sustained influence on patient and clinician communication behaviors in contrast to prior ad hoc educational efforts targeting patients or clinicians. Trial Registration: ClinicalTrials.gov NCT03385512; https://clinicaltrials.gov/ct2/show/NCT03385512 International Registered Report Identifier (IRRID): DERR1-10.2196/30431