Hao Zhang

Hao Zhang

Auto-generated profile, based on publicly available information

Hao Zhang

Affiliation NYU Grossman School of Medicine, New York, New York, United States of America

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 July 2024 in JAMIA Open
Verified
Journal Article
Open access
PDF file
JYJie Yang
Jie Yang
JSJonathan So
Jonathan So
HZHao Zhang
Hao Zhang
SJSimon Jones
Simon Jones
DCDenise M Connolly
Denise M Connolly
VMVincent J Major
Vincent J Major
Objectives: Accelerating demand for patient messaging has impacted the practice of many providers. Messages are not recommended for urgent medical issues, but some do require rapid attention. This presents an opportunity for artificial intelligence (AI) methods to prioritize review of messages. Our study aimed to highlight some patient portal messages for prioritized review using a custom AI system integrated into the electronic health record (EHR). Materials and Methods: We developed a Bidirectional Encoder Representations from Transformers (BERT)-based large language model using 40 132 patient-sent messages to identify patterns involving high acuity topics that warrant an immediate callback. The model was then implemented into 2 shared pools of patient messages managed by dozens of registered nurses. A primary outcome, such as the time before messages were read, was evaluated with a difference-in-difference methodology. Results: Model validation on an expert-reviewed dataset (n = 7260) yielded very promising performance (C-statistic = 97%, average-precision = 72%). A binarized output (precision = 67%, sensitivity = 63%) was integrated into the EHR for 2 years. In a pre-post analysis (n = 396 466), an improvement exceeding the trend was observed in the time high-scoring messages sit unread (21 minutes, 63 vs 42 for messages sent outside business hours). Discussion: Our work shows great promise in improving care when AI is aligned with human workflow. Future work involves audience expansion, aiding users with suggested actions, and drafting responses. Conclusion: Many patients utilize patient portal messages, and while most messages are routine, a small fraction describe alarming symptoms. Our AI-based workflow shortens the turnaround time to get a trained clinician to review these messages to provide safer, higher-quality care.

11 December 2024 in PLOS Digital Health
Verified
Journal Article
Open access
PDF file
HZHao Zhang
Hao Zhang
NJNeil Jethani
Neil Jethani
SJSimon Jones
Simon Jones
NGNicholas Genes
Nicholas Genes
VMVincent J. Major
Vincent J. Major
NRNarges Razavian
Narges Razavian
Ensuring reliability of Large Language Models (LLMs) in clinical tasks is crucial. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss’ Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

1 May 2024 in JACC: Clinical Electrophysiology
Verified
Journal Article
Closed access
HZHao Zhang
Hao Zhang
CTConstantine Tarabanis
Constantine Tarabanis
NJNeil Jethani
Neil Jethani
MGMark Goldstein
Mark Goldstein
SSSilas Smith
Silas Smith
LJLior Jankelson
Lior Jankelson

31 July 2024 in JACC: Cardiovascular Imaging
Verified
Journal Article
Closed access
JMJacob A. Martin
Jacob A. Martin
THTheodore Hill
Theodore Hill
MSMuhamed Saric
Muhamed Saric
AVAlan F. Vainrib
Alan F. Vainrib
DBDaniel Bamira
Daniel Bamira
LJLior Jankelson
Lior Jankelson

1 December 2024 in Alzheimer's & Dementia
Verified
Journal Article
Open access
HZHao Zhang
Hao Zhang
NJNeil Jethani
Neil Jethani
SJSimon Jones
Simon Jones
NGNicholas Genes
Nicholas Genes
VMVincent J Major
Vincent J Major
NRNarges Razavian
Narges Razavian
Background: Large language models (LLMs) provide powerful natural language processing capabilities in medical and clinical tasks. Evaluating LLM performance is crucial due to potential false results. In this study, we assessed ChatGPT and Llama2, two state‐of‐the‐art LLMs, in extracting information from clinical notes, focusing on cognitive tests, specifically the Mini Mental State Exam (MMSE) and Cognitive Dementia Rating (CDR).Method: We compiled a dataset consisting of 765 clinical notes mentioning MMSE and CDR. 22 medically trained experts provided the ground truth. ChatGPT (GPT‐4, version “2023‐03‐15‐preview”) and Llama2 (“Llama‐2‐70b‐chat”) were used to extract MMSE and CDR instances with corresponding dates. Inference was successful for 742 notes. We used 20 notes for fine‐tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 assigned to two reviewers simultaneously. Precision, sensitivity, true/false negative rates, and accuracy were calculated. For double‐reviewed notes, we qualitatively assessed the errors.Result: The patient and note characteristics can be found in Table 1. For MMSE information extraction, ChatGPT (vs. Llama2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true‐negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true‐negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and Llama2 on double‐reviewed notes. Llama2 errors included 27 cases of total hallucination, 19 cases where other scores were reported instead of MMSE, 25 missed scores, and 23 cases where the wrong date was reported for the right score. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date.Conclusion: ChatGPT exhibited high accuracy in extracting MMSE scores and dates, with better performance compared to Llama2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.