Jamie A. Schlacter

Jamie A. Schlacter

Auto-generated profile, based on publicly available information

Jamie A. Schlacter

NYU Grossman School of Medicine New York New York USA

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
11 December 2024 in PLOS Digital Health
Verified
Journal Article
Open access
PDF file
HZHao Zhang
Hao Zhang
NJNeil Jethani
Neil Jethani
SJSimon Jones
Simon Jones
NGNicholas Genes
Nicholas Genes
VMVincent J. Major
Vincent J. Major
NRNarges Razavian
Narges Razavian
Ensuring reliability of Large Language Models (LLMs) in clinical tasks is crucial. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss’ Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

12 July 2023 in medRxiv
Verified
Preprint
Open access
PDF file
HZHao Zhang
Hao Zhang
NJNeil Jethani
Neil Jethani
SJSimon Jones
Simon Jones
NGNicholas Genes
Nicholas Genes
VMVincent J. Major
Vincent J. Major
NRNarges Razavian
Narges Razavian
Importance: Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Objective: Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates. Methods: Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss’ Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. Results: For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusions: In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

29 March 2023 in Cochlear Implants International
Verified
Journal Article
Closed access
JSJamie A. Schlacter
Jamie A. Schlacter
EKEmily Kay-Rivest
Emily Kay-Rivest
SMSean O. McMenomey
Sean O. McMenomey
JJJ. Thomas Roland Jr.
J. Thomas Roland Jr.
DFDavid R. Friedmann
David R. Friedmann
DJDaniel Jethanamest
Daniel Jethanamest
Immune-checkpoint inhibitors (ICI) are being utilized with increasing frequency and may be linked to neurologic and audiovestibular toxicities. This report aimed to describe a case of ICI-induced sensorineural hearing loss ultimately requiring bilateral cochlear implantation. A 42-year-old female with stage IV metastatic melanoma of the perianal skin was treated with ipilimumab (blocker of cytotoxic T-lymphocyte associated protein 4 [CTLA-4]) and nivolumab (anti-programmed cell death protein 1 [PD1]). After 21 weeks of therapy, she developed sudden vertigo and bilateral hearing loss. A full workup including MRI and lumbar puncture ruled out intracranial parenchymal metastases, leptomeningeal metastases, stroke and intracranial infection. ICI-associated aseptic meningoencephalitis was therefore diagnosed. The patient received systemic steroids as well as intratympanic steroids, which temporarily improved hearing, but eventually developed permanent, bilateral profound sensorineural hearing loss. The patient received bilateral cochlear implants and has demonstrated good performance one year after implantation. ICI are being increasingly used to treat a variety of advanced malignancies. This is the first report of bilateral cochlear implants in the context of profound hearing loss after an immunotherapy induced meningoencephalitis. ICI carries the risk of potential ototoxicity, including profound SNHL as depicted in our case. Cochlear implantation proved to be beneficial and may be considered in patients with ICI-related hearing loss.

13 January 2025 in Health Affairs Scholar
Verified
Journal Article
Open access
STSarah Tsuruo
Sarah Tsuruo
JSJamie Schlacter
Jamie Schlacter
SDSanket S Dhruva
Sanket S Dhruva
JRJoseph S Ross
Joseph S Ross
LHLeora I Horwitz
Leora I Horwitz
Lay Summary: In 2001, the Centers for Medicare and Medicaid Services (CMS) established the New Technology Add-On Payment (NTAP) program to help Medicare patients access costly new technologies. These technologies, approved by the Food and Drug Administration (FDA) must show “substantial clinical improvement” when compared to existing technologies. Recently, the FDA introduced two fast-track approval pathways allowing technologies to skip the “substantial clinical improvement” requirement. In this study, we describe the technologies and payments associated with NTAPs after this policy change.

20 December 2024 in Prenatal Diagnosis
Verified
Journal Article
Closed access
JSJamie A. Schlacter
Jamie A. Schlacter
EDEnrico Danzer
Enrico Danzer
CPClaire H. Packer
Claire H. Packer
EJEric Johnson
Eric Johnson
ACAaron B. Caughey
Aaron B. Caughey
KSKunj R. Sheth
Kunj R. Sheth

17 April 2023 in Journal of Vascular Surgery
Verified
Journal Article
Closed access
JSJamie A. Schlacter
Jamie A. Schlacter
MRMolly Ratner
Molly Ratner
JSJeffrey J. Siracuse
Jeffrey J. Siracuse
VPVirendra Patel
Virendra Patel
WJWilliam Johnson
William Johnson
KGKaran Garg
Karan Garg