Marc D. Succi

Marc D. Succi

Auto-generated profile, based on publicly available information

Marc D. Succi

Harvard Medical School, Boston, USA

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
22 August 2023 in Journal of Medical Internet Research
Verified
Journal Article
Open access
ARArya Rao
Arya Rao
MPMichael Pang
Michael Pang
JKJohn Kim
John Kim
MKMeghana Kamineni
Meghana Kamineni
WLWinston Lie
Winston Lie
MSMarc D Succi
Marc D Succi
Background: Large language model (LLM)–based artificial intelligence chatbots direct the power of large training data sets toward successive, related tasks as opposed to single-ask tasks, for which artificial intelligence already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as artificial physicians, has not yet been evaluated. Objective: This study aimed to evaluate ChatGPT’s capacity for ongoing clinical decision support via its performance on standardized clinical vignettes. Methods: We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared its accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. Accuracy was measured by the proportion of correct responses to the questions posed within the clinical vignettes tested, as calculated by human scorers. We further conducted linear regression to assess the contributing factors toward ChatGPT’s performance on clinical tasks. Results: ChatGPT achieved an overall accuracy of 71.7% (95% CI 69.3%-74.1%) across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI 67.8%-86.1%) and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI 54.2%-66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=–15.8%; P<.001) and clinical management (β=–7.4%; P=.02) question types. Conclusions: ChatGPT achieves impressive accuracy in clinical decision-making, with increasing strength as it gains more clinical information at its disposal. In particular, ChatGPT demonstrates the greatest accuracy in tasks of final diagnosis as compared to initial diagnosis. Limitations include possible model hallucinations and the unclear composition of ChatGPT’s training data set.

21 June 2023 in Journal of the American College of Radiology
Verified
Journal Article
Closed access
ARArya Rao
Arya Rao
JKJohn Kim
John Kim
MKMeghana Kamineni
Meghana Kamineni
MPMichael Pang
Michael Pang
WLWinston Lie
Winston Lie
MSMarc D. Succi
Marc D. Succi

7 February 2023 in medRxiv
Verified
Preprint
Open access
PDF file
ARArya Rao
Arya Rao
JKJohn Kim
John Kim
MKMeghana Kamineni
Meghana Kamineni
MPMichael Pang
Michael Pang
WLWinston Lie
Winston Lie
MSMarc D. Succi
Marc D. Succi
BACKGROUND ChatGPT, a popular new large language model (LLM) built by OpenAI, has shown impressive performance in a number of specialized applications. Despite the rising popularity and performance of AI, studies evaluating the use of LLMs for clinical decision support are lacking. PURPOSE To evaluate ChatGPT’s capacity for clinical decision support in radiology via the identification of appropriate imaging services for two important clinical presentations: breast cancer screening and breast pain. MATERIALS AND METHODS We compared ChatGPT’s responses to the American College of Radiology (ACR) Appropriateness Criteria for breast pain and breast cancer screening. Our prompt formats included an open-ended (OE) format, where ChatGPT was asked to provide the single most appropriate imaging procedure, and a select all that apply (SATA) format, where ChatGPT was given a list of imaging modalities to assess. Scoring criteria evaluated whether proposed imaging modalities were in accordance with ACR guidelines. RESULTS ChatGPT achieved an average OE score of 1.83 (out of 2) and a SATA average percentage correct of 88.9% for breast cancer screening prompts, and an average OE score of 1.125 (out of 2) and a SATA average percentage correct of 58.3% for breast pain prompts. CONCLUSION Our results demonstrate the feasibility of using ChatGPT for radiologic decision making, with the potential to improve clinical workflow and responsible use of radiology services.

26 February 2023 in medRxiv
Verified
Preprint
Open access
PDF file
ARArya S. Rao
Arya S. Rao
MPMichael Pang
Michael Pang
JKJohn Kim
John Kim
MKMeghana Kamineni
Meghana Kamineni
WLWinston Lie
Winston Lie
MSMarc D. Succi
Marc D. Succi
IMPORTANCE: Large language model (LLM) artificial intelligence (AI) chatbots direct the power of large training datasets towards successive, related tasks, as opposed to single-ask tasks, for which AI already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as virtual physicians, has not yet been evaluated. OBJECTIVE: To evaluate ChatGPT’s capacity for ongoing clinical decision support via its performance on standardized clinical vignettes. DESIGN: We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. SETTING: ChatGPT, a publicly available LLM PARTICIPANTS: Clinical vignettes featured hypothetical patients with a variety of age and gender identities, and a range of Emergency Severity Indices (ESIs) based on initial clinical presentation. EXPOSURES: MSD Clinical Manual vignettes MAIN OUTCOMES AND MEASURES: We measured the proportion of correct responses to the questions posed within the clinical vignettes tested. RESULTS: ChatGPT achieved 71.7% (95% CI, 69.3% to 74.1%) accuracy overall across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI, 67.8% to 86.1%), and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI, 54.2% to 66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=-15.8%, p<0.001) and clinical management (β=-7.4%, p=0.02) type questions. CONCLUSIONS AND RELEVANCE: ChatGPT achieves impressive accuracy in clinical decision making, with particular strengths emerging as it has more clinical information at its disposal.

18 August 2022 in JAMA Network Open
Verified
Journal Article
Open access
PDF file
SGSoham Ghoshal
Soham Ghoshal
GRGrant Rigney
Grant Rigney
DCDebby Cheng
Debby Cheng
RBRyan Brumit
Ryan Brumit
MGMichael S. Gee
Michael S. Gee
MSMarc D. Succi
Marc D. Succi
The SARS-CoV2 pandemic has had an unprecedented impact on health care workers and the medical system. In following the US government’s recommendations to defer nonessential surgical care, many institutions made the decision to temporarily cancel elective surgery procedures to slow the spread of the virus.1 However, deferrals of certain elective surgeries have led to large backlogs in procedural care.2 Studies have estimated that these deferrals will grow to a cumulative backlog of more than 1 million cases 2 years after the end of elective surgery deferment.3 The pandemic also is associated with a reduction in surgical care for various racial and ethnic groups.4 Furthermore, there have been stark declines in both ambulatory care and necessary outpatient care visits, which other researchers have suggested may result in the development of more advanced disease states in the coming months or years.5-9 Other specialties, such as radiology, have experienced a significant reduction in imaging volumes with inconsistent recovery, including reduced emergency department diagnoses and initial cancer workups.10-15

1 April 2026 in JAMA Network Open
Verified
Journal Article
Open access
ARArya S. Rao
Arya S. Rao
KEKaiz P. Esmail
Kaiz P. Esmail
RLRichard S. Lee
Richard S. Lee
SJSharon Jiang
Sharon Jiang
BCBianca Arraiza Carlo
Bianca Arraiza Carlo
MSMarc D. Succi
Marc D. Succi
Importance: Large language models (LLMs) are increasingly marketed for clinical use, yet their ability to replicate full-spectrum clinical reasoning remains uncertain. Existing evaluations often rely on multiple-choice examinations that do not reflect the complexity of patient care. Objectives: To evaluate the longitudinal clinical reasoning ability of state-of-the-art LLMs and to introduce a multidimensional, clinically meaningful benchmark for clinical-grade artificial intelligence (AI). Design, Setting, and Participants: In this cross-sectional study, performance was evaluated using standardized clinical vignettes from the January 2025 update of MSD Manual vignettes. A total of 21 off-the-shelf LLMs, including recently released GPT-5, Claude 4.5 Opus, Gemini 3.0 Flash and Pro, and Grok 4, were evaluated. Models were assessed by medical student scorers in triplicate across sequential stages of the standard clinical workflow. Analyses were performed from January to December 2025. Main Outcomes and Measures: The primary outcome was the Proportional Index of Medical Evaluation for LLMs (PrIME-LLM) score, defined as the normalized polygonal area representing balanced accuracy across 5 domains of clinical reasoning as follows: differential diagnosis, diagnostic testing, final diagnosis, management, and miscellaneous clinical reasoning questions. Analyses including analyses of variance, t tests, and regression models were used to compare AI model performance and demographic associations. Results: LLMs were tested across 29 clinical vignettes (representing 16 254 responses in total). PrIME-LLM scores ranged from 0.64 (range, 0.63-0.65) (Gemini 1.5 Flash) to 0.78 (range, 0.77-0.79) (Grok 4), with reasoning-optimized models outperforming nonreasoning models and GPT models scoring highest overall. Differential diagnosis was less accurate than diagnostic testing, while final diagnosis, management, and miscellaneous reasoning were more accurate. Failure rates exceeded 0.80 (range, 0.90-1.00) for differential diagnosis in all models but were less than 0.40 (range, 0.09-0.39) for final diagnosis. Multimodal performance was robust; most LLM models showed improved accuracy with image inputs. Conclusions and Relevance: In this cross-sectional study of 21 LLMs, frontier LLMs achieved high accuracy on final diagnoses but performed poorly in generating differential diagnoses and navigating uncertainty relative to other reasoning stages. The PrIME-LLM framework provided greater separation than raw accuracy, revealing critical reasoning gaps obscured by traditional benchmarks. Thus, despite version-based improvements and advantages in reasoning-optimized models, off-the-shelf LLMs have not yet achieved the intelligence required for safe deployment and remain limited in demonstrating advanced clinical reasoning.