22 August 2023 in Journal of Medical Internet Research
Background: Large language model (LLM)–based artificial intelligence chatbots direct the power of large training data sets toward successive, related tasks as opposed to single-ask tasks, for which artificial intelligence already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as artificial physicians, has not yet been evaluated. Objective: This study aimed to evaluate ChatGPT’s capacity for ongoing clinical decision support via its performance on standardized clinical vignettes. Methods: We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared its accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. Accuracy was measured by the proportion of correct responses to the questions posed within the clinical vignettes tested, as calculated by human scorers. We further conducted linear regression to assess the contributing factors toward ChatGPT’s performance on clinical tasks. Results: ChatGPT achieved an overall accuracy of 71.7% (95% CI 69.3%-74.1%) across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI 67.8%-86.1%) and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI 54.2%-66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=–15.8%; P<.001) and clinical management (β=–7.4%; P=.02) question types. Conclusions: ChatGPT achieves impressive accuracy in clinical decision-making, with increasing strength as it gains more clinical information at its disposal. In particular, ChatGPT demonstrates the greatest accuracy in tasks of final diagnosis as compared to initial diagnosis. Limitations include possible model hallucinations and the unclear composition of ChatGPT’s training data set.
21 June 2023 in Journal of the American College of Radiology
15 March 2023 in The Journal of Arthroplasty
2 May 2023 in JMIR Preprints
BACKGROUND Large language model (LLM)–based artificial intelligence chatbots direct the power of large training data sets toward successive, related tasks as opposed to single-ask tasks, for which artificial intelligence already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as artificial physicians, has not yet been evaluated. OBJECTIVE This study aimed to evaluate ChatGPT’s capacity for ongoing clinical decision support via its performance on standardized clinical vignettes. METHODS We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared its accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. Accuracy was measured by the proportion of correct responses to the questions posed within the clinical vignettes tested, as calculated by human scorers. We further conducted linear regression to assess the contributing factors toward ChatGPT’s performance on clinical tasks. RESULTS ChatGPT achieved an overall accuracy of 71.7% (95% CI 69.3%-74.1%) across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI 67.8%-86.1%) and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI 54.2%-66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=–15.8%; P<.001) and clinical management (β=–7.4%; P=.02) question types. CONCLUSIONS ChatGPT achieves impressive accuracy in clinical decision-making, with increasing strength as it gains more clinical information at its disposal. In particular, ChatGPT demonstrates the greatest accuracy in tasks of final diagnosis as compared to initial diagnosis. Limitations include possible model hallucinations and the unclear composition of ChatGPT’s training data set.
16 October 2024 in Arthroplasty Today
Spinopelvic mechanics are critical in total hip arthroplasty; however, there is no established consensus for adjusting acetabular component positioning based on spinopelvic parameters. This study aimed to (1) validate a recently developed Patient-Specific acetabular safe-zone calculator that factors in spinopelvic parameters and (2) compare differences with hip-spine classification targets. A total of 3750 patients underwent primary total hip arthroplasty across 3 academic referral centers, with 33 (0.88%) requiring revision for instability. Spinopelvic parameters were measured before initial total hip arthroplasty, and acetabular component position was measured following the index and revision procedures. Most operations employed either computer navigation or robotic assistance (94%). Surgical approaches included both anterior and posterior techniques. Utilizing our recently developed patient-specific safe-zone calculator, theoretical intraoperative positions were calculated and compared to true component positions before and after revision. Among 33 patients who underwent revision, none dislocated at an average follow-up of 5.1 years. In the external validation cohort, the average absolute differences between the patient-specific safe-zone and the median hip-spine classification recommendation were 3.8° ± 2.1° inclination and 5.0° ± 3.2° version. For the pooled cohort, the absolute differences between the patient-specific safe-zone targets and the prerevision component positions were 7.9° ± 5.1° inclination and 11.4° ± 6.9° version. After revision, the mean absolute differences decreased to 3.6° ± 3.1° inclination and 5.8° ± 3.5° version (P < .001). A patient-specific approach improved acetabular component positioning accuracy within 6° of version and 4° of inclination of stable, revised hips. Patient-specific safe zones provide quantitative targets for nuanced spinopelvic preoperative planning that may mitigate risk of instability and may indicate use of assisted technologies.
15 April 2025 in The Journal of Foot and Ankle Surgery