Background: Recent advancements in artificial intelligence, such as GPT-3.5 Turbo (OpenAI) and GPT-4, have demonstrated significant potential by achieving good scores on text-only United States Medical Licensing Examination (USMLE) exams and effectively answering questions from physicians. However, the ability of these models to interpret medical images remains underexplored. Objective: This study aimed to comprehensively evaluate the performance, interpretability, and limitations of GPT-3.5 Turbo, GPT-4, and its successor, GPT-4 Vision (GPT-4V), specifically focusing on GPT-4V’s newly introduced image-understanding feature. By assessing the models on medical licensing examination questions that require image interpretation, we sought to highlight the strengths and weaknesses of GPT-4V in handling complex multimodal clinical information, thereby exposing hidden flaws and providing insights into its readiness for integration into clinical settings. Methods: This cross-sectional study tested GPT-4V, GPT-4, and ChatGPT-3.5 Turbo on a total of 227 multiple-choice questions with images from USMLE Step 1 (n=19), Step 2 clinical knowledge (n=14), Step 3 (n=18), the Diagnostic Radiology Qualifying Core Exam (DRQCE) (n=26), and AMBOSS question banks (n=150). AMBOSS provided expert-written hints and question difficulty levels. GPT-4V’s accuracy was compared with 2 state-of-the-art large language models, GPT-3.5 Turbo and GPT-4. The quality of the explanations was evaluated by choosing human preference between an explanation by GPT-4V (without hint), an explanation by an expert, or a tie, using 3 qualitative metrics: comprehensive explanation, question information, and image interpretation. To better understand GPT-4V’s explanation ability, we modified a patient case report to resemble a typical “curbside consultation” between physicians. Results: For questions with images, GPT-4V achieved an accuracy of 84.2%, 85.7%, 88.9%, and 73.1% in Step 1, Step 2 clinical knowledge, Step 3 of USMLE, and DRQCE, respectively. It outperformed GPT-3.5 Turbo (42.1%, 50%, 50%, 19.2%) and GPT-4 (63.2%, 64.3%, 66.7%, 26.9%). When GPT-4V answered correctly, its explanations were nearly as good as those provided by domain experts from AMBOSS. However, incorrect answers often had poor explanation quality: 18.2% (10/55) contained inaccurate text, 45.5% (25/55) had inference errors, and 76.3% (42/55) demonstrated image misunderstandings. With human expert assistance, GPT-4V reduced errors by an average of 40% (22/55). GPT-4V accuracy improved with hints, maintaining stable performance across difficulty levels, while medical student performance declined as difficulty increased. In a simulated curbside consultation scenario, GPT-4V required multiple specific prompts to interpret complex case data accurately. Conclusions: GPT-4V achieved high accuracy on multiple-choice questions with images, highlighting its potential in medical assessments. However, significant shortcomings were observed in the quality of explanations when questions were answered incorrectly, particularly in the interpretation of images, which could not be efficiently resolved through expert interaction. These findings reveal hidden flaws in the image interpretation capabilities of GPT-4V, underscoring the need for more comprehensive evaluations beyond multiple-choice questions before integrating GPT-4V into clinical settings.

Introduction: High-risk multiple myeloma (HRMM) is primarily defined by adverse cytogenetic abnormalities that are associated with inferior outcomes despite contemporary treatment with novel induction agents and autologous stem cell transplantation (ASCT). While the accumulation of high-risk lesions has been proposed to further stratify prognosis, the relative impact of single-hit versus double-hit HRMM on post-transplant outcomes in real-world clinical practice remains incompletely characterized. Methods: We conducted a multicenter retrospective cohort study of transplant-eligible adults with HRMM undergoing upfront ASCT between 2009 and 2024 at three U.S. academic centers. Patients were categorized as single-hit HRMM or double-hit HRMM based on the number of high-risk cytogenetic abnormalities at diagnosis. Response rates, progression-free survival (PFS), and overall survival (OS) were compared between groups using standard statistical methods. Results: A total of 154 patients were included, of whom 63 had SH-HRMM, and 91 had DH-HRMM. Following induction therapy, overall response rates (ORR) were high and comparable between groups. After ASCT, ORR was significantly higher in SH-HRMM compared with DH-HRMM (97% vs. 82%, p = 0.006), although depth of response did not differ significantly. With a median follow-up of 150 months, median OS was 103 months in SH-HRMM and 94 months in DH-HRMM (p = 0.40). Median PFS was 36 months and 31 months, respectively (p = 0.20). Restricted mean survival time analyses similarly demonstrated no significant differences in OS or PFS between groups. Although ORR differed, depth of response and long-term survival did not. Conclusions: In this real-world cohort of transplant-treated HRMM, single-hit disease was associated with a higher likelihood of achieving post-ASCT response; however, no significant differences in long-term survival outcomes were observed between single-hit and double-hit HRMM. These findings suggest that cytogenetic hit burden alone may be insufficient to predict post-transplant survival, highlighting the need for refined risk stratification incorporating additional biological and response-based markers.