The diagnostic accuracy of differential diagnoses generated by artificial intelligence (AI) chatbots, including the generative pretrained transformer 3 (GPT-3) chatbot (ChatGPT-3) is unknown. This study evaluated the accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical vignettes with common chief complaints. General internal medicine physicians created clinical cases, correct diagnoses, and five differential diagnoses for ten common chief complaints. The rate of correct diagnosis by ChatGPT-3 within the ten differential-diagnosis lists was 28/30 (93.3%). The rate of correct diagnosis by physicians was still superior to that by ChatGPT-3 within the five differential-diagnosis lists (98.3% vs. 83.3%, p = 0.03). The rate of correct diagnosis by physicians was also superior to that by ChatGPT-3 in the top diagnosis (53.3% vs. 93.3%, p < 0.001). The rate of consistent differential diagnoses among physicians within the ten differential-diagnosis lists generated by ChatGPT-3 was 62/88 (70.5%). In summary, this study demonstrates the high diagnostic accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical cases with common chief complaints. This suggests that AI chatbots such as ChatGPT-3 can generate a well-differentiated diagnosis list for common chief complaints. However, the order of these lists can be improved in the future.

Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: The medical interview remains a cornerstone of clinical training. There is growing interest in applying generative artificial intelligence (AI) in medical education, including medical interview training. However, its utility in culturally and linguistically specific contexts, including Japanese, remains underexplored. This study investigated the utility of generative AI for Japanese medical interview training. Objective: This pilot study aimed to evaluate the utility of generative AI as a tool for medical interviews training by comparing its performance with that of traditional face-to-face training methods using a simulated patient. Methods: We conducted a randomized crossover pilot study involving 20 postgraduate year 1-2 physicians from a university hospital. Participants were randomly allocated into two groups. Group A began with an AI-based station on a case involving abdominal pain, followed by a traditional station with a standardized patient presenting chest pain. Group B followed the reverse order, starting with the traditional station for abdominal pain, and subsequently within AI-based station for the chest pain scenario. In the AI-based stations, participants interacted with a GPTs-configured platform that simulated patient behaviors. GPTs are customizable versions of ChatGPT adapted for specific purposes. The traditional stations involved face-to-face interviews with a simulated patient. Both groups used identical, standardized case scenarios to ensure uniformity. Two independent evaluators, blinded to the study conditions, assessed participants' performances using six defined metrics: patient care and communication, history taking, physical examination, accuracy and clarity of transcription, clinical reasoning, and patient management. A 6-point Likert scale was employed for scoring. Discrepancy between the evaluators resolved through discussion. To ensure cultural and linguistic authenticity, all interviews and evaluations were conducted in Japanese. Results: AI-based stations scored lower across most categories, particularly in patient care and communication, than traditional stations (4.48 vs. 4.95, P=.009). However, AI-based stations demonstrated comparable performance in clinical reasoning, with a non-significant difference (4.43 vs. 4.85, P=.10). Conclusions: The comparable performance of generative AI in clinical reasoning highlights its potential as a complementary tool in medical interview training. One of its main advantages lies in enabling self-learning, allowing trainees to independently practice interviews without the need for simulated patients. Nonetheless, the lower scores in patient care and communication underline the importance of maintaining traditional methods that capture the nuances of human interaction. These findings support the adoption of hybrid training models that combine generative AI with conventional approaches to enhance the overall effectiveness of medical interview training in Japan. Clinical Trial: UMIN-CTR UMIN000053747; https://center6.umin.ac.jp/cgi-open-bin/ctr_e/ctr_view.cgi?recptno=R000061336.

Background: The Objective Structured Clinical Examination (OSCE) has been widely used to evaluate students in medical education. However, it is resource-intensive, presenting challenges in implementation. We hypothesized that generative artificial intelligence (AI) such as ChatGPT-4 could serve as a complementary assessor and alleviate the burden of physicians in evaluating OSCE. Objective: By comparing the evaluation scores between generative AI and physicians, this study aims to evaluate the validity of generative AI as a complementary assessor for OSCE. Methods: This experimental study was conducted at a medical university in Japan. We recruited 11 fifth-year medical students during the general internal medicine clerkship from April 2023 to December 2023. Participants conducted a mock medical interview with a patient experiencing abdominal pain and wrote patient notes. Four physicians independently evaluated the participants by reviewing medical interview videos and patient notes, while ChatGPT-4 was provided with interview transcripts and notes. Evaluations were conducted using the 6-domain rubric (patient care and communication, history taking, physical examination, patient notes, clinical reasoning, and management). Each domain was scored using a 6-point Likert scale, ranging from 1 (very poor) to 6 (excellent). Median scores were compared using the Wilcoxon signed-rank test, and the agreement between ChatGPT-4 and physicians was assessed using intraclass correlation coefficients (ICCs). All P values <.05 were considered statistically significant. Results: Although ChatGPT-4 assigned higher scores than physicians in terms of physical examination (median 4.0, IQR 4.0-5.0 vs median 4.0, IQR 3.0-4.0; P=.02), patient notes (median 6.0, IQR 5.0-6.0 vs median 4.0, IQR 4.0-4.0; P=.002), clinical reasoning (median 5.0, IQR 5.0-5.0 vs median 4.0, IQR 3.0-4.0; P<.001), and management (median 6.0, IQR 5.0-6.0 vs median 4.0, IQR 2.5-4.5; P=.002), there were no significant differences in the scores of patient care and communication (median 5.0, IQR 5.0-5.0 vs median 5.0, IQR 4.0-5.0; P=.06) and history taking (median 5.0, IQR 4.0-5.0 vs median 5.0, IQR 4.0-5.0; P>.99), respectively. ICC values were low in all domains, except for history taking, where the agreement was still poor (ICC=0.36, 95% CI –0.32 to 0.78). Conclusions: ChatGPT-4 produced higher evaluation scores than physicians in several OSCE domains, though the agreement between them was poor. Although these preliminary results suggest that generative AI may be able to support assessment in some domains of OSCE, further research is needed to establish its reproducibility and validity. Generative AI like ChatGPT-4 shows potential as a complementary assessor for OSCE. Trial Registration: University Hospital Medical Information Network Clinical Trials Registry UMIN000050489; https://center6.umin.ac.jp/cgi-open-bin/ctr/ctr_his_list.cgi?recptno=R000057513

Purpose: The data on the diagnostic contribution of general internal medicine (GIM) consultations for undiagnosed health problems from specialists are scarce. This study aims to explore the role of generalists as diagnostic medicine consultants in tertiary care settings. Patients and Methods: We conducted a retrospective observational study at a Japanese university hospital. GIM consultations for diagnosis from other departments on outpatients aged ≧ 20 years from January 2018 to December 2020 were included. Data were extracted from electronic medical records. The primary outcome was new diagnosis rates. The secondary outcomes were new diagnosis rates with clinical significance and clinical outcomes at 90 days from the index visit. Results: A total of 328 patients were included. The top five consulting departments were orthopedics (17.0%), cardiovascular (10.3%), otorhinolaryngology (8.8%), neurology (8.8%), and gastroenterology (7.9%). GIM identified 456 chief complaints (CCs), and the top five were fever (10.9%), abnormal laboratory results (8.3%), fatigue (5.9%), and pain (7.4%) or numbness (4.6%) in the extremities. There were 139 (104/328 patients: 31.8%) specialty consultations from GIM, and the top five departments were rheumatology (21.1%), gastroenterology (19.2%), orthopedics (9.6%), psychiatry (9.6%), and neurology (9.6%). In total, 277 new diagnoses were established in 232 patients (70.7%), and 203 patients had new diagnoses with clinical significance (61.8%). Clinical outcomes at 90 days from the time of the index visit were resolution/improvement (60.7%), unchanged/worsened (22.3%), and unknown (17.0%). Conclusion: Over 70% of GIM consultations from other departments established new diagnoses with favorable outcomes in > 60% of the patients.

Purpose: The general internal medicine (GIM) department can be an effective diagnostic coordinator for undiagnosed outpatients. We investigated the contribution of GIM consultations to the diagnosis of patients admitted to specialty departments in hospitals in Japan that have not yet adopted a hospitalist system. Patients and Methods: This single-center, retrospective observational study was conducted at a university hospital in Japan. GIM consultations from other departments on inpatients aged ≥ 20 years, from April 2016 to March 2021, were included. Data were extracted from electronic medical records, and consultation purposes were categorized into diagnosis, treatment, and diagnosis and treatment. The primary outcome was new diagnosis during hospitalization for patients with consultation purpose of diagnosis or diagnosis and treatment. The secondary outcomes were the purposes of consultation with the Diagnostic and Generalist Medicine department. Results: In total, 342 patients were included in the analysis. The purpose of the consultations was diagnosis for 253 patients (74%), treatment for 60 (17.5%), and diagnosis and treatment for 29 patients (8.5%). In 282 consultations for diagnosis and diagnosis and treatment, 179 new diagnoses were established for 162 patients (57.5%, 95% confidence interval [CI], 51.5– 63.3). Conclusion: The GIM department can function as a diagnostic consultant for inpatients with diagnostic problems admitted to other specialty departments in hospitals where hospitalist or other similar systems are not adopted.

Background: Atypical presentations have been increasingly recognized as a significant contributing factor to diagnostic errors in internal medicine. However, research to address associations between atypical presentations and diagnostic errors has not been evaluated due to the lack of widely applicable definitions and criteria for what is considered an atypical presentation. Objective: The aim of the study is to describe how atypical presentations are defined and measured in studies of diagnostic errors in internal medicine and use this new information to develop new criteria to identify atypical presentations at high risk for diagnostic errors. Methods: This study will follow an established framework for conducting scoping reviews. Inclusion criteria are developed according to the participants, concept, and context framework. This review will consider studies that fulfill all of the following criteria: include adult patients (participants); explore the association between atypical presentations and diagnostic errors using any definition, criteria, or measurement to identify atypical presentations and diagnostic errors (concept); and focus on internal medicine (context). Regarding the type of sources, this scoping review will consider quantitative, qualitative, and mixed methods study designs; systematic reviews; and opinion papers for inclusion. Case reports, case series, and conference abstracts will be excluded. The data will be extracted through MEDLINE, Web of Science, CINAHL, Embase, Cochrane Library, and Google Scholar searches. No limits will be applied to language, and papers indexed from database inception to December 31, 2023, will be included. Two independent reviewers (YH and RK) will conduct study selection and data extraction. The data extracted will include specific details about the patient characteristics (eg, age, sex, and disease), the definitions and measuring methods for atypical presentations and diagnostic errors, clinical settings (eg, department and outpatient or inpatient), type of evidence source, and the association between atypical presentations and diagnostic errors relevant to the review question. The extracted data will be presented in tabular format with descriptive statistics, allowing us to identify the key components or types of atypical presentations and develop new criteria to identify atypical presentations for future studies of diagnostic errors. Developing the new criteria will follow guidance for a basic qualitative content analysis with an inductive approach. Results: As of January 2024, a literature search through multiple databases is ongoing. We will complete this study by December 2024. Conclusions: This scoping review aims to provide rigorous evidence to develop new criteria to identify atypical presentations at high risk for diagnostic errors in internal medicine. Such criteria could facilitate the development of a comprehensive conceptual model to understand the associations between atypical presentations and diagnostic errors in internal medicine. Trial Registration: Open Science Framework; www.osf.io/27d5m International Registered Report Identifier (IRRID): DERR1-10.2196/56933