Background: The competence of ChatGPT (Chat Generative Pre-Trained Transformer) in non-English languages is not well studied. Objective: This study compared the performances of GPT-3.5 (Generative Pre-trained Transformer) and GPT-4 on the Japanese Medical Licensing Examination (JMLE) to evaluate the reliability of these models for clinical reasoning and medical knowledge in non-English languages. Methods: This study used the default mode of ChatGPT, which is based on GPT-3.5; the GPT-4 model of ChatGPT Plus; and the 117th JMLE in 2023. A total of 254 questions were included in the final analysis, which were categorized into 3 types, namely general, clinical, and clinical sentence questions. Results: The results indicated that GPT-4 outperformed GPT-3.5 in terms of accuracy, particularly for general, clinical, and clinical sentence questions. GPT-4 also performed better on difficult questions and specific disease questions. Furthermore, GPT-4 achieved the passing criteria for the JMLE, indicating its reliability for clinical reasoning and medical knowledge in non-English languages. Conclusions: GPT-4 could become a valuable tool for medical education and clinical support in non–English-speaking regions, such as Japan.

Background: The reliability of GPT-4, a state-of-the-art expansive language model specializing in clinical reasoning and medical knowledge, remains largely unverified across non-English languages. Objective: This study aims to compare fundamental clinical competencies between Japanese residents and GPT-4 by using the General Medicine In-Training Examination (GM-ITE). Methods: We used the GPT-4 model provided by OpenAI and the GM-ITE examination questions for the years 2020, 2021, and 2022 to conduct a comparative analysis. This analysis focused on evaluating the performance of individuals who were concluding their second year of residency in comparison to that of GPT-4. Given the current abilities of GPT-4, our study included only single-choice exam questions, excluding those involving audio, video, or image data. The assessment included 4 categories: general theory (professionalism and medical interviewing), symptomatology and clinical reasoning, physical examinations and clinical procedures, and specific diseases. Additionally, we categorized the questions into 7 specialty fields and 3 levels of difficulty, which were determined based on residents’ correct response rates. Results: Upon examination of 137 GM-ITE questions in Japanese, GPT-4 scores were significantly higher than the mean scores of residents (residents: 55.8%, GPT-4: 70.1%; P<.001). In terms of specific disciplines, GPT-4 scored 23.5 points higher in the “specific diseases,” 30.9 points higher in “obstetrics and gynecology,” and 26.1 points higher in “internal medicine.” In contrast, GPT-4 scores in “medical interviewing and professionalism,” “general practice,” and “psychiatry” were lower than those of the residents, although this discrepancy was not statistically significant. Upon analyzing scores based on question difficulty, GPT-4 scores were 17.2 points lower for easy problems (P=.007) but were 25.4 and 24.4 points higher for normal and difficult problems, respectively (P<.001). In year-on-year comparisons, GPT-4 scores were 21.7 and 21.5 points higher in the 2020 (P=.01) and 2022 (P=.003) examinations, respectively, but only 3.5 points higher in the 2021 examinations (no significant difference). Conclusions: In the Japanese language, GPT-4 also outperformed the average medical residents in the GM-ITE test, originally designed for them. Specifically, GPT-4 demonstrated a tendency to score higher on difficult questions with low resident correct response rates and those demanding a more comprehensive understanding of diseases. However, GPT-4 scored comparatively lower on questions that residents could readily answer, such as those testing attitudes toward patients and professionalism, as well as those necessitating an understanding of context and communication. These findings highlight the strengths and limitations of artificial intelligence applications in medical education and practice.

Background: Qualitative research is crucial for understanding the values and beliefs underlying individual experiences, emotions, and behaviors, particularly in social sciences and health care. Traditionally reliant on manual analysis by experienced researchers, this methodology requires significant time and effort. The advent of artificial intelligence (AI) technology, especially large language models such as ChatGPT (OpenAI), holds promise for enhancing qualitative data analysis. However, existing studies have predominantly focused on AI’s application to English-language datasets, leaving its applicability to non-English languages, particularly structurally and contextually complex languages such as Japanese, insufficiently explored. Objective: This study aims to evaluate the feasibility, strengths, and limitations of ChatGPT-4 in analyzing qualitative Japanese interview data by directly comparing its performance with that of experienced human researchers. Methods: A comparative qualitative study was conducted to assess the performance of ChatGPT-4 and human researchers in analyzing transcribed Japanese semistructured interviews. The analysis focused on thematic agreement rates, interpretative depth, and ChatGPT’s ability to process culturally nuanced concepts, particularly for descriptive and socio-culturally embedded themes. This study analyzed transcripts from 30 semistructured interviews conducted between February and March 2024 in an urban community hospital (Hospital A) and a rural university hospital (Hospital B) in Japan. Interviews centered on the theme of “sacred moments” and involved health care providers and patients. Transcripts were digitized using NVivo (version 14; Lumivero) and analyzed using ChatGPT-4 with iterative prompts for thematic analysis. The results were compared with a reflexive thematic analysis performed by human researchers. Furthermore, to assess the adaptability and consistency of ChatGPT in qualitative analysis, Charmaz’s grounded theory and Pope’s five-step framework approach were applied. Results: ChatGPT-4 demonstrated high thematic agreement rates (>80%) with human researchers for descriptive themes such as “personal experience of a sacred moment” and “building relationships.” However, its performance declined for themes requiring deeper cultural and emotional interpretation, such as “difficult to answer, no experience of sacred moments” and “fate.” For these themes, agreement rates were approximately 30%, revealing significant limitations in ChatGPT’s ability to process context-dependent linguistic structures and implicit emotional expressions in Japanese. Conclusions: ChatGPT-4 demonstrates potential as an auxiliary tool in qualitative research, particularly for efficiently identifying descriptive themes within Japanese-language datasets. However, its limited capacity to interpret cultural and emotional nuances highlights the continued necessity of human expertise in qualitative analysis. These findings emphasize the complementary role of AI-assisted qualitative research and underscore the importance of further advancements in AI models tailored to non-English linguistic and cultural contexts. Future research should explore strategies to enhance AI’s interpretability, expand multilingual training datasets, and assess the applicability of emerging AI models in diverse cultural settings. In addition, ethical and legal considerations in AI-driven qualitative analysis require continued scrutiny.

Virtual Patient Simulations (VPSs) have been cited as a novel learning strategy, but there is little evidence that VPSs yield improvements in clinical reasoning skills and medical knowledge. This study aimed to clarify the effectiveness of VPSs for improving clinical reasoning skills among medical students, and to compare improvements in knowledge or clinical reasoning skills relevant to specific clinical scenarios. We enrolled 210 fourth-year medical students in March 2017 and March 2018 to participate in a real-time pre-post experimental design conducted in a large lecture hall by using a clicker. A VPS program (^®Body Interact, Portugal) was implemented for one two-hour class session using the same methodology during both years. A pre–post 20-item multiple-choice questionnaire (10 knowledge and 10 clinical reasoning items) was used to evaluate learning outcomes. A total of 169 students completed the program. Participants showed significant increases in average total post-test scores, both on knowledge items (pre-test: median = 5, mean = 4.78, 95% CI (4.55–5.01); post-test: median = 5, mean = 5.12, 95% CI (4.90–5.43); p-value = 0.003) and clinical reasoning items (pre-test: median = 5, mean = 5.3 95%, CI (4.98–5.58); post-test: median = 8, mean = 7.81, 95% CI (7.57–8.05); p-value < 0.001). Thus, VPS programs could help medical students improve their clinical decision-making skills without lecturer supervision.

Background: Diagnostic errors constitute an important medical safety problem that needs improvement, and their frequency and severity are high in emergency room settings. Previous studies have suggested that diagnostic errors occur in 0.6-12% of first-time patients in the emergency room and that one or more cognitive factors are involved in 96% of these cases. This study aimed to identify the types of cognitive biases experienced by physicians in emergency rooms in Japan. Methods: We conducted a questionnaire survey using Nikkei Medical Online (Internet) from January 21 to January 31, 2019. Of the 159,519 physicians registered with Nikkei Medical Online when the survey was administered, those who volunteered their most memorable diagnostic error cases in the emergency room participated in the study. EZR was used for the statistical analyses. Results: A total of 387 physicians were included. The most common cognitive biases were overconfidence (22.5%), confirmation (21.2%), availability (12.4%), and anchoring (11.4%). Of the error cases, the top five most common initial diagnoses were upper gastrointestinal disease (22.7%), trauma (14.7%), cardiovascular disease (10.9%), respiratory disease (7.5%), and primary headache (6.5%). The corresponding final diagnoses for these errors were intestinal obstruction or peritonitis (27.3%), overlooked traumas (47.4%), other cardiovascular diseases (66.7%), cardiovascular disease (41.4%), and stroke (80%), respectively. Conclusions: A comparison of the initial and final diagnoses of cases with diagnostic errors shows that there were more cases with diagnostic errors caused by overlooking another disease in the same organ or a disease in a closely related organ.

Diagnostic errors are prevalent and associated with increased economic burden; however, little is known about their characteristics at the national level in Japan. This study aimed to investigate clinical outcomes and indemnity payment in cases of diagnostic errors using Japan’s largest database of national claims. We analyzed characteristics of diagnostic error cases closed between 1961 and 2017, accessed through the national Japanese malpractice claims database. We compared diagnostic error-related claims (DERC) with non-diagnostic error-related claims (non-DERC) in terms of indemnity, clinical outcomes, and factors underlying physicians’ diagnostic errors. All 1,802 malpractice claims were included in the analysis. The median patient age was 33 years (interquartile range = 10–54), and 54.2% were men. Deaths were the most common outcome of claims (939/1747; 53.8%). In total, 709 (39.3%, 95% CI: 37.0%–41.6%) DERC cases were observed. The adjusted total billing amount, acceptance rate, adjusted median claims payments, and proportion of deaths were significantly higher in DERC than non-DERC cases. Departments of internal medicine and surgery were 1.42 and 1.55 times more likely, respectively, to have DERC cases than others. Claims involving the emergency room (adjusted odds ratio [OR] = 5.88) and outpatient office (adjusted OR = 2.87) were more likely to be DERC than other cases. The initial diagnoses most likely to lead to diagnostic error were upper respiratory tract infection, non-bleeding digestive tract disease, and “no abnormality.” Cases of diagnostic errors produced severe patient outcomes and were associated with high indemnity. These cases were frequently noted in general exam and emergency rooms as well as internal medicine and surgery departments and were initially considered to be common, mild diseases.