Background: Over the past 2 years, researchers have used various medical licensing examinations to test whether ChatGPT (OpenAI) possesses accurate medical knowledge. The performance of each version of ChatGPT on the medical licensing examination in multiple environments showed remarkable differences. At this stage, there is still a lack of a comprehensive understanding of the variability in ChatGPT’s performance on different medical licensing examinations. Objective: In this study, we reviewed all studies on ChatGPT performance in medical licensing examinations up to March 2024. This review aims to contribute to the evolving discourse on artificial intelligence (AI) in medical education by providing a comprehensive analysis of the performance of ChatGPT in various environments. The insights gained from this systematic review will guide educators, policymakers, and technical experts to effectively and judiciously use AI in medical education. Methods: We searched the literature published between January 1, 2022, and March 29, 2024, by searching query strings in Web of Science, PubMed, and Scopus. Two authors screened the literature according to the inclusion and exclusion criteria, extracted data, and independently assessed the quality of the literature concerning Quality Assessment of Diagnostic Accuracy Studies-2. We conducted both qualitative and quantitative analyses. Results: A total of 45 studies on the performance of different versions of ChatGPT in medical licensing examinations were included in this study. GPT-4 achieved an overall accuracy rate of 81% (95% CI 78-84; P<.01), significantly surpassing the 58% (95% CI 53-63; P<.01) accuracy rate of GPT-3.5. GPT-4 passed the medical examinations in 26 of 29 cases, outperforming the average scores of medical students in 13 of 17 cases. Translating the examination questions into English improved GPT-3.5’s performance but did not affect GPT-4. GPT-3.5 showed no difference in performance between examinations from English-speaking and non–English-speaking countries (P=.72), but GPT-4 performed better on examinations from English-speaking countries significantly (P=.02). Any type of prompt could significantly improve GPT-3.5’s (P=.03) and GPT-4’s (P<.01) performance. GPT-3.5 performed better on short-text questions than on long-text questions. The difficulty of the questions affected the performance of GPT-3.5 and GPT-4. In image-based multiple-choice questions (MCQs), ChatGPT’s accuracy rate ranges from 13.1% to 100%. ChatGPT performed significantly worse on open-ended questions than on MCQs. Conclusions: GPT-4 demonstrates considerable potential for future use in medical education. However, due to its insufficient accuracy, inconsistent performance, and the challenges posed by differing medical policies and knowledge across countries, GPT-4 is not yet suitable for use in medical education. Trial Registration: PROSPERO CRD42024506687; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=506687
31 January 2025 in Patient Education and Counseling
This study conducted a systematic review of the systematic reviews of readability assessment studies to contribute to future readability research and practice by providing a comprehensive overview of information readability for patients. We searched multiple databases. We included systematic reviews of studies that quantitatively assessed the readability of health information. This study included 24 systematic reviews, which assessed the readability of 29,424 materials across 438 studies from 1990 to 2022. All systematic reviews reported that the readability of most materials exceeded the recommended sixth to eight-grade reading level. The readability level did not improve between 2001 and 2022, when the included systematic reviews were published. We found that the required reading level of information was too high for patients in all clinical areas included in this systematic review. We also identified gaps in readability assessment research in clinical areas and across media types and languages, which should be addressed by future studies. Health professionals should use available guidelines to make existing patient information easy to read and to write easy-to-read patient information, thereby improving readability. Such efforts are needed regardless of the organizations these professionals work at.
Japan has the highest number of cases of infertility treatment in the world. Studies have indicated that women undergoing infertility treatment feel stigmatized and suffer from psychological symptoms such as anxiety and depression. However, in Japan, few studies have quantitatively examined the association between the stigma of infertility and psychological symptoms, and, to our knowledge, no study has examined its association using a scale with tested reliability and validity. This study aims to quantitatively examine the relationship between infertility stigma and anxiety, depression, and psychological distress among women undergoing infertility treatment, using a scale that has been validated for reliability and validity. The cross-sectional study was conducted in December 2021 through a web-based survey of 254 participants undergoing infertility treatment. A multiple regression analysis was performed to examine the relationship between stigma and anxiety, depression, and psychological distress. Stigma was a statistically significant predictor of anxiety, depression, and psychological distress (standardized β = 0.58, p < 0.001; β = 0.50, p < 0.001; β = 0.62, p < 0.001, respectively) after controlling for sociodemographic and infertility characteristics. Future studies should examine the causal relationship between stigma and anxiety, depression, and psychological distress and how to intervene to reduce stigma among women undergoing infertility treatment.
During the COVID-19 pandemic, a social lockdown should be put in place and individuals should stay at home. Behavioral change is the only way to prevent the pandemic and overwhelmed healthcare systems until vaccines are available. We aimed to examine the psychological factors that predict staying at home during the COVID-19 pandemic and social lockdown. A total of 1980 participants in Japan completed a survey for this study from 9 to 11 May 2020, when the state of emergency covered all prefectures in the country. Self-reported behavior in terms of staying at home, the perceived severity of the pandemic, vulnerability to the pandemic, response efficacy, and self-efficacy based on protection motivation theory were assessed. Multiple regression analysis showed that perceived severity (standardized β = 0.11, p < 0.001) and self-efficacy (standardized β = 0.32, p < 0.001) significantly predicted greater levels of staying at home, after controlling for socio-demographics. However, perceived vulnerability and response efficacy did not. To encourage people to stay at home during the pandemic and social lockdown, increasing the perceived severity of infection by COVID-19 and self-efficacy in terms of exercising restraint with respect to going out may consequently encourage people to stay at home.
This study systematically reviews and conducts a meta-analysis to evaluate the performance of various large language models (LLMs) in dental licensing examinations worldwide. The aim is to assess the accuracy of these models in different linguistic and geographical contexts. This will inform their potential application in dental education and diagnostics. Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we conducted a comprehensive search across PubMed, Web of Science, and Scopus for studies published from 1 January 2022 to 1 May 2024. Two authors independently reviewed the literature based on the inclusion and exclusion criteria, extracted data, and evaluated the quality of the studies in accordance with the Quality Assessment of Diagnostic Accuracy Studies-2. We conducted qualitative and quantitative analyses to evaluate the performance of LLMs. Eleven studies met the inclusion criteria, encompassing dental licensing examinations from eight countries. GPT-3.5, GPT-4, and Bard achieved integrated accuracy rates of 54%, 72%, and 56%, respectively. GPT-4 outperformed GPT-3.5 and Bard, passing more than half of the dental licensing examinations. Subgroup analyses and meta-regression showed that GPT-3.5 performed significantly better in English-speaking countries. GPT-4's performance, however, remained consistent across different regions. LLMs, particularly GPT-4, show potential in dental education and diagnostics, yet their accuracy remains below the threshold required for clinical application. The lack of sufficient training data in dentistry has affected LLMs' accuracy. The reliance on image-based diagnostics also presents challenges. As a result, their accuracy in dental exams is lower compared to medical licensing exams. Additionally, LLMs even provide more detailed explanation for incorrect answer than correct one. Overall, the current LLMs are not yet suitable for use in dental education and clinical diagnosis.
Health materials to promote health behaviors should be readable and generate favorable evaluations of the message. Processing fluency (the subjective experience of ease with which people process information) has been increasingly studied over the past decade. In this review, we explore effects and instantiations of processing fluency and discuss the implications for designing effective health materials. We searched seven online databases using “processing fluency” as the key word. In addition, we gathered relevant publications using reference snowballing. We included published records that were written in English and applicable to the design of health materials. We found 40 articles that were appropriate for inclusion. Various instantiations of fluency have a uniform effect on human judgment: fluently processed stimuli generate positive judgments (e.g., liking, confidence). Processing fluency is used to predict the effort needed for a given task; accordingly, it has an impact on willingness to undertake the task. Physical perceptual, lexical, syntactic, phonological, retrieval, and imagery fluency were found to be particularly relevant to the design of health materials. Health-care professionals should consider the use of a perceptually fluent design, plain language, numeracy with an appropriate degree of precision, a limited number of key points, and concrete descriptions that make recipients imagine healthy behavior. Such fluently processed materials that are easy to read and understand have enhanced perspicuity and persuasiveness.