Introduction: Publicly available AI language models such as ChatGPT have demonstrated utility in text generation and even problem-solving when provided with clear instructions. Amidst this transformative shift, the aim of this study is to assess ChatGPT's performance on the orthopaedic surgery in-training examination (OITE). Methods: All 213 OITE 2021 web-based questions were retrieved from the AAOS-ResStudy website (https://www.aaos.org/education/examinations/ResStudy). Two independent reviewers copied and pasted the questions and response options into ChatGPT Plus (version 4.0) and recorded the generated answers. All media-containing questions were flagged and carefully examined. Twelve OITE media-containing questions that relied purely on images (clinical pictures, radiographs, MRIs, CT scans) and could not be rationalized from the clinical presentation were excluded. Cohen's Kappa coefficient was used to examine the agreement of ChatGPT-generated responses between reviewers. Descriptive statistics were used to summarize the performance (% correct) of ChatGPT Plus. The 2021 norm table was used to compare ChatGPT Plus' performance on the OITE to national orthopaedic surgery residents in that same year. Results: A total of 201 questions were evaluated by ChatGPT Plus. Excellent agreement was observed between raters for the 201 ChatGPT-generated responses, with a Cohen's Kappa coefficient of 0.947. 45.8% (92/201) were media-containing questions. ChatGPT had an average overall score of 61.2% (123/201). Its score was 64.2% (70/109) on non-media questions. When compared to the performance of all national orthopaedic surgery residents in 2021, ChatGPT Plus performed at the level of an average PGY3. Discussion: ChatGPT Plus is able to pass the OITE with an overall score of 61.2%, ranking at the level of a third-year orthopaedic surgery resident. It provided logical reasoning and justifications that may help residents improve their understanding of OITE cases and general orthopaedic principles. Further studies are still needed to examine their efficacy and impact on long-term learning and OITE/ABOS performance.

Introduction: Online educational materials have become a fundamental resource for orthopaedic surgery patients; however, they are frequently written at a high level of reading ability and are too complicated for some patients to understand. The aim of this study was to appraise the readability of the Orthopaedic Trauma Association (OTA) patient education materials. Methods: All 41 articles on the OTA patient education website (https://ota.org/for-patients ) were analyzed for readability. Readability scores were calculated by 2 independent reviewers using the (1) FleschKincaid Grade Level (FKGL) and the (2) Flesch Reading Ease algorithms. Mean readability scores were compared across anatomic categories. One-sample t test was performed to compare mean FKGL with the recommended sixth-grade readability level and the average American adult reading level. Results: The average (SD) FKGL of the 41 OTA articles was 8.15 (1.14). The average (SD) Flesch Reading Ease for the OTA patient education materials was 65.5 (6.60). Four (11%) of the articles were at or below a sixth-grade reading level. The average readability of the OTA articles was significantly higher than the recommended sixth-grade level (P < 0.001; 95% confidence interval, 7.79–8.51). The average readability of the OTA articles was not significantly different from the average eighth-grade reading skill level of US adults (P = 0.41; 95% confidence interval, 7.79–8.51). Conclusion: Our findings suggest that despite most OTA patient education materials having readability levels suitable for the average US adult, these reading materials are still above the recommended sixth-grade level and may be too difficult for patient comprehension.

Introduction: The artificial intelligence language model Chat Generative Pretrained Transformer (ChatGPT) has shown potential as a reliable and accessible educational resource in orthopaedic surgery. Yet, the accuracy of the references behind the provided information remains elusive, which poses a concern for maintaining the integrity of medical content. This study aims to examine the accuracy of the references provided by ChatGPT-4 concerning the Airway, Breathing, Circulation, Disability, Exposure (ABCDE) approach in trauma surgery. Methods: Two independent reviewers critically assessed 30 ChatGPT-4–generated references supporting the well-established ABCDE approach to trauma protocol, grading them as 0 (nonexistent), 1 (inaccurate), or 2 (accurate). All discrepancies between the ChatGPT-4 and PubMed references were carefully reviewed and bolded. Cohen's Kappa coefficient was used to examine the agreement of the accuracy scores of the ChatGPT-4–generated references between reviewers. Descriptive statistics were used to summarize the mean reference accuracy scores. To compare the variance of the means across the 5 categories, one-way analysis of variance was used. Results: ChatGPT-4 had an average reference accuracy score of 66.7%. Of the 30 references, only 43.3% were accurate and deemed “true” while 56.7% were categorized as “false” (43.3% inaccurate and 13.3% nonexistent). The accuracy was consistent across the 5 trauma protocol categories, with no significant statistical difference (p = 0.437). Discussion: With 57% of references being inaccurate or nonexistent, ChatGPT-4 has fallen short in providing reliable and reproducible references—a concerning finding for the safety of using ChatGPT-4 for professional medical decision making without thorough verification. Only if used cautiously, with cross-referencing, can this language model act as an adjunct learning tool that can enhance comprehensiveness as well as knowledge rehearsal and manipulation.

Introduction: Web-based resources serve as a fundamental educational platform for orthopaedic trauma patients; however, they are frequently written above the recommended sixth-grade reading level, and previous studies have demonstrated this for the American Academy of Orthopaedic Surgeons (AAOS) web-based articles. In this study, we perform an updated assessment of the readability of AAOS trauma-related educational articles as compared with injury-matched education materials developed by the Orthopaedic Trauma Association (OTA). Methods: All 46 AAOS trauma-related web-based (https://www.orthoinfo.org/) patient education articles were analyzed for readability. Two independent reviewers used (1) the Flesch-Kincaid Grade Level (FKGL) and (2) the Flesch Reading Ease (FRE) algorithms to calculate the readability level. Mean readability scores were compared across body part categories. A one-sample t-test was done to compare mean FKGL with the recommended sixth-grade readability level and the average American adult reading level. A two-sample t-test was used to compare the readability scores of the AAOS trauma-related articles with those of the OTA. Results: The average (SD) FKGL and FRE for the AAOS articles were 8.9 (0.74) and 57.2 (5.8), respectively. All articles were written above the sixth-grade reading level. The average readability of the AAOS articles was significantly greater than the recommended sixth-grade reading level (P < 0.001). The average FKGL and FRE for all AAOS articles were significantly higher compared with those of the OTA articles (8.9 ± 0.74 versus 8.1 ± 1.14, P < 0.001 and 57.2 ± 5.8 versus 65.6 ± 6.6, P < 0.001, respectively). Excellent agreement was observed between raters for the FKGL 0.956 (95% confidence interval, 0.922 to 0.975) and FRE 0.993 (95% confidence interval, 0.987 to 0.996). Discussion: Our findings suggest that after almost a decade, the readability of the AAOS trauma-related articles remains unchanged. The AAOS and OTA trauma patient education materials have high readability levels and may be too difficult for patient comprehension. A need remains to improve the readability of these commonly used trauma education materials.