Objective: The objective of the current study was to assess the quality of large language model (LLM) chatbot versus physician‐generated responses to patient‐generated rheumatology questions.Methods: We conducted a single‐center cross‐sectional survey of rheumatology patients (n = 17) in Edmonton, Alberta, Canada. Patients evaluated LLM chatbot versus physician‐generated responses for comprehensiveness and readability, with four rheumatologists also evaluating accuracy by using a Likert scale from 1 to 10 (1 being poor, 10 being excellent).Results: Patients rated no significant difference between artificial intelligence (AI) and physician‐generated responses in comprehensiveness (mean 7.12 ± SD 0.99 vs 7.52 ± 1.16; P = 0.1962) or readability (7.90 ± 0.90 vs 7.80 ± 0.75; P = 0.5905). Rheumatologists rated AI responses significantly poorer than physician responses on comprehensiveness (AI 5.52 ± 2.13 vs physician 8.76 ± 1.07; P < 0.0001), readability (AI 7.85 ± 0.92 vs physician 8.75 ± 0.57; P = 0.0003), and accuracy (AI 6.48 ± 2.07 vs physician 9.08 ± 0.64; P < 0.0001). The proportion of preference to AI‐ versus physician‐generated responses by patients and physicians was 0.45 ± 0.18 and 0.15 ± 0.08, respectively (P = 0.0106). After learning that one answer for each question was AI generated, patients were able to correctly identify AI‐generated answers at a lower proportion compared to physicians (0.49 ± 0.26 vs 0.97 ± 0.04; P = 0.0183). The average word count of AI answers was 69.10 ± 25.35 words, as compared to 98.83 ± 34.58 words for physician‐generated responses (P = 0.0008).Conclusion: Rheumatology patients rated AI‐generated responses to patient questions similarly to physician‐generated responses in terms of comprehensiveness, readability, and overall preference. However, rheumatologists rated AI responses significantly poorer than physician‐generated responses, suggesting that LLM chatbot responses are inferior to physician responses, a difference that patients may not be aware of. image
Background: Fatigue and burnout are prevalent among resident physicians across Canada. Shifts exceeding 24 hours are commonly purported as detrimental to resident health and performance. Residency training programs have employed strategies towards understanding and intervening upon the complex issue of resident fatigue, where alternative resident scheduling models have been an area of active investigation. This study sought to characterize drivers and outcomes of fatigue and burnout amongst internal medicine residents across different scheduling models. Methods: We conducted cross-sectional surveys were among internal medicine resident physicians at the University of Alberta. We collected anonymized socioeconomic demographics and medical education background, and estimated associations between demographic or work characteristics and fatigue and burnout outcomes. Results: Sixty-nine participants competed burnout questionnaires, and 165 fatigue questionnaires were completed (response rate of 48%). The overall prevalence of burnout was 58%. Lower burnout prevalence was noted among respondents with dependent(s) (p = 0.048), who identified as a racial minority (p = 0.018), or completed their medical degree internationally (p = 0.006). The 1-in-4 model was associated with the highest levels of fatigue, reported increased risk towards personal health (OR 4.98, 95%CI 1.77–13.99) and occupational or household harm (OR 5.69, 95%CI 1.87–17.3). Alternative scheduling models were not associated with these hazards. Conclusions: The 1-in-4 scheduling model was associated with high rates of resident physician fatigue, and alternative scheduling models were associated with less fatigue. Protective factors against fatigue are best characterized as strong social supports outside the workplace. Further studies are needed to characterize the impacts of alternative scheduling models on resident education and patient safety.