Background/Objectives: Artificial intelligence (AI) tools are used in clinical education and patient communication. Concordance with American Academy of Otolaryngology–Head and Neck Surgery Foundation clinical practice guidelines (CPG) and overall readability remain poorly defined. Methods: Over 2 months, ChatGPT, Google Gemini, and Google Search AI were queried using a single LF prompt and a layered set of sub-questions. Responses were scored with a 12-point rubric from the CPG plain-language tonsillectomy guideline. Readability was assessed using Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL). Temporal trends and between-group differences were analyzed using regression and repeated-measures methods. Results: Layered prompts demonstrated greater alignment with guideline subtopics than LF (89.5% vs. 71.4%, p < 0.001). For LF queries, mean concordance was 74.5% (95% CI 72.0–77.0) for Google AI, 71.3% (68.9–73.7) for Gemini, and 68.4% (66.0–70.9) for ChatGPT. Overall, Google AI outperformed Gemini and ChatGPT on pairwise comparisons (p = 0.001). Google AI demonstrated improvement in concordance (slope = 0.03, p = 0.003), while Gemini showed no change (p > 0.05) and ChatGPT worsened over time (slope = −0.03, p < 0.001). Readability differed by strategy and model. Layered prompts produced more accessible text (FKGL 44.6, FRE 10.3) compared with LF outputs (FKGL 35.2, FRE 11.3) and the CPG (FKGL 10.6, FRE 41.7). IRR was high (ICC = 0.96). Conclusions: Model type and prompt structure shaped the quality of AI-generated tonsillectomy guidance. Google AI consistently outperformed ChatGPT, with Gemini intermediate, and uniquely showed temporal improvement. Layered prompting produced more CPG-aligned and readable responses across all models. Readability was consistently above a 7th grade level.
16 September 2026 in European Archives of Oto-Rhino-Laryngology
Purpose: To evaluate the accuracy and readability of large language model (LLM)-generated patient education materials regarding tympanostomy tube placement. Methods: Over a two-month period, ChatGPT 4o, Gemini 2.5 Flash, and Google Search AI were prompted daily using long-form and layered prompt formats covering typical concerns regarding tympanostomy. Responses were scored on a 12-point rubric adapted from the AAO-HNS Clinical Practice Guidelines (CPG), assessing diagnostic accuracy, procedural clarity, and postoperative care. Readability was evaluated using Flesch Reading Ease and Flesch-Kincaid Grade Level. For each model and prompt type, average scores and variability were analyzed with 95% confidence intervals. Between-model differences were tested with Welch’s t -tests and Cohen’s d; temporal trends were analyzed via linear regression. Results: For long-form outputs, Google Search AI and Gemini 2.5 Flash demonstrated the highest mean CPG adherence (96.4% and 96.6%, respectively), both significantly exceeding ChatGPT 4o (84.3%; both P < .001; Cohen’s d ≈ 2.05). For layered prompt sessions, Google Search AI again demonstrated the highest adherence (91.7%), followed by Gemini 2.5 Flash (88.1%) and ChatGPT 4o (76.2%). Guideline adherence was temporally stable across all models ( P > .05 for most). All outputs exceeded the recommended sixth-grade reading threshold (mean FKGL, 9.4 for long-form; 10.6 for layered), with no statistically significant readability differences between prompt strategies. Conclusions: Google Search AI demonstrated the highest guideline concordance, though all models produced material too complex for typical patient comprehension. Structured prompting enhances clinical accuracy, but readability remains a key barrier to accessibility.