Chatbots using the Large Language Model (LLM) generate human responses to questions from all categories. Due to staff shortages in healthcare systems, patients waiting for an appointment increasingly use chatbots to get information about their condition. Given the number of chatbots currently available, assessing the responses they generate is essential. Five chatbots with free access were selected (Gemini, Microsoft Copilot, PiAI, ChatGPT, ChatSpot) and blinded using letters (A, B, C, D, E). Each chatbot was asked questions about cardiology, oncology, and psoriasis. Responses were compared to guidelines from the European Society of Cardiology, American Academy of Dermatology and American Society of Clinical Oncology. All answers were assessed using readability scales (Flesch Reading Scale, Gunning Fog Scale Level, Flesch-Kincaid Grade Level and Dale-Chall Score). Using a 3-point Likert scale, two independent medical professionals assessed the compliance of the responses with the guidelines. A total of 45 questions were asked of all chatbots. Chatbot C gave the shortest answers, 7.0 (6.0 - 8.0), and Chatbot A the longest 17.5 (13.0 - 24.5). The Flesch Reading Ease Scale ranged from 16.3 (12.2 - 21.9) (Chatbot D) to 39.8 (29.0 - 50.4) (Chatbot A). Flesch-Kincaid Grade Level ranged from 12.5 (10.6 - 14.6) (Chatbot A) to 15.9 (15.1 - 17.1) (Chatbot D). Gunning Fog Scale Level ranged from 15.77 (Chatbot A) to 19.73 (Chatbot D). Dale-Chall Score ranged from 10.3 (9.3 - 11.3) (Chatbot A) to 11.9 (11.5 - 12.4) (Chatbot D). This study indicates that chatbots vary in length, quality, and readability. They answer each question in their own way, based on the data they have pulled from the web. Reliability of the responses generated by chatbots is high. This suggests that people who want information from a chatbot need to be careful and verify the answers they receive, particularly when they ask about medical and health aspects.
The rapid development of large language models (LLMs), including ChatGPT, Gemini, and Copilot, has led to their increasing use in health communication and patient education. However, their growing popularity raises important concerns about whether the language they generate aligns with recommended readability standards and patient health literacy levels. This review synthesizes evidence on the readability of medical information generated by chatbots using established linguistic readability indices. A comprehensive search of PubMed, Scopus, Web of Science, and Cochrane Library identified 4209 records, from which 140 studies met the eligibility criteria. Across the included publications, 21 chatbots and 14 readability scales were examined, with the Flesch–Kincaid Grade Level and Flesch Reading Ease being the most frequently applied metrics. The results demonstrated substantial variability in readability across chatbot models; however, most texts corresponded to a secondary or early tertiary reading level, exceeding the commonly recommended 8th-grade level for patient-facing materials. ChatGPT-4, Gemini, and Copilot exhibited more consistent readability patterns, whereas ChatGPT-3.5 and Perplexity produced more linguistically complex content. Notably, DeepSeek-V3 and DeepSeek-R1 generated the most accessible responses. The findings suggest that, despite technological advances, AI-generated medical content remains insufficiently readable for general audiences, posing a potential barrier to equitable health communication. These results underscore the need for readability-aware AI design, standardized evaluation frameworks, and future research integrating quantitative readability metrics with patient-level comprehension outcomes.
Psoriasis is a chronic, inflammatory, often relapsing disease that is frequently associated with other diseases of similar pathogenesis. The multi-morbidity in the psoriasis population significantly impedes both diagnosis and implementation of appropriate preventive measures. However, the common denominator for this group of diseases is the inflammatory process that initiates the appearance of subsequent symptoms and health consequences, most of which can be avoided or alleviated by modifying the patient’s lifestyle and incorporating appropriate treatment. Health consequences associated with systemic inflammation include cardiovascular incidents and other cardiometabolic diseases. This article was based on available publications on the onset, incidence, and prevention of cardiovascular disease in the psoriasis patient population.
16 December 2024 in Advances in Clinical and Experimental Medicine
The coronavirus disease 2019 (COVID-19) pandemic has significantly accelerated the development and use of new healthcare technologies. While younger individuals may have been able to quickly embrace virtual advancements, older adults may still have different needs in terms of health communication. To identify areas of interest and preferred sources of information related to the COVID-19 pandemic among older adults and to verify their eHealth competencies. The study was conducted between February 2022 and July 2022. It included listeners from the University of the Third Age (U3A) and younger students. Both groups received information about the HealthBuddy+ chatbot, a questionnaire that addressed respondents' interests about COVID-19, and the PL-eHEALS (eHealth Literacy Scale) questionnaire to measure their eHealth competencies. There were 573 participants in the study (U3A listeners - 303 participants, median age: 73 years (interquartile range (IQR): 69-77); young adult students - 270, median age: 24 years (IQR: 23-24). The primary source of information about COVID-19 for older adults was television (84.5%), and for younger adults, internet (84.4%). Among the older adults, only 17% ever interacted with a chatbot (younger adults - 78% respectively), and 19% considered it a trustworthy source of information on COVID-19 compared to 79% of younger respondents. Older adults and younger adults in our study were most interested in COVID-19 treatment methods (45.5% and 69.3%, respectively), symptoms of the disease (36.6% and 35.2%, respectively) and chronic diseases coexisting with COVID-19 (35.0% and 51.5%, respectively). However, their eHealth competencies were generally low (median (Me): 34; IQR: 30-39) compared to younger adults (Me: 42; IQR: 40-47). Health education for older adults should be appropriately tailored to their current needs and differentiated. The level of eHealth competencies of older adults suggests that much work remains to narrow the gap between the eHealth competencies of the younger and older generations.
Aim: This study aimed to evaluate the readability and citation practices of artificial intelligence (AI)-generated responses to questions about human metapneumovirus, a respiratory virus of growing public health concern. Subject and methods: Five widely used AI chatbots—ChatGPT-4, Copilot, Gemini, Claude.ai, and Grok—were prompted with 14 standardized questions based on official guidelines from the World Health Organization, the Centers for Disease Control and Prevention, and the Australian National Health and Medical Research Council. Responses were anonymized and assessed using six established readability metrics: Flesch–Kincaid Reading Ease and Flesch–Kincaid Grade Level, Gunning Fog Index, SMOG (Simple Measure of Gobbledygook) Index, Coleman–Liau Index, and Automated Readability Index. Scores were compared to standards recommended by the American Medical Association and the National Institutes of Health. Citation frequency and credibility were also analyzed. Results: Among 70 chatbot responses, only one met the recommended readability level. Median readability scores ranged from grade 10.4 to 16.0, indicating high complexity. One chatbot generated the most readable content, while another scored lowest. Only two chatbots included source citations. One cited 68 reliable sources, primarily from health organizations and academic institutions, while the other referenced 31 sources of varying quality. Conclusion: AI-generated health content often exceeds recommended readability thresholds and lacks consistent citation practices. These issues may hinder understanding and trust. Improving default readability settings and integrating real-time citation features could enhance the accessibility and credibility of chatbot-based health communication.