Background: Large language models (LLMs) have the potential to enhance information processing and clinical reasoning in the healthcare industry but are hindered by inaccuracies and hallucinations. The retrieval-augmented generation (RAG) technique may address these problems by integrating external knowledge sources. Methods: We developed a RAG-based chatbot called Thyro-GenAI by integrating a database of textbooks and guidelines with LLM. Thyro-GenAI and three service LLMs: OpenAI’s ChatGPT-4o, Perplexity AI’s ChatGPT-4o, and Anthropic’s Claude 3.5 Sonnet, were asked personalized clinical questions about thyroid disease. Three thyroid specialists assessed the quality of the generated responses and references without being blinded, which allowed them to interact with different chatbot interfaces. Results: Thyro-GenAI achieved the highest inverse-weighted mean rank for overall response quality. The overall inverse-weighted mean rankings for Thyro-GenAI, ChatGPT, Perplexity, and Claude were 3.0, 2.3, 2.8, and 1.9, respectively. Thyro-GenAI also achieved the second-highest inverse-weighted mean rank for overall reference quality. The overall inverse-weighted mean rankings for Thyro-GenAI, ChatGPT, Perplexity, and Claude were 3.1, 2.3, 3.2, and 1.8, respectively. Conclusions: Thyro-GenAI produced patient-specific clinical reasoning output based on a vector database, with fewer hallucinations and more reliability, compared to service LLMs. This emphasis on evidence-based responses ensures its safety and validity, addressing a critical limitation of existing LLMs. By integrating RAG with LLMs, it has the potential to support frontline clinical decision-making, especially helping first-line physicians by offering reliable decision support while managing thyroid disease patients.

Background: Anticoagulants are high-alert medications with substantial risk of serious bleeding, yet dosing and prescribing errors remain common. Although clinical decision support systems (CDSSs) can mitigate these errors, their impact is constrained by alert fatigue and limited interpretability. Objective: We developed and validated a hybrid CDSS that integrates rule-based logic with machine learning to improve the safe use of anticoagulants. Methods: This multicenter study used electronic health record data on anticoagulant prescriptions from 3 tertiary hospitals (1 for system development and internal validation and 2 for external validation). The hybrid CDSS combined a knowledge-based rule engine with a machine learning model trained to predict whether an anticoagulant prescription would require pharmacist intervention. The system was iteratively refined through pilot testing, internal validation, and external validation. Results: A total of 75,200 anticoagulant prescriptions were used for model development. The final hybrid CDSS comprised 44 patient-specific rules and 1129 drug-drug interaction rules, combined with a CatBoost classifier (version 1.2.5; Yandex) for alert prioritization. During internal validation, 88 (18.9%) alerts were generated; all were technically correct; 88.6% (n=78) were deemed clinically relevant; 86.4% (n=76) were considered clinically useful; and 13.6% (n=12) required pharmacist intervention. In external validation across 2 hospitals, alert rates ranged from 22.6% (7/31) to 32.1% (310/966), with 18.8% (22/117) to 57.1% (4/7) of alerts requiring pharmacist intervention. The hybrid CDSS showed strong discrimination (area under the receiver operating characteristic curve 0.871‐0.963). No false negatives were identified, but the estimates should be interpreted cautiously given the short validation periods and limited number of intervention-requiring prescriptions. Conclusions: A hybrid CDSS integrating rule-based logic with machine learning demonstrated high technical accuracy and clinical relevance across multiple institutions, suggesting its potential as a practical tool for supporting pharmacist-led anticoagulant prescription review.