Background/Objectives: The demand for accessible and reliable digital health services has increased significantly in recent years, particularly in regions facing physician shortages. HomeDOCtor, a conversational AI platform developed in Slovenia, addresses this need with a nationally adapted architecture that combines retrieval-augmented generation (RAG) and a Redis-based vector database of curated medical guidelines. The objective of this study was to assess the performance and impact of HomeDOCtor in providing AI-powered healthcare assistance. Methods: HomeDOCtor is designed for human-centered communication and clinical relevance, supporting multilingual and multimedia citizen inputs while being available 24/7. It was tested using a set of 100 international clinical vignettes and 150 internal medicine exam questions from the University of Ljubljana to validate its clinical performance. Results: During its six-month nationwide deployment, HomeDOCtor received overwhelmingly positive user feedback with minimal criticism, and exceeded initial expectations, especially in light of widespread media narratives warning about the risks of AI. HomeDOCtor autonomously delivered localized, evidence-based guidance, including self-care instructions and referral suggestions, with average response times under three seconds. On international benchmarks, the system achieved ≥95% Top-1 diagnostic accuracy, comparable to leading medical AI platforms, and significantly outperformed stand-alone ChatGPT-4o in the national context (90.7% vs. 80.7%, p = 0.0135). Conclusions: Practically, HomeDOCtor eases the burden on healthcare professionals by providing citizens with 24/7 autonomous, personalized triage and self-care guidance for less complex medical issues, ensuring that these cases are self-managed efficiently. The system also identifies more serious cases that might otherwise be neglected, directing them to professionals for appropriate care. Theoretically, HomeDOCtor demonstrates that domain-specific, nationally adapted large language models can outperform general-purpose models. Methodologically, it offers a framework for integrating GDPR-compliant AI solutions in healthcare. These findings emphasize the value of localization in conversational AI and telemedicine solutions across diverse national contexts.
The paper introduces RAGCare-QA, an extensive dataset of 420 theoretical medical knowledge questions for assessing Retrieval-Augmented Generation (RAG) pipelines in medical education and evaluation settings. The dataset includes one-choice-only questions from six medical specialties (Cardiology, Endocrinology, Gastroenterology, Family Medicine, Oncology, and Neurology) with three levels of complexity (Basic, Intermediate, and Advanced). Each question is accompanied by the best fit of RAG implementation complexity level, such as Basic RAG (315 questions, 75.0 %), Multi-vector RAG (82 questions, 19.5 %), and Graph-enhanced RAG (23 questions, 5.5 %). The questions emphasize theoretical medical knowledge on fundamental concepts, pathophysiology, diagnostic criteria, and treatment principles important in medical education. The dataset is a useful tool for the assessment of RAG- based medical education systems, allowing researchers to fine-tune retrieval methods for various categories of theoretical medical knowledge questions.
After years of promises that artificial intelligence (AI) will reform healthcare, healthcare systems around the world and in Slovenia, still face serious problems. In this article, we tested the abilities of the HomeDOCtor and GPT-4o in answering medical questions. Additionally we also analyzed how HomeDOCtor could help with the problem slovenian Healthcare system is facing and what could be the financial benefits of Slovenia by incorporating it into its healthcare system. The test results showed that both HomeDOCtor and GPT‑4o answer medical questions very well, with HomeDOCtor being able to provide users with additional material in its answers. The financial assessment of the savings suggests that the Slovenian healthcare system could save between 19 and 347 million annually by introducing a system like HomeDOCtor.