Background/Objectives: Large language models (LLMs) show promise in healthcare but face challenges with hallucinations, particularly in rapidly evolving fields like diabetes management. Traditional LLM updating methods are resource-intensive, necessitating new approaches for delivering reliable, current medical information. This study aimed to develop and evaluate a novel retrieval system to enhance LLM reliability in diabetes management across different languages and guidelines. Methods: We developed a dual retrieval-augmented generation (RAG) system integrating both Korean Diabetes Association and American Diabetes Association 2023 guidelines. The system employed dense retrieval with 11 embedding models (including OpenAI, Upstage, and multilingual models) and sparse retrieval using BM25 algorithm with language-specific tokenizers. Performance was evaluated across different top-k values, leading to optimized ensemble retrievers for each guideline. Results: For dense retrievers, Upstage’s Solar Embedding-1-large and OpenAI’s text-embedding-3-large showed superior performance for Korean and English guidelines, respectively. Multilingual models outperformed language-specific models in both cases. For sparse retrievers, the ko_kiwi tokenizer demonstrated superior performance for Korean text, while both ko_kiwi and porter_stemmer showed comparable effectiveness for English text. The ensemble retrievers, combining optimal dense and sparse configurations, demonstrated enhanced coverage while maintaining precision. Conclusions: This study presents an effective dual RAG system that enhances LLM reliability in diabetes management across different languages. The successful implementation with both Korean and American guidelines demonstrates the system’s cross-regional capability, laying a foundation for more trustworthy AI-assisted healthcare applications.

Integrating Artificial Intelligence of Things (AIoT) in healthcare offers transformative potential for real-time diagnostics and collaborative learning but presents critical challenges, including privacy preservation, computa... | Find, read and cite all the research you need on Tech Science Press

BACKGROUND The Korean National Health Screening Program assigns each laboratory and anthropometric result to a tier of Normal A, Normal B (borderline), or suspected disease. Large language models (LLMs) are increasingly used to interpret laboratory results, but their ability to perform national-program tier determination accurately and reproducibly, particularly near decision boundaries, has not been evaluated. OBJECTIVE This study aimed to construct and release a dual-verified benchmark for Korean health-checkup determination and to evaluate rule grounding, zero-shot failure patterns, and test–retest reliability in 2 frontier LLMs. METHODS Using published criteria, we generated a tier-classification set of 144 cases across 20 items and a value-ladder set of 528 cases in 22 ladders, yielding 6,321 ordered pairs. Gold labels were independently verified by a deterministic rule engine (100% agreement; 0/6,321 violations). Claude Fable 5 and GPT-5.6 Sol were evaluated under zero-shot and rule-grounded conditions in 10 runs per model per condition. Each run was a stateless full-benchmark pass; run-level inference concerned stochastic evaluations of fixed cases, not new clinical samples. The primary outcome was monotonicity violation rate; key secondary outcomes were the rule-grounding contrast and test–retest reliability. Analyses used ladder-cluster bootstrap, exact unpaired rank tests across runs, and paired case-level McNemar tests. BMI stratification was pilot-informed and specified before the main 10-run evaluation. RESULTS Under rule grounding, every observed run achieved 100% tier accuracy and 0 monotonicity violations, whereas no zero-shot run achieved 100% accuracy (exact unpaired run-level rank test, P<.001 per model). The paired case-level majority-vote comparison was not statistically significant (McNemar P=.0625), the minimum attainable 2-sided value with 5 one-directional discordant pairs. Zero-shot failures showed opposite model/configuration-specific patterns: excluding body mass index, GPT produced 4.5 times as many violations as Claude (P=.01), whereas on body mass index ladders, Claude produced 2.5 times as many as GPT (P<.001); the aggregate comparison showed no difference (P=.99). At equal run counts, flip proportions were 6.4% for Claude and 11.2% for GPT (McNemar P=.004). Ten runs met a development-stage, study-specific SEM criterion for all nonzero zero-shot ladder estimates examined; this was not a prospectively prespecified or universal minimum. CONCLUSIONS For the public rules evaluated, no errors were observed when the criteria were supplied in context, whereas zero-shot errors were model/item specific and aggregate accuracy obscured reliability differences. These findings support explicit rule grounding and repeated runs with justified sufficiency for evaluating this prerequisite rule-application capability; they do not establish end-to-end clinical deployment safety.

Clinical documentation burden contributes significantly to physician burnout, with health care professionals spending much of their time on electronic health record interactions. Automatic speech recognition (ASR) systems offer a promising solution; however, their application in Korean medical settings faces unique challenges due to widespread Korean-English code-switching, where clinicians routinely alternate between Korean conversational language and English medical terminology within single utterances. This study aimed to develop and evaluate a hybrid postprocessing approach combining medical terminology dictionary normalization with large language model (LLM)-based postprocessing to improve ASR accuracy for Korean-English code-switched medical speech. We constructed a speech dataset from 23,652 nursing progress notes, with a linguistic composition of 67.73% (512,626/756,866) Korean, 23.54% (178,166/756,866) English, and 8.73% (66,074/756,866) numerals or special symbols. Four Korean nurses recorded the notes using 5 microphone types in an acoustically isolated environment. Speech recognition was performed using OpenAI's gpt-4o-transcribe model. For postprocessing, a medical terminology dictionary containing 1070 mapping entries was constructed from 1000 nursing progress notes to normalize Korean phonetic renderings of English medical terms. Six LLMs (2 GPT and 4 Claude variants) were then evaluated across 5 temperature settings (0.0-0.8). Performance was assessed using BERTScore (bidirectional encoder representations from transformers score; F1), Sentence-BERT cosine similarity, word error rate, and character error rate (CER), comparing postprocessed outputs against the original written notes. Statistical significance was assessed using paired Wilcoxon signed-rank tests with Holm correction (α=.05). Temperature optimization showed that all postprocessing models had small temperature-related effect sizes (all |Cohen dz| ≤0.15), with GPT-4o exhibiting the largest dependency and statistically significant improvement at temperature 0.6 (Holm-adjusted P<.001 for both BERTScore and CER) and the 4 Claude variants and GPT-4.1 exhibiting practically consistent performance across all settings. Baseline ASR achieved a BERTScore of 0.9131 and CER of 0.2336. Dictionary-based normalization performed 43,507 word-level substitutions in 70.8% (16,754/23,652) of transcribed sentences. LLM-only postprocessing reduced CER by 36.09% (Claude Sonnet 4) and 32.53% (GPT-4o) compared to baseline. The combined dictionary-LLM approach achieved the best performance: Claude Sonnet 4 attained a BERTScore of 0.9638 and CER of 0.0820, representing a 64.9% reduction in CER from baseline (P<.001). The hybrid pipeline integrating rule-based dictionary normalization with LLM postprocessing significantly improved Korean-English code-switched medical ASR accuracy. Dictionary-based normalization yielded consistent additional improvements over LLM-only postprocessing for both GPT-4o and Claude Sonnet 4. As this modular framework requires no model retraining, it offers a practical means of mitigating multilingual challenges in medical ASR.