14 January 2025 in Nature Communications
Large Language Models have demonstrated expert-level accuracy on medical board examinations, suggesting potential for clinical decision support systems. However, their metacognitive abilities, crucial for medical decision-making, remain largely unexplored. To address this gap, we developed MetaMedQA, a benchmark incorporating confidence scores and metacognitive tasks into multiple-choice medical questions. We evaluated twelve models on dimensions including confidence-based accuracy, missing answer recall, and unknown recall. Despite high accuracy on multiple-choice questions, our study revealed significant metacognitive deficiencies across all tested models. Models consistently failed to recognize their knowledge limitations and provided confident answers even when correct options were absent. In this work, we show that current models exhibit a critical disconnect between perceived and actual capabilities in medical reasoning, posing significant risks in clinical settings. Our findings emphasize the need for more robust evaluation frameworks that incorporate metacognitive abilities, essential for developing reliable Large Language Model enhanced clinical decision support systems.
1 February 2018 in Investigative Ophthalmology & Visual Science
Purpose: Sweep visual evoked potentials (sVEPs) provide an implicit, objective, and sensitive evaluation of low-level visual functions such as visual acuity and contrast sensitivity. For practical and traditional reasons, sVEPs in ophthalmologic examinations have usually been recorded over a single or a limited number of electrodes over the medial occipital region. Here we examined whether a higher density of recording electrodes improves the estimation of individual low-level visual thresholds with sVEPS, and to which extent such testing could be streamlined for clinical application. Methods: To this end, we tested contrast sensitivity and visual acuity in 26 healthy adult volunteers with a 68-electrode electroencephalogram (EEG) system. Results: While the most sensitive electrophysiologic response was found at the traditional medial occipital electrode Oz in a small majority of individuals, it was found at neighboring electrodes for the remaining participants. At the group level, lower spatial frequencies were also associated with right lateralized responses. More generally, visual function was evaluated more sensitively based on EEG recorded at the most sensitive electrode defined individually for each participant. Our data suggest that recording over seven posterior electrodes while limiting the testing session to less than 15 minutes ensures a sensitive and consistent estimation of acuity and contrast sensitivity threshold estimates in every individual. Conclusions: The present study shows that sampling from a larger number of posterior scalp electrodes is relevant to optimize visual function assessment and could be achieved efficiently in the time-constrained clinical setting.
23 May 2024 in Journal of the American Medical Informatics Association
Objective: To optimize the training strategy of large language models for medical applications, focusing on creating clinically relevant systems that efficiently integrate into healthcare settings, while ensuring high standards of accuracy and reliability. Materials and Methods: We curated a comprehensive collection of high-quality, domain-specific data and used it to train several models, each with different subsets of this data. These models were rigorously evaluated against standard medical benchmarks, such as the USMLE, to measure their performance. Furthermore, for a thorough effectiveness assessment, they were compared with other state-of-the-art medical models of comparable size. Results: The models trained with a mix of high-quality, domain-specific, and general data showed superior performance over those trained on larger, less clinically relevant datasets (P < .001). Our 7-billion-parameter model Med5 scores 60.5% on MedQA, outperforming the previous best of 49.3% from comparable models, and becomes the first of its size to achieve a passing score on the USMLE. Additionally, this model retained its proficiency in general domain tasks, comparable to state-of-the-art general domain models of similar size. Discussion: Our findings underscore the importance of integrating high-quality, domain-specific data in training large language models for medical purposes. The balanced approach between specialized and general data significantly enhances the model’s clinical relevance and performance. Conclusion: This study sets a new standard in medical language models, proving that a strategically trained, smaller model can outperform larger ones in clinical relevance and general proficiency, highlighting the importance of data quality and expert curation in generative artificial intelligence for healthcare applications.
1 April 2020 in Journal of American Association for Pediatric Ophthalmology and Strabismus
22 May 2020 in Ophthalmic Epidemiology
To describe the vision screening procedures in the three Belgian linguistic communities and present Belgian screening results. Analyses were carried out on 636 260 Belgian children between 2009 and...
29 July 2023 in American Journal of Ophthalmology