Background: The potential of artificial intelligence (AI)–based large language models, such as ChatGPT, has gained significant attention in the medical field. This enthusiasm is driven not only by recent breakthroughs and improved accessibility, but also by the prospect of democratizing medical knowledge and promoting equitable health care. However, the performance of ChatGPT is substantially influenced by the input language, and given the growing public trust in this AI tool compared to that in traditional sources of information, investigating its medical accuracy across different languages is of particular importance. Objective: This study aimed to compare the performance of GPT-3.5 and GPT-4 with that of medical students on the written German medical licensing examination. Methods: To assess GPT-3.5’s and GPT-4's medical proficiency, we used 937 original multiple-choice questions from 3 written German medical licensing examinations in October 2021, April 2022, and October 2022. Results: GPT-4 achieved an average score of 85% and ranked in the 92.8th, 99.5th, and 92.6th percentiles among medical students who took the same examinations in October 2021, April 2022, and October 2022, respectively. This represents a substantial improvement of 27% compared to GPT-3.5, which only passed 1 out of the 3 examinations. While GPT-3.5 performed well in psychiatry questions, GPT-4 exhibited strengths in internal medicine and surgery but showed weakness in academic research. Conclusions: The study results highlight ChatGPT’s remarkable improvement from moderate (GPT-3.5) to high competency (GPT-4) in answering medical licensing examination questions in German. While GPT-4’s predecessor (GPT-3.5) was imprecise and inconsistent, it demonstrates considerable potential to improve medical education and patient care, provided that medically trained users critically evaluate its results. As the replacement of search engines by AI tools seems possible in the future, further studies with nonprofessional questions are needed to assess the safety and accuracy of ChatGPT for the general population.
11 October 2010 in Journal of Bone and Mineral Research
The EU In-Vitro Diagnostic Device Regulation (IVDR) aims for transparent risk-and purpose-based validation of diagnostic devices, traceability of results to uniquely identified devices, and post-market surveillance. The IVDR regulates design, manufacture and putting into use of devices, but not medical services using these devices. In the absence of suitable commercial devices, the laboratory can resort to laboratory-developed tests (LDT) for in-house use. Documentary obligations (IVDR Art 5.5), the performance and safety specifications of ANNEX I, and development and manufacture under an ISO 15189-equivalent quality system apply. LDTs serve specific clinical needs, often for low volume niche applications, or correspond to the translational phase of new tests and treatments, often extremely relevant for patient care. As some commercial tests may disappear with the IVDR roll-out, many will require urgent LDT replacement. The workload will also depend on which modifications to commercial tests turns them into an LDT, and on how national legislators and competent authorities (CA) will handle new competences and responsibilities. We discuss appropriate interpretation of ISO 15189 to cover IVDR requirements. Selected cases illustrate LDT implementation covering medical needs with commensurate management of risk emanating from intended use and/or design of devices. Unintended collateral damage of the IVDR comprises loss of non-profitable niche applications, increases of costs and wasted resources, and migration of innovative research to more cost-efficient environments. Taking into account local specifics, the legislative framework should reduce the burden on and associated opportunity costs for the health care system, by making diligent use of existing frameworks.
Reference intervals (RIs) can be determined by direct and indirect procedures. Both approaches identify a reference population from which the RIs are defined. The crucial difference between direct and indirect methods is that direct methods select particular individuals after individual anamnesis and medical examination have confirmed the absence of pathological conditions. These individuals form a reference subpopulation. Indirect methods select a reference subpopulation in which the individuals are not identified. They isolate a reference population from a mixed population of patients with pathological and non-pathological conditions by statistical reasoning. At present, the direct procedure internationally recommended is the “gold standard”. It has, however, the disadvantage of high expenses which cannot easily be afforded by most medical laboratories. Therefore, laboratories adopt RIs established by direct methods from external sources requiring a high responsibility for transference problems which are usually neglected by most laboratories. These difficulties can be overcome by indirect procedures which can easily be performed by most laboratories without causing economic problems. The present review focuses on indirect approaches. Various procedures are presented with their benefits and limitations. Preliminary simulation studies indicate that more recently developed concepts are superior to older approaches.
29 May 2024 in Clinical Chemistry and Laboratory Medicine (CCLM)
Objectives: Laboratory medical reports are often not intuitively comprehensible to non-medical professionals. Given their recent advancements, easier accessibility and remarkable performance on medical licensing exams, patients are therefore likely to turn to artificial intelligence-based chatbots to understand their laboratory results. However, empirical studies assessing the efficacy of these chatbots in responding to real-life patient queries regarding laboratory medicine are scarce. Methods: Thus, this investigation included 100 patient inquiries from an online health forum, specifically addressing Complete Blood Count interpretation. The aim was to evaluate the proficiency of three artificial intelligence-based chatbots (ChatGPT, Gemini and Le Chat) against the online responses from certified physicians. Results: The findings revealed that the chatbots’ interpretations of laboratory results were inferior to those from online medical professionals. While the chatbots exhibited a higher degree of empathetic communication, they frequently produced erroneous or overly generalized responses to complex patient questions. The appropriateness of chatbot responses ranged from 51 to 64 %, with 22 to 33 % of responses overestimating patient conditions. A notable positive aspect was the chatbots’ consistent inclusion of disclaimers regarding its non-medical nature and recommendations to seek professional medical advice. Conclusions: The chatbots’ interpretations of laboratory results from real patient queries highlight a dangerous dichotomy – a perceived trustworthiness potentially obscuring factual inaccuracies. Given the growing inclination towards self-diagnosis using AI platforms, further research and improvement of these chatbots is imperative to increase patients’ awareness and avoid future burdens on the healthcare system.
In an increasingly interconnected health care system, laboratory medicine can facilitate diagnosis and treatment of patients effectively. This article describes necessary changes and points to potential challenges on a technical, content, and organizational level. As a technical precondition, electronic laboratory reports have to become machine-readable and interpretable. Terminologies such as Logical Observation Identifiers Names and Codes (LOINC), Nomenclature for Properties and Units (NPU), Unified Code for Units of Measure (UCUM), and SNOMED-CT can lead to the necessary semantic interoperability. Even if only single “atomized” results of the whole report are extracted, the necessary information for correct interpretation must be available. Therefore, interpretive comments, e.g., concerns about an increased measurement uncertainty must be electronically attached to every affected measurement result. Standardization of laboratory analyses with traceable standards and reference materials will enable knowledge transfer and safe interpretation of laboratory analyses from multiple laboratories. In an interconnected health care system, laboratories should strive to transform themselves into a data hub that not only receives samples but also extensive information about the patient. On that basis, they can return measurement results enriched with high-quality interpretive comments tailored to the individual patient and unlock the full potential of laboratory medicine.