Artificial intelligence chatbots driven by large language models (LLMs) have the potential to increase public science literacy and support scientific research, as they can quickly summarize complex scientific information in accessible terms. However, when summarizing scientific texts, LLMs may omit details that limit the scope of research conclusions, leading to generalizations of results broader than warranted by the original study. We tested 10 prominent LLMs, including ChatGPT-4o, ChatGPT-4.5, DeepSeek, LLaMA 3.3 70B, and Claude 3.7 Sonnet, comparing 4900 LLM-generated summaries to their original scientific texts. Even when explicitly prompted for accuracy, most LLMs produced broader generalizations of scientific results than those in the original texts, with DeepSeek, ChatGPT-4o, and LLaMA 3.3 70B overgeneralizing in 26–73% of cases. In a direct comparison of LLM-generated and human-authored science summaries, LLM summaries were nearly five times more likely to contain broad generalizations (odds ratio = 4.85, 95% CI [3.06, 7.70], p < 0.001). Notably, newer models tended to perform worse in generalization accuracy than earlier ones. Our results indicate a strong bias in many widely used LLMs towards overgeneralizing scientific conclusions, posing a significant risk of large-scale misinterpretations of research findings. We highlight potential mitigation strategies, including lowering LLM temperature settings and benchmarking LLMs for generalization accuracy.
1 November 2022 in Journal of Medical Internet Research
Background: The field of oncology is at the forefront of advances in artificial intelligence (AI) in health care, providing an opportunity to examine the early integration of these technologies in clinical research and patient care. Hope that AI will revolutionize health care delivery and improve clinical outcomes has been accompanied by concerns about the impact of these technologies on health equity.Objective: We aimed to conduct a scoping review of the literature to address the question, “What are the current and potential impacts of AI technologies on health equity in oncology?”Methods: Following PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines for scoping reviews, we systematically searched MEDLINE and Embase electronic databases from January 2000 to August 2021 for records engaging with key concepts of AI, health equity, and oncology. We included all English-language articles that engaged with the 3 key concepts. Articles were analyzed qualitatively for themes pertaining to the influence of AI on health equity in oncology.Results: Of the 14,011 records, 133 (0.95%) identified from our review were included. We identified 3 general themes in the literature: the use of AI to reduce health care disparities (58/133, 43.6%), concerns surrounding AI technologies and bias (16/133, 12.1%), and the use of AI to examine biological and social determinants of health (55/133, 41.4%). A total of 3% (4/133) of articles focused on many of these themes.Conclusions: Our scoping review revealed 3 main themes on the impact of AI on health equity in oncology, which relate to AI’s ability to help address health disparities, its potential to mitigate or exacerbate bias, and its capability to help elucidate determinants of health. Gaps in the literature included a lack of discussion of ethical challenges with the application of AI technologies in low- and middle-income countries, lack of discussion of problems of bias in AI algorithms, and a lack of justification for the use of AI technologies over traditional statistical methods to address specific research questions in oncology. Our review highlights a need to address these gaps to ensure a more equitable integration of AI in cancer research and clinical practice. The limitations of our study include its exploratory nature, its focus on oncology as opposed to all health care sectors, and its analysis of solely English-language articles.
7 February 2020 in Journal of Evaluation in Clinical Practice
BACKGROUND: The role of humoral immunity in COVID-19 is not fully understood, owing, in large part, to the complexity of antibodies produced in response to the SARS-CoV-2 infection. There is a pressing need for serology tests to assess patient-specific antibody response and predict clinical outcome. METHODS: Using SARS-CoV-2 proteome and peptide microarrays, we screened 146 COVID-19 patients’ plasma samples to identify antigens and epitopes. This enabled us to develop a master epitope array and an epitope-specific agglutination assay to gauge antibody responses systematically and with high resolution. RESULTS: We identified linear epitopes from the spike (S) and nucleocapsid (N) proteins and showed that the epitopes enabled higher resolution antibody profiling than the S or N protein antigen. Specifically, we found that antibody responses to the S-811–825, S-881–895, and N-156–170 epitopes negatively or positively correlated with clinical severity or patient survival. Moreover, we found that the P681H and S235F mutations associated with the coronavirus variant of concern B.1.1.7 altered the specificity of the corresponding epitopes. CONCLUSION: Epitope-resolved antibody testing not only affords a high-resolution alternative to conventional immunoassays to delineate the complex humoral immunity to SARS-CoV-2 and differentiate between neutralizing and non-neutralizing antibodies, but it also may potentially be used to predict clinical outcome. The epitope peptides can be readily modified to detect antibodies against variants of concern in both the peptide array and latex agglutination formats. FUNDING: Ontario Research Fund (ORF) COVID-19 Rapid Research Fund, Toronto COVID-19 Action Fund, Western University, Lawson Health Research Institute, London Health Sciences Foundation, and Academic Medical Organization of Southwestern Ontario (AMOSO) Innovation Fund.
Secondary erythrocytosis refers to an elevation in hemoglobin or hematocrit due to elevated serum erythropoietin levels. Medications including testosterone and sodium-glucose cotransporter-2 (SGLT-2) inhibitors are increasingly recognized as causes of secondary erythrocytosis. We conducted a systematic review to inform the clinical management of drug-induced erythrocytosis. Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we performed a systematic literature search in MEDLINE, EMBASE, CENTRAL (all via Ovid), and Google Scholar. Of the 2036 articles screened for eligibility, 45 studies were included in our review, with 35 studies on testosterone and other androgen use, 5 studies on SGLT-2 inhibitors, 3 studies on antiangiogenic tyrosine kinase inhibitors (TKIs), 1 study on erythropoiesis-stimulating agents, and 1 study on a treatment regimen for multidrug-resistant tuberculosis. Cisgender and transgender men on prescription testosterone had erythrocytosis rates of up to 66.7%, with intramuscular formulations, higher doses, and older age associated with increased risk of erythrocytosis. Up to 2.7% of men on testosterone therapy developed thromboembolic events. Among individuals on SGLT-2 inhibitors, erythrocytosis rates ranged from 2.1% to 22%, with those who discontinued therapy demonstrating improvement or resolution of erythrocytosis. Thromboembolic events were reported in up to 10% of these individuals. Antiangiogenic TKIs were studied in patients with cancer, with erythrocytosis developing in up to 43.5% of patients. Drug-induced erythrocytosis is a heterogeneous condition for which there is no clear consensus among clinicians about its diagnosis and management. We offer recommendations for clinical practice within the scope of this systematic review, although further research is required.