Recent advances in large language models (LLMs) have demonstrated remarkable successes in zero- and few-shot performance on various downstream tasks, paving the way for applications in high-stakes domains. In this study, we systematically examine the capabilities and limitations of LLMs, specifically GPT-3.5 and ChatGPT, in performing zero-shot medical evidence summarization across six clinical domains. We conduct both automatic and human evaluations, covering several dimensions of summary quality. Our study demonstrates that automatic metrics often do not strongly correlate with the quality of summaries. Furthermore, informed by our human evaluations, we define a terminology of error types for medical evidence summarization. Our findings reveal that LLMs could be susceptible to generating factually inconsistent summaries and making overly convincing or uncertain statements, leading to potential harm due to misinformation. Moreover, we find that models struggle to identify the salient information and are more error-prone when summarizing over longer textual contexts.

PubMed^® is an essential resource for the medical domain, but useful concepts are either difficult to extract or are ambiguous, which has significantly hindered knowledge discovery. To address this issue, we constructed a PubMed knowledge graph (PKG) by extracting bio-entities from 29 million PubMed abstracts, disambiguating author names, integrating funding data through the National Institutes of Health (NIH) ExPORTER, collecting affiliation history and educational background of authors from ORCID^®, and identifying fine-grained affiliation data from MapAffil. Through the integration of these credible multi-source data, we could create connections among the bio-entities, authors, articles, affiliations, and funding. Data validation revealed that the BioBERT deep learning method of bio-entity extraction significantly outperformed the state-of-the-art models based on the F1 score (by 0.51%), with the author name disambiguation (AND) achieving an F1 score of 98.09%. PKG can trigger broader innovations, not only enabling us to measure scholarly impact, knowledge usage, and knowledge transfer, but also assisting us in profiling authors and organizations based on their connections with bio-entities.

Background: Among patients with nonvalvular atrial fibrillation (AF) and an elevated stroke risk, guidelines recommend direct oral anticoagulants (DOACs) over warfarin for stroke prevention. Changes in DOAC use over the past decade have not been well described. Methods and Results: We evaluated trends in use of DOACs and warfarin from 2011 to 2020 among adults with AF and a CHA ₂ DS ₂ ‐VASc score ≥2 based on electronic health record data from 88 health systems in the United States contributing to Cerner Real World Data. The use of DOACs and warfarin was described over time, by age, sex, race, and ethnicity, and at the health‐system level. We identified 436 864 patients with AF at risk for stroke (median age, 78 years; 52.1% men). From 2011 to 2020, overall anticoagulation rates increased from 56.3% to 64.7%, as DOAC use increased steadily (from 4.7% to 47.9%), while warfarin use declined (from 52.4% to 17.7%). DOAC uptake was similar across age, sex, and race and ethnicity groups but varied by health system. In 2020, the median health‐system‐level proportion of patients with AF on a DOAC was 49% (interquartile range, 40%–54%). Conclusions: Over the past decade, anticoagulation rates for patients with AF have increased modestly as DOACs largely replaced warfarin, though significant gaps remain: One in 3 high‐risk patients with AF is not on any anticoagulant. While DOAC adoption was generally consistent across major demographic groups, use between health systems remained highly variable, suggesting that provider and system factors influence DOAC uptake use more than patient‐level factors.

Recent studies indicate that Generative Pre-trained Transformer 4 with Vision (GPT-4V) outperforms human physicians in medical challenge tasks. However, these evaluations primarily focused on the accuracy of multi-choice questions alone. Our study extends the current scope by conducting a comprehensive analysis of GPT-4V’s rationales of image comprehension, recall of medical knowledge, and step-by-step multimodal reasoning when solving New England Journal of Medicine (NEJM) Image Challenges—an imaging quiz designed to test the knowledge and diagnostic capabilities of medical professionals. Evaluation results confirmed that GPT-4V performs comparatively to human physicians regarding multi-choice accuracy (81.6% vs. 77.8%). GPT-4V also performs well in cases where physicians incorrectly answer, with over 78% accuracy. However, we discovered that GPT-4V frequently presents flawed rationales in cases where it makes the correct final choices (35.5%), most prominent in image comprehension (27.2%). Regardless of GPT-4V’s high accuracy in multi-choice questions, our findings emphasize the necessity for further in-depth evaluations of its rationales before integrating such multimodal AI models into clinical workflows.