Background: Clinical problem-solving requires processing of semantic medical knowledge, such as illness scripts, and numerical medical knowledge of diagnostic tests for evidence-based decision-making. As large language models (LLMs) show promising results in many aspects of language-based clinical practice, their ability to generate nonlanguage evidence-based answers to clinical questions is inherently limited by tokenization. Objective: This study aimed to evaluate LLMs’ performance on two question types: numeric (correlating findings) and semantic (differentiating entities), while examining differences within and between LLMs in medical aspects and comparing their performance to humans. Methods: To generate straightforward multichoice questions and answers (Q and As) based on evidence-based medicine (EBM), we used a comprehensive medical knowledge graph (containing data from more than 50,000 peer-reviewed studies) and created the EBM questions and answers (EBMQAs). EBMQA comprises 105,222 Q and As, categorized by medical topics (eg, medical disciplines) and nonmedical topics (eg, question length), and classified into numerical or semantic types. We benchmarked a dataset of 24,000 Q and As on two state-of-the-art LLMs, GPT-4 (OpenAI) and Claude 3 Opus (Anthropic). We evaluated the LLM’s accuracy on semantic and numerical question types and according to sublabeled topics. In addition, we examined the question-answering rate of LLMs by enabling them to choose to abstain from responding to questions. For validation, we compared the results for 100 unrelated numerical EBMQA questions between six human medical experts and the two language models. Results: In an analysis of 24,542 Q and As, Claude 3 and GPT-4 performed better on semantic Q and As (68.7%, n=1593 and 68.4%, n=1709), respectively. Then on numerical Q and As (61.3%, n=8583 and 56.7%, n=12,038), respectively, with Claude 3 outperforming GPT-4 in numeric accuracy (P<.001). A median accuracy gap of 7% (IQR 5%‐10%) was observed between the best and worst sublabels per topic, with different LLMs excelling in different sublabels. Focusing on Medical Discipline sublabels, Claude 3 performed well in neoplastic disorders but struggled with genitourinary disorders (69%, n=676 vs 58%, n=464; P<.0001), while GPT-4 excelled in cardiovascular disorders but struggled with neoplastic disorders (60%, n=1076 vs 53%, n=704; P=.0002). Furthermore, humans (82.3%, n=82.3) surpassed both Claude 3 (64.3%, n=64.3; P<.001) and GPT-4 (55.8%, n=55.8; P<.001) in the validation test. Spearman correlation between question-answering and accuracy rate in both Claude 3 and GPT-4 was insignificant (ρ=0.12, P=.69; ρ=0.43, P=.13). Conclusions: Both LLMs excelled more in semantic than numerical Q and As, with Claude 3 surpassing GPT-4 in numerical Q and As. However, both LLMs showed inter- and intramodel gaps in different medical aspects and remained inferior to humans. In addition, their ability to respond or abstain from answering a question does not reliably predict how accurately they perform when they do attempt to answer questions. Thus, their medical advice should be addressed carefully.

Anosmia is common in COVID-19 patients, lasting for weeks or months following recovery. The biological mechanism underlying olfactory deficiency in COVID-19 does not involve direct damage to nasal olfactory neurons, which do not express the proteins required for SARS-CoV-2 infection. A recent study suggested that anosmia results from downregulation of olfactory receptors. We hypothesized that anosmia in COVID-19 may also reflect SARS-CoV-2 infection-driven elevated expression of regulator of G protein signaling 2 (RGS2), a key regulator of odorant receptors, thereby silencing their signaling. To test our hypothesis, we analyzed gene expression of nasopharyngeal swabs from SARS-CoV-2 positive patients and non-infected controls (two published RNA-sequencing datasets, 580 individuals). Our analysis found upregulated RGS2 expression in SARS-CoV-2 positive patients (FC = 14.5, Padj = 1.69 × 10⁻⁵ and FC = 2.4; Padj = 0.001, per dataset). Additionally, RGS2 expression was strongly correlated with PTGS2, IL1B, CXCL8, NAMPT and other inflammation markers with substantial upregulation in early infection. These observations suggest that upregulated expression of RGS2 may underlie anosmia in COVID-19 patients. As a regulator of numerous G-protein coupled receptors, RGS2 may drive further neurological symptoms of COVID-19. Studies are required for clarifying the cellular mechanisms by which SARS-CoV-2 infection drives the upregulation of RGS2 and other genes implicated in inflammation. Insights on these pathway(s) may assist in understanding anosmia and additional neurological symptoms reported in COVID-19 patients.

Background: Druze individuals, like many genetically homogeneous and isolated populations, harbor recurring pathogenic variants (PV) in autosomal recessive (AR) disorders. Methods: Variant calling of whole-genome sequencing (WGS) of 40 Druze from the Human Genome Diversity Project (HGDP) was performed (HGDP-cohort). Additionally, we performed whole exome sequencing (WES) of 118 Druze individuals: 38 trios and 2 couples, representing geographically distinct clans (WES-cohort). Rates of validated PV were compared with rates in worldwide and Middle Eastern populations, from the gnomAD and dbSNP datasets. Results: Overall, 34 PVs were identified: 30 PVs in genes underlying AR disorders, 3 additional PVs were associated with autosomal dominant (AD) disorders, and 1 PV with X-linked-dominant inherited disorder in the WES cohort. Conclusions: The newly identified PVs associated with AR conditions should be considered for incorporation into prenatal-screening options offered to Druze individuals after an extension and validation of the results in a larger study.

Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Evidence-based clinical intake tools (EBCITs) are structured assessment tools used to gather information about patients and help healthcare providers make more informed decisions. The growing demand for personalized medicine, along with the big data revolution, have rendered EBCITs a promising solution. EBCITs have the potential to provide comprehensive and individualized assessments of symptoms, enabling accurate and timely diagnosis, while contributing to the grounding of medical care. Objective: This work examines whether EBCITs cover data concerning disorders and symptoms to a similar extent as physicians, thus can reliably address medical conditions in clinical settings. We also explore the potential of EBCITs to discover and ground real prevalence of symptoms in different disorders thereby expanding medical knowledge and further support medical diagnoses made by physicians. Methods: Between August 1, 2022, and January 15, 2023, patients who used the services of a virtual healthcare (VH) provider in the USA were first assessed by the Kahun EBCIT. Kahun platform gathered, documented, and analyzed the information from the sessions and its clinical findings. In this study, we estimated the prevalence of patients' symptoms in medical disorders, using two datasets. The first set analyzed symptoms prevalence, as determined by the Kahun's knowledge engine. The second set analyzed symptoms prevalence, relying solely on data from the VH patients gathered by Kahun. The difference in variance between these two prevalence datasets, helped us assess Kahun's ability to incorporate new data, while integrating existing knowledge. To analyze the comprehensiveness of the Kahun's knowledge engine, we compared how well it covers weighted data for the symptoms and disorders found in the 2019 National Ambulatory Medical Care Survey (NAMCS). To assess Kahun's diagnosis accuracy, physicians independently diagnosed 250 of Kahun-VH’s sessions. Their diagnoses were compared with Kahun's diagnoses. Results: As part of this work, 2,550 patients used Kahun to complete a full assessment, among them 1,714 females and 836 males. Kahun collected 314 different chief complaints and proposed 108,523 suggestions related to symptoms during the intake process. At the end of the intake process, 6,496 conditions were presented to the caregiver. Kahun covered 94% (526,157,569/562,150,572) of the weighted symptoms and 91% (1,582,637,476/173,4783,244) of the weighted disorders in NAMCS 2019. In 90% (224/250) of the sessions, at least one identical disorder suggested by both the physicians and Kahun, with total accuracy rate of 72% (367/507). Kahun’s engine yielded 519 prevalences while the Kahun-VH cohort yielded 599; 156 prevalences were unique to the latter and 443 prevalences were shared by both databases. Conclusions: ECBITs, such as Kahun, encompass extensive amounts of knowledge and could serve as a reliable database for inferring medical insights and diagnosis. Using this credible database, potential prevalence of symptoms in different disorders were discovered or grounded. This highlights the ability of ECBITs to refine the understanding of relationships between disorders and symptoms, which further supports physicians in medical diagnosis.

BACKGROUND Evidence-based clinical intake tools (EBCITs) are structured assessment tools used to gather information about patients and help health care providers make informed decisions. The growing demand for personalized medicine, along with the big data revolution, has rendered EBCITs a promising solution. EBCITs have the potential to provide comprehensive and individualized assessments of symptoms, enabling accurate diagnosis, while contributing to the grounding of medical care. OBJECTIVE This work aims to examine whether EBCITs cover data concerning disorders and symptoms to a similar extent as physicians, and thus can reliably address medical conditions in clinical settings. We also explore the potential of EBCITs to discover and ground the real prevalence of symptoms in different disorders thereby expanding medical knowledge and further supporting medical diagnoses made by physicians. METHODS Between August 1, 2022, and January 15, 2023, patients who used the services of a digital health care (DH) provider in the United States were first assessed by the Kahun EBCIT. Kahun platform gathered and analyzed the information from the sessions. This study estimated the prevalence of patients’ symptoms in medical disorders using 2 data sets. The first data set analyzed symptom prevalence, as determined by Kahun’s knowledge engine. The second data set analyzed symptom prevalence, relying solely on data from the DH patients gathered by Kahun. The variance difference between these 2 prevalence data sets helped us assess Kahun’s ability to incorporate new data while integrating existing knowledge. To analyze the comprehensiveness of Kahun’s knowledge engine, we compared how well it covers weighted data for the symptoms and disorders found in the 2019 National Ambulatory Medical Care Survey (NMCAS). To assess Kahun’s diagnosis accuracy, physicians independently diagnosed 250 of Kahun-DH’s sessions. Their diagnoses were compared with Kahun’s diagnoses. RESULTS In this study, 2550 patients used Kahun to complete a full assessment. Kahun proposed 108,523 suggestions related to symptoms during the intake process. At the end of the intake process, 6496 conditions were presented to the caregiver. Kahun covered 94% (526,157,569/562,150,572) of the weighted symptoms and 91% (1,582,637,476/173,4783,244) of the weighted disorders in the 2019 NMCAS. In 90% (224/250) of the sessions, both physicians and Kahun suggested at least one identical disorder, with a 72% (367/507) total accuracy rate. Kahun’s engine yielded 519 prevalences while the Kahun-DH cohort yielded 599; 156 prevalences were unique to the latter and 443 prevalences were shared by both data sets. CONCLUSIONS ECBITs, such as Kahun, encompass extensive amounts of knowledge and could serve as a reliable database for inferring medical insights and diagnoses. Using this credible database, the potential prevalence of symptoms in different disorders was discovered or grounded. This highlights the ability of ECBITs to refine the understanding of relationships between disorders and symptoms, which further supports physicians in medical diagnosis.