Mehr Kashyap

Mehr Kashyap

Auto-generated profile, based on publicly available information

Mehr Kashyap

Department of Medicine, Stanford University School of Medicine, Stanford, Calif

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
16 April 2024 in medRxiv
Verified
Preprint
Open access
PDF file
SBSuhana Bedi
Suhana Bedi
YLYutong Liu
Yutong Liu
LOLucy Orr-Ewing
Lucy Orr-Ewing
DDDebadutta Dash
Debadutta Dash
SKSanmi Koyejo
Sanmi Koyejo
OKOluwasanmi Koyejo
Oluwasanmi Koyejo
1 Importance: Large Language Models (LLMs) can assist in a wide range of healthcare-related activities. Current approaches to evaluating LLMs make it difficult to identify the most impactful LLM application areas. Objective: To summarize the current evaluation of LLMs in healthcare in terms of 5 components: evaluation data type, healthcare task, Natural Language Processing (NLP)/Natural Language Understanding (NLU) task, dimension of evaluation, and medical specialty. Data Sources: A systematic search of PubMed and Web of Science was performed for studies published between 01-01-2022 and 02-19-2024. Study Selection: Studies evaluating one or more LLMs in healthcare. Data Extraction and Synthesis: Three independent reviewers categorized 519 studies in terms of data used in the evaluation, the healthcare tasks (the what) and the NLP/NLU tasks (the how) examined, the dimension(s) of evaluation, and the medical specialty studied. Results: Only 5% of reviewed studies utilized real patient care data for LLM evaluation. The most popular healthcare tasks were assessing medical knowledge (e.g. answering medical licensing exam questions, 44.5%), followed by making diagnoses (19.5%), and educating patients (17.7%). Administrative tasks such as assigning provider billing codes (0.2%), writing prescriptions (0.2%), generating clinical referrals (0.6%) and clinical notetaking (0.8%) were less studied. For NLP/NLU tasks, the vast majority of studies examined question answering (84.2%). Other tasks such as summarization (8.9%), conversational dialogue (3.3%), and translation (3.1%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias and toxicity (15.8%), robustness (14.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in internal medicine (42%), surgery (11.4%) and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%) and medical genetics (0.2%) being the least represented. Conclusions and Relevance: Existing evaluations of LLMs mostly focused on accuracy of question answering for medical exams, without consideration of real patient care data. Dimensions like fairness, bias and toxicity, robustness, and deployment considerations received limited attention. To draw meaningful conclusions and improve LLM adoption, future studies need to establish a standardized set of LLM applications and evaluation dimensions, perform evaluations using data from routine care, and broaden testing to include administrative tasks as well as multiple medical specialties. Key Points: Question: How are healthcare applications of large language models (LLMs) currently evaluated? Findings: Studies rarely used real patient care data for LLM evaluation. Administrative tasks such as generating provider billing codes and writing prescriptions were understudied. Natural Language Processing (NLP)/Natural Language Understanding (NLU) tasks like summarization, conversational dialogue, and translation were infrequently explored. Accuracy was the predominant dimension of evaluation, while fairness, bias and toxicity assessments were neglected. Evaluations in specialized fields, such as nuclear medicine and medical genetics were rare. Meaning: Current LLM assessments in healthcare remain shallow and fragmented. To draw concrete insights on their performance, evaluations need to use real patient care data across a broad range of healthcare and NLP/NLU tasks and medical specialties with standardized dimensions of evaluation.

1 January 2025 in Radiology
Verified
Journal Article
Closed access
MKMehr Kashyap
Mehr Kashyap
XWXi Wang
Xi Wang
NPNeil Panjwani
Neil Panjwani
MHMohammad Hasan
Mohammad Hasan
QZQin Zhang
Qin Zhang
MGMichael F. Gensheimer
Michael F. Gensheimer
Using routine pre–radiation treatment CT simulation scans and tumor segmentation data, a deep learning model was developed to detect and segment lung tumors, and good performance was achieved on heterogeneous datasets.

6 May 2020 in Journal of the American Medical Informatics Association
Verified
Journal Article
Open access
PDF file
MKMehr Kashyap
Mehr Kashyap
MSMartin Seneviratne
Martin Seneviratne
JBJuan Banda
Juan Banda
TFThomas Falconer
Thomas Falconer
BRBorim Ryu
Borim Ryu
NSNigam H Shah
Nigam H Shah
Objective: Accurate electronic phenotyping is essential to support collaborative observational research. Supervised machine learning methods can be used to train phenotype classifiers in a high-throughput manner using imperfectly labeled data. We developed 10 phenotype classifiers using this approach and evaluated performance across multiple sites within the Observational Health Data Sciences and Informatics (OHDSI) network. Materials and Methods: We constructed classifiers using the Automated PHenotype Routine for Observational Definition, Identification, Training and Evaluation (APHRODITE) R-package, an open-source framework for learning phenotype classifiers using datasets in the Observational Medical Outcomes Partnership Common Data Model. We labeled training data based on the presence of multiple mentions of disease-specific codes. Performance was evaluated on cohorts derived using rule-based definitions and real-world disease prevalence. Classifiers were developed and evaluated across 3 medical centers, including 1 international site. Results: Compared to the multiple mentions labeling heuristic, classifiers showed a mean recall boost of 0.43 with a mean precision loss of 0.17. Performance decreased slightly when classifiers were shared across medical centers, with mean recall and precision decreasing by 0.08 and 0.01, respectively, at a site within the USA, and by 0.18 and 0.10, respectively, at an international site. Discussion and Conclusion: We demonstrate a high-throughput pipeline for constructing and sharing phenotype classifiers across sites within the OHDSI network using APHRODITE. Classifiers exhibit good portability between sites within the USA, however limited portability internationally, indicating that classifier generalizability may have geographic limitations, and, consequently, sharing the classifier-building recipe, rather than the pretrained classifiers, may be more useful for facilitating collaborative observational research.

1 November 2020 in Radiotherapy and Oncology
Verified
Journal Article
Closed access
DTDiego A.S. Toesca
Diego A.S. Toesca
FAFaisal Ahmed
Faisal Ahmed
MKMehr Kashyap
Mehr Kashyap
JBJ Richelcyn M. Baclay
J Richelcyn M. Baclay
RERie von Eyben
Rie von Eyben
DCDaniel T Chang
Daniel T Chang

6 October 2020 in Cancer Medicine
Verified
Journal Article
Open access
PDF file
JHJeremy P. Harris
Jeremy P. Harris
MKMehr Kashyap
Mehr Kashyap
JHJessica N. Humphreys
Jessica N. Humphreys
EPErqi L. Pollom
Erqi L. Pollom
DCDaniel T Chang
Daniel T Chang
The clinical and financial effects of mental disorders are largely unknown among gastrointestinal (GI) cancer patients. Using the Surveillance, Epidemiology, and End Results (SEER)‐Medicare linked database, we identified patients whose first cancer was a primary colorectal, pancreatic, gastric, hepatic/biliary, esophageal, or anal cancer as well as those with coexisting depression, anxiety, psychotic, or bipolar disorder. Survival, chemotherapy use, total healthcare expenditures, and patient out‐of‐pocket expenditures were estimated and compared based on the presence of a mental disorder. We identified 112,283 patients, 23,726 (21%) of whom had a coexisting mental disorder. Median survival for patients without a mental disorder was 52 months (95% CI 50–53 months) and for patients with a mental disorder was 43 months (95% CI 42–44 months) (p < 0.001). Subgroup analysis identified patients with colorectal, gastric, or anal cancer to have a significant association between survival and presence of a mental disorder. Chemotherapy use was lower among patients with a mental disorder within regional colorectal cancer (43% vs. 41%, p = 0.01) or distant colorectal cancer subgroups (71% vs. 63%, p < 0.0001). The mean total healthcare expenditures were higher for patients with a mental disorder in first year following the cancer diagnosis (increase of $16,823, 95% CI $15,777‐$18,173), and mean patient out‐of‐pocket expenses were also higher (increase of $1,926, 95% CI $1753–$2091). There are a substantial number of GI cancer patients who have a coexisting mental disorder, which is associated with inferior survival, higher healthcare expenditures, and greater personal financial burden.

23 February 2021 in Cancer Medicine
Verified
Journal Article
Open access
PDF file
MKMehr Kashyap
Mehr Kashyap
JHJeremy P. Harris
Jeremy P. Harris
DCDaniel T Chang
Daniel T Chang
EPErqi L. Pollom
Erqi L. Pollom
Background: Elderly patients with gastrointestinal cancer and mental illness have significant comorbidities that can impact the quality of their care. We investigated the relationship between mental illness and frequent emergency department (ED) use in the last month of life, an indicator for poor end‐of‐life care quality, among elderly patients with gastrointestinal cancers.Methods: We used SEER‐Medicare data to identify decedents with gastrointestinal cancers who were diagnosed between 2004 and 2013 and were at least 66 years old at time of diagnosis (median age: 80 years, range: 66–117 years). We evaluated the association between having a diagnosis of depression, bipolar disorders, psychotic disorders, anxiety, dementia, and/or substance use disorders and ED use in the last 30 days of life using logistic regression models.Results: Of 160,367 patients included, 54,661 (34.1%) had a mental illness diagnosis between one year prior to cancer diagnosis and death. Patients with mental illness were more likely to have > 1 ED visit in the last 30 days of life (15.6% vs. 13.3%, p < 0.01). ED use was highest among patients with substance use (17.7%), bipolar (16.5%), and anxiety disorders (16.4%). Patients with mental illness who were male, younger, non‐white, residing in lower income areas, and with higher comorbidity were more likely to have multiple end‐of‐life ED visits. Patients who received outpatient treatment from a mental health professional were less likely to have multiple end‐of‐life ED visits (adjusted odds ratio 0.82, 95% confidence interval 0.78–0.87).Conclusions: In elderly patients with gastrointestinal cancers, mental illness is associated with having multiple end‐of‐life ED visits. Increasing access to mental health services may improve quality of end‐of‐life care in this vulnerable population.