Arnold Milstein

Arnold Milstein

Auto-generated profile, based on publicly available information

Arnold Milstein

Stanford Clinical Excellence Research Center, Stanford University, Stanford, USA

Research Focus & Metrics

Academic Career Length

15Years (2012-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
9 September 2020 in Nature
Verified
Journal Article
Closed access
PDF file
AHAlbert Haque
Albert Haque
AMArnold Milstein
Arnold Milstein
LFLi Fei-Fei
Li Fei-Fei

5 February 2025 in Nature Medicine
Verified
Journal Article
Closed access
PDF file
EGEthan Goh
Ethan Goh
RGRobert J. Gallo
Robert J. Gallo
ESEric Strong
Eric Strong
YWYingjie Weng
Yingjie Weng
HKHannah Kerman
Hannah Kerman
ARAdam Rodman
Adam Rodman

22 February 2021 in Proceedings of the National Academy of Sciences
Verified
Journal Article
Closed access
PDF file
RKRobert M. Kaplan
Robert M. Kaplan
AMArnold Milstein
Arnold Milstein
Significance: Acceptance of vaccines has been on the decline in recent years. Despite encouraging early results for coronavirus vaccine trials, achieving herd immunity requires substantial uptake. We presented scenarios varying vaccine efficacy, minor side effects, and severe reactions to a sample representative of the US population. Vaccine acceptance improved when the efficacy increased beyond 70%. Respondents were unaffected by the probability of minor side effects, such as a sore arm or fever lasting 24 h. The chances of accepting the vaccine were lower when the probability of serious adverse reactions was 1/100,000 in contrast to 1/million or 1/100 million. A replication showed that the results were largely unchanged following the public announcement that the vaccines were 95% effective.

3 June 2020 in npj Digital Medicine
Verified
Journal Article
Open access
PDF file
AMAdam S. Miner
Adam S. Miner
AHAlbert Haque
Albert Haque
JFJason A. Fries
Jason A. Fries
SFScott L. Fleming
Scott L. Fleming
DWDenise E. Wilfley
Denise E. Wilfley
NSNigam H. Shah
Nigam H. Shah
Accurate transcription of audio recordings in psychotherapy would improve therapy effectiveness, clinician training, and safety monitoring. Although automatic speech recognition software is commercially available, its accuracy in mental health settings has not been well described. It is unclear which metrics and thresholds are appropriate for different clinical use cases, which may range from population descriptions to individual safety monitoring. Here we show that automatic speech recognition is feasible in psychotherapy, but further improvements in accuracy are needed before widespread use. Our HIPAA-compliant automatic speech recognition system demonstrated a transcription word error rate of 25%. For depression-related utterances, sensitivity was 80% and positive predictive value was 83%. For clinician-identified harm-related sentences, the word error rate was 34%. These results suggest that automatic speech recognition may support understanding of language patterns and subgroup variation in existing treatments but may not be ready for individual-level safety surveillance.

1 August 2015 in The Journal of Hand Surgery
Verified
Journal Article
Closed access
CNChristine Nguyen
Christine Nguyen
AMArnold Milstein
Arnold Milstein
THTina Hernandez-Boussard
Tina Hernandez-Boussard
CCCatherine M. Curtin
Catherine M. Curtin

16 April 2024 in medRxiv
Verified
Preprint
Open access
PDF file
SBSuhana Bedi
Suhana Bedi
YLYutong Liu
Yutong Liu
LOLucy Orr-Ewing
Lucy Orr-Ewing
DDDebadutta Dash
Debadutta Dash
SKSanmi Koyejo
Sanmi Koyejo
OKOluwasanmi Koyejo
Oluwasanmi Koyejo
1 Importance: Large Language Models (LLMs) can assist in a wide range of healthcare-related activities. Current approaches to evaluating LLMs make it difficult to identify the most impactful LLM application areas. Objective: To summarize the current evaluation of LLMs in healthcare in terms of 5 components: evaluation data type, healthcare task, Natural Language Processing (NLP)/Natural Language Understanding (NLU) task, dimension of evaluation, and medical specialty. Data Sources: A systematic search of PubMed and Web of Science was performed for studies published between 01-01-2022 and 02-19-2024. Study Selection: Studies evaluating one or more LLMs in healthcare. Data Extraction and Synthesis: Three independent reviewers categorized 519 studies in terms of data used in the evaluation, the healthcare tasks (the what) and the NLP/NLU tasks (the how) examined, the dimension(s) of evaluation, and the medical specialty studied. Results: Only 5% of reviewed studies utilized real patient care data for LLM evaluation. The most popular healthcare tasks were assessing medical knowledge (e.g. answering medical licensing exam questions, 44.5%), followed by making diagnoses (19.5%), and educating patients (17.7%). Administrative tasks such as assigning provider billing codes (0.2%), writing prescriptions (0.2%), generating clinical referrals (0.6%) and clinical notetaking (0.8%) were less studied. For NLP/NLU tasks, the vast majority of studies examined question answering (84.2%). Other tasks such as summarization (8.9%), conversational dialogue (3.3%), and translation (3.1%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias and toxicity (15.8%), robustness (14.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in internal medicine (42%), surgery (11.4%) and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%) and medical genetics (0.2%) being the least represented. Conclusions and Relevance: Existing evaluations of LLMs mostly focused on accuracy of question answering for medical exams, without consideration of real patient care data. Dimensions like fairness, bias and toxicity, robustness, and deployment considerations received limited attention. To draw meaningful conclusions and improve LLM adoption, future studies need to establish a standardized set of LLM applications and evaluation dimensions, perform evaluations using data from routine care, and broaden testing to include administrative tasks as well as multiple medical specialties. Key Points: Question: How are healthcare applications of large language models (LLMs) currently evaluated? Findings: Studies rarely used real patient care data for LLM evaluation. Administrative tasks such as generating provider billing codes and writing prescriptions were understudied. Natural Language Processing (NLP)/Natural Language Understanding (NLU) tasks like summarization, conversational dialogue, and translation were infrequently explored. Accuracy was the predominant dimension of evaluation, while fairness, bias and toxicity assessments were neglected. Evaluations in specialized fields, such as nuclear medicine and medical genetics were rare. Meaning: Current LLM assessments in healthcare remain shallow and fragmented. To draw concrete insights on their performance, evaluations need to use real patient care data across a broad range of healthcare and NLP/NLU tasks and medical specialties with standardized dimensions of evaluation.