David Talby

David Talby

Auto-generated profile, based on publicly available information

David Talby

John Snow Labs Inc, 16192 Coastal Highway, Lewes, DE, 19958, United States, +1 (302) 786-5227

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
4 December 2025 in JMIR AI
Verified
Journal Article
Open access
VKVeysel Kocaman
Veysel Kocaman
MKMustafa Aytuğ Kaya
Mustafa Aytuğ Kaya
Andrei Marian FeierAndrei Marian Feier
Andrei Marian Feier
DTDavid Talby
David Talby
Background: The proliferation of both general purpose and health care–specific large language models (LLMs) has intensified the challenge of effectively evaluating and comparing them. Data contamination plagues the validity of public benchmarks, self-preference distorts LLM-as-a-judge approaches, and there is a gap between the tasks used to test models and those used in clinical practice. Objective: In response, we propose CLEVER (Clinical Large Language Model Evaluation–Expert Review), a methodology for blind, randomized, preference-based evaluation by practicing medical doctors on specific tasks. Methods: We demonstrate the methodology by comparing GPT-4o (OpenAI) against 2 health care–specific LLMs, with 8 billion and 70 billion parameters, over 3 tasks: clinical text summarization, clinical information extraction, and question answering on biomedical research. Results: Medical doctors prefer the medical model–small LLM trained by John Snow Labs over GPT-4o 45% to 92% more often on the dimensions of factuality, clinical relevance, and conciseness. Conclusions: The models show comparable performance on open-ended medical question answering, suggesting that health care–specific LLMs can outperform much larger general purpose LLMs in tasks that require understanding of clinical context. We test the validity of CLEVER evaluations by conducting interannotator agreement, interclass correlation, and washout period analysis.

5 September 2025 in JMIR AI
Verified
Journal Article
Open access
VKVeysel Kocaman
Veysel Kocaman
FCFu-Yuan Cheng
Fu-Yuan Cheng
JBJulio Bonis
Julio Bonis
Ganesh RautGanesh Raut
Ganesh Raut
PTPrem Timsina
Prem Timsina
Arash KiaArash Kia
Arash Kia
Background: Clinical notes house rich, yet unstructured, patient data, making analysis challenging due to medical jargon, abbreviations, and synonyms causing ambiguity. This complicates real-time extraction for decision support tools. Objective: This study aimed to examine the data curation, technology, and workflow of the named entity recognition (NER) pipeline, a component of a broader clinical decision support tool that identifies key entities using NER models and classifies these entities as present or absent in the patient through an NER assertion model. Methods: We gathered progress care, radiology, and pathology notes from 5000 patients, dividing them into 5 batches of 1000 patients each. Metrics such as notes and reports per patient, sentence count, token size, runtime, central processing unit, and memory use were measured per note type. We also evaluated the precision of the NER outputs and then the precision and recall of NER assertion models against manual annotations by a clinical expert. Results: Using Spark natural language processing clinical pretrained NER models on 138,250 clinical notes, we observed excellent NER precision, with a peak in procedures at 0.989 (95% CI 0.977-1.000) and an accuracy in the assertion model of 0.889 (95% CI 0.856-0.922). Our analysis highlighted long-tail distributions in notes per patient, note length, and entity density. Progress care notes had notably more entities per sentence than radiology and pathology notes, showing 4-fold and 16-fold differences, respectively. Conclusions: Further research should explore the analysis of clinical notes beyond the scope of our study, including discharge summaries and psychiatric evaluation notes. Recognizing the unique linguistic characteristics of different note types underscores the importance of developing specialized NER models or natural language processing pipeline setups tailored to each type. By doing so, we can enhance their performance across a more diverse range of clinical scenarios.

7 April 2024 in JMIR Preprints
Verified
Preprint
Open access
VKVeysel Kocaman
Veysel Kocaman
FCFu-Yuan Cheng
Fu-Yuan Cheng
JBJulio Bonis
Julio Bonis
Ganesh RautGanesh Raut
Ganesh Raut
PTPrem Timsina
Prem Timsina
Arash KiaArash Kia
Arash Kia
BACKGROUND Clinical notes house rich, yet unstructured, patient data, making analysis challenging due to medical jargon, abbreviations, and synonyms causing ambiguity. This complicates real-time extraction for decision support tools. OBJECTIVE This study aimed to examine the data curation, technology, and workflow of the named entity recognition (NER) pipeline, a component of a broader clinical decision support tool that identifies key entities using NER models and classifies these entities as present or absent in the patient through an NER assertion model. METHODS We gathered progress care, radiology, and pathology notes from 5000 patients, dividing them into 5 batches of 1000 patients each. Metrics such as notes and reports per patient, sentence count, token size, runtime, central processing unit, and memory use were measured per note type. We also evaluated the precision of the NER outputs and then the precision and recall of NER assertion models against manual annotations by a clinical expert. RESULTS Using Spark natural language processing clinical pretrained NER models on 138,250 clinical notes, we observed excellent NER precision, with a peak in procedures at 0.989 (95% CI 0.977-1.000) and an accuracy in the assertion model of 0.889 (95% CI 0.856-0.922). Our analysis highlighted long-tail distributions in notes per patient, note length, and entity density. Progress care notes had notably more entities per sentence than radiology and pathology notes, showing 4-fold and 16-fold differences, respectively. CONCLUSIONS Further research should explore the analysis of clinical notes beyond the scope of our study, including discharge summaries and psychiatric evaluation notes. Recognizing the unique linguistic characteristics of different note types underscores the importance of developing specialized NER models or natural language processing pipeline setups tailored to each type. By doing so, we can enhance their performance across a more diverse range of clinical scenarios.

26 January 2021 in arXiv
Verified
Preprint
Open access
PDF file
VKVeysel Kocaman
Veysel Kocaman
DTDavid Talby
David Talby
Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant and accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100 pre trained pipelines and models in more than 192 languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing nine times growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the worlds most widely used NLP library in the enterprise.

4 February 2025 in JMIR Preprints
Verified
Preprint
Open access
VKVeysel Kocaman
Veysel Kocaman
MKMustafa Aytuğ Kaya
Mustafa Aytuğ Kaya
AFAndrei Marian Feier
Andrei Marian Feier
DTDavid Talby
David Talby
BACKGROUND The proliferation of both general-purpose and healthcare-specific Large Language Models (LLMs) has intensified the challenge of effectively evaluating and comparing them. Data contamination plagues the validity of public benchmarks; self-preference distorts LLM-as-a-judge approaches; and there’s a gap between the tasks used to test models and those used in clinical practice. OBJECTIVE In response, we propose CLEVER: A methodology for blind, randomized, preference-based evaluation by practicing medical doctors on specific tasks. METHODS We demonstrate the methodology by comparing GPT-4o against two healthcare-specific LLMs, with 8B and 70B parameters, over three tasks: clinical text summarization, clinical information extraction, and question answering on biomedical research. RESULTS Medical doctors prefer the Small Medical LLM over GPT-4o 45% to 92% more often on the dimensions of factuality, clinical relevance, and conciseness. CONCLUSIONS The models show comparable performance on open-ended medical Q&A, suggesting that healthcare-specific LLMs can outperform much larger general-purpose LLMs in tasks that require understanding of clinical context. We test the validity of CLEVER evaluations by conducting inter-annotator agreement, inter-class correlation, and washout period analysis. CLINICALTRIAL n/a