Mikel Bahn

Mikel Bahn

Auto-generated profile, based on publicly available information

Mikel Bahn

Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany

Research Focus & Metrics

Academic Career Length

2Years (2025-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
25 March 2026 in Journal of Medical Internet Research
Verified
Journal Article
Open access
KAKamyar Arzideh
Kamyar Arzideh
HSHenning Schäfer
Henning Schäfer
AIAhmad Idrissi-Yaghir
Ahmad Idrissi-Yaghir
CSCynthia Sabrina Schmidt
Cynthia Sabrina Schmidt
BEBahadir Eryilmaz
Bahadir Eryilmaz
RHRené Hosch
René Hosch
Background: Embedding models are critical components of Retrieval Augmented Generation (RAG) systems for retrieving and searching unstructured medical data. However, existing models are predominantly trained on publicly available English datasets, limiting their effectiveness in non-English health care settings. More importantly, these models lack training on real-world clinical documents, leading to inaccurate context retrieval when integrated into RAG systems for health care applications. This gap is particularly pronounced in specialized medical documentation containing domain-specific terminology, abbreviations, and nuanced clinical language. Objective: This retrospective study aimed to develop and validate embedding models specifically trained on real-world clinical documents from multiple medical specialties to improve medical information retrieval (IR) and RAG system performance in both German and English language contexts. Methods: We fine-tuned embedding models, so-called sentence transformers, using the multilingual-e5-large architecture as a foundation. Training data consisted of approximately 11 million question-answer pairs synthetically generated from 400,000 diverse clinical documents from a large German tertiary hospital, spanning 163,840 patients and 282,728 clinical cases between 2018 and 2023. The SauerkrautLM-SOLAR-Instruct large language model generated medically relevant questions and corresponding answers for each document. The dataset was additionally pseudonymized and translated into English to aim for broader applicability. Models were evaluated in 2 distinct scenarios: IR using questions with multiple relevant passages, and RAG system performance in both cross-patient and patient-centered contexts. Results: In the IR evaluation, the fine-tuned miracle model achieved a mAP@100 of 0.27, outperforming the multilingual-e5-large baseline (0.14) and state-of-the-art models such as bge-m3 (0.11). In the RAG evaluation, the model demonstrated robust performance comparable with the baseline in the constrained patient-centered scenario (BERTScore F1 0.781 vs 0.778) and showed moderate improvements in the unconstrained cross-patient setting (BLEURT 0.56 vs 0.53). Notably, the model trained on pseudonymized data achieved comparable retrieval performance (mAP@100 0.25) and the highest scores for patient-centered contextual precision (0.93). Performance gains were robust in the German dataset, while the translated English model demonstrated promising results as a proof of concept for cross-lingual transfer. Conclusions: By leveraging a comprehensive real-world dataset spanning multiple medical specialties and using large language models for synthetic question generation, we successfully created and validated domain-specific embedding models. These models can improve medical IR in large-scale search spaces and perform competitively in constrained RAG applications. By publishing the models trained on pseudonymized data, other health care institutions can integrate or adapt these embedding models to their needs. This work establishes a reproducible framework for developing domain-specific clinical embedding models, with the potential to improve data retrieval in medical settings.

12 August 2025 in Journal of Medical Internet Research
Verified
Journal Article
Open access
AIAhmad Idrissi-Yaghir
Ahmad Idrissi-Yaghir
KAKamyar Arzideh
Kamyar Arzideh
HSHenning Schäfer
Henning Schäfer
BEBahadir Eryilmaz
Bahadir Eryilmaz
MBMikel Bahn
Mikel Bahn
RHRené Hosch
René Hosch
Background: Recent natural language processing breakthroughs, particularly with the emergence of large language models (LLMs), have demonstrated remarkable capabilities on general knowledge benchmarks. However, there is limited data on the performance and understanding of these models in relation to the Fast Healthcare Interoperability Resources (FHIR) standard. The complexity and specialized nature of FHIR present challenges for LLMs, which are typically trained on broad datasets and may have a limited understanding of the nuances required for domain-specific tasks. Improving health data interoperability can greatly benefit the use of clinical data and interaction with electronic health records. Objective: This study presents the Fast Healthcare Interoperability Resources (FHIR) Workbench, a comprehensive suite of datasets designed to evaluate the ability of LLMs to understand and apply the FHIR standard. Methods: In total, 4 evaluation datasets were created to assess the FHIR knowledge and capabilities of LLMs. These tasks include multiple-choice questions on general FHIR concepts and the FHIR Representational State Transfer (REST) application programming interface, as well as correctly identifying the resource type and generating FHIR resources from unstructured clinical patient notes. In addition, we evaluate open-source LLMs, such as Qwen 2.5 Coder and DeepSeek-V3, and commercial LLMs, including GPT-4o and Gemini 2, on these tasks in a zero-shot setting. To provide context for interpreting LLM performance, a subset of the datasets was human-evaluated by recruiting 6 participants with varying levels of FHIR expertise. Results: Our evaluation across multiple FHIR tasks revealed nuanced performance metrics. Commercial models demonstrated exceptional capabilities, with GPT-4o achieving a 0.9990 F1-score on the FHIR-ResourceID task, 0.9400 on the FHIR-QA task, and 0.9267 on the FHIR-RESTQA task. Open-source models also demonstrated strong performance, with DeepSeek-v3 achieving 0.9400 on FHIR-QA, 0.9400 on FHIR-RESTQA, and 0.9142 on FHIR-ResourceID. Qwen 2.5 Coder-7B-Instruct demonstrated high accuracy, scoring 0.9533 on FHIR-QA and 0.8920 on FHIR-ResourceID. However, all models struggled with the Note2FHIR task, with performance ranging from 0.0382 (OLMo) to a maximum of 0.3633 (GPT-4.5-preview), highlighting the significant challenge of converting unstructured clinical text into FHIR-compliant resources. Human participants achieved accuracy scores ranging from 0.50 to 1.0 across the first 3 tasks. Conclusions: This study highlights the competitive performance of both open-source models, such as Qwen and DeepSeek, and commercial models, such as GPT-4o and Gemini, in FHIR-related tasks. While open-source models are advancing rapidly, commercial models still have an advantage for specific, complex tasks. The FHIR Workbench offers a valuable platform for evaluating the capabilities of these models and promoting improvements in health data interoperability.

1 January 2026 in JCO Clinical Cancer Informatics
Verified
Journal Article
Closed access
KAKamyar Arzideh
Kamyar Arzideh
RHRené Hosch
René Hosch
ATAmin T. Turki
Amin T. Turki
BEBahadir Eryilmaz
Bahadir Eryilmaz
MBMikel Bahn
Mikel Bahn
JKJulius Keyl
Julius Keyl
PURPOSE: Manual coding of pathology reports with International Classification of Diseases for Oncology (ICD-O)-3 codes is time-consuming, error-prone, and resource-intensive for health care institutions. To evaluate the performance of multiple state-of-the-art large language models (LLMs) in extracting ICD-O-3 topography and morphology codes from real-world pathology reports and assess their potential for clinical implementation, this study compares the performance of state-of-the-art open-source models in multiple evaluation setups. METHODS: We analyzed 21,364 pathology reports from 10,823 patients documented between 2013 and 2025 at a large German hospital. Five LLMs were evaluated: Llama-3.3-70B-Instruct, DeepSeek-R1-Distill-Llama (8B and 70B variants), Qwen3-235B-A22B, and Gemma-3-12B-it. All models were deployed on secured private information technology hospital infrastructure. Three different prompts were developed for topography extraction (with and without anatomic context) and morphology extraction. Performance was evaluated using exact code matches and first three-position matches. RESULTS: For exact ICD-O topography code prediction, Qwen3-235B-A22B achieved the highest performance (microaverage F1: 71.6%), whereas Llama-3.3-70B-Instruct performed best at predicting the first three characters (micro-average F1: 84.6%). For morphology codes, DeepSeek-R1-Distill-Llama-70B outperformed other models (exact microaverage F1: 34.7%; first three characters’ microaverage F1: 77.8%). Large disparities between micro- and macroaverage F1-scores indicated poor generalization to rare conditions. CONCLUSION: Although LLMs demonstrate promising capabilities as support systems for expert-guided pathology coding, their performance is not yet sufficient for fully automated, unsupervised use in routine clinical workflows. LLMs showed poor performance on rare conditions, heavy dependence on contextual information, and substantially lower scores for morphology versus topography classification.

2 July 2026 in Journal of Medical Internet Research
Verified
Journal Article
Open access
BEBahadır Eryılmaz
Bahadır Eryılmaz
KAKamyar Arzideh
Kamyar Arzideh
MBMikel Bahn
Mikel Bahn
HDHendrik Damm
Hendrik Damm
SWSina Warmer
Sina Warmer
RHRené Hosch
René Hosch
Background: Unstructured clinical text remains a major barrier to interoperable data reuse and large-scale secondary analysis in health care. Large language models (LLMs) have the potential to automate the extraction of structured clinical information; however, their application is limited by the scarcity of high-quality annotated training data. Objective: To address these limitations, this study aims to develop and validate a scalable, privacy-preserving framework that uses synthetic data generated from structured Fast Healthcare Interoperability Resources (FHIR) to fine-tune open-source LLMs for the effective extraction of interoperable clinical information from unstructured text. Methods: We evaluated an LLM-based framework for extracting structured clinical information from cancer-related discharge letters and mapping it to representations compatible with FHIR. To enable large-scale supervised training, we developed a random sample generator that creates synthetic discharge letters using Qwen3-235B by randomly sampling and aggregating structured FHIR data from 41,175 patients with cancer. The resulting synthetic discharge letters (n=75,000) were paired with their originating structured data, forming a large-scale dataset for fine-tuning MedGemma 27B, a 27-billion-parameter medical language model. Evaluation was conducted on the synthetic test dataset (n=7500), real-world discharge letters (n=30), which were evaluated by physicians and a medical student, and a comparative one-shot approach using open-source models (Qwen3, LLaMA, and GPT-OSS). Results: The fine-tuned model achieved high extraction performance across multiple clinical entities on the synthetic test set, with F 1 -scores of 0.84 for full International Classification of Diseases diagnosis codes, 0.99 for tumor-related information, 0.99 for laboratory values, 0.99 for medication names and dosages, and 0.94 for Anatomical Therapeutic Chemical medication codes. The extraction of procedure-related information was more challenging, with F 1 -scores of 0.63 for OPS codes and 0.90 for procedure descriptions. The fine-tuned model consistently outperformed general-purpose LLMs in a one-shot comparison across nearly all extraction categories. When evaluated by physicians on real-world discharge letters, the model achieved case-level correctness rates of 78.9% for International Classification of Diseases diagnoses, 86.1% for tumor-related information, 93.0% for medications, and 61.3% for procedures. Conclusions: These results demonstrate that synthetic text generation from structured clinical data enables the effective and scalable training of LLMs for extracting interoperable, multientity clinical information from unstructured documentation.

1 January 2026 in IEEE Access
Verified
Journal Article
Open access
MBMikel Bahn
Mikel Bahn
BEBahadir Eryilmaz
Bahadir Eryilmaz
HSHenning Schafer
Henning Schafer
AIAhmad Idrissi-Yaghir
Ahmad Idrissi-Yaghir
MAMohamed Albahri
Mohamed Albahri
FNFelix Nensa
Felix Nensa

10 August 2026 in JMIR Preprints
Verified
Preprint
Open access
BEBahadır Eryılmaz
Bahadır Eryılmaz
MBMikel Bahn
Mikel Bahn
HDHendrik Damm
Hendrik Damm
BCBohao Chu
Bohao Chu
NBNoëlle Bender
Noëlle Bender
FNFelix Nensa
Felix Nensa
BACKGROUND Large language models (LLMs) answering medical questions produce fluent but sometimes incorrect responses, and a clinically safe system must either avoid such errors or flag them. Deliberate multi-step prompting is widely proposed to prevent them, and sampling-based uncertainty estimation to detect them. Self-consistency prompting connects the two: it resamples a question several times, then discards the disagreement among those samples in favor of a majority vote. OBJECTIVE We tested whether deliberate prompting prevents unreliable answers, and whether the discarded disagreement detects them. METHODS We conducted a pre-registered computational evaluation (Open Science Framework) crossing 12 prompting techniques, organized on a System-1/System-2 dual-process ladder from direct answering to multi-pass verification, with 5 datasets and 7 open-weight LLMs at 10 random seeds per cell. Datasets were PubMedQA, MIMIC-IV-ED emergency-department disposition, LongHealth, BioASQ Task 13B, and MedHallu. All inference was local with models pinned by commit hash; no clinical text was sent to hosted application programming interfaces, as required by the PhysioNet Data Use Agreement. A separate pass resampled each question's zero-shot answer 10 times; we scored the entropy of the resulting answer distribution against correctness by area under the receiver operating characteristic curve (AUROC). Accuracy contrasts used a paired bootstrap (10,000 resamples) with Wilcoxon signed-rank tests and Holm-Bonferroni correction. RESULTS Deliberate prompting did not reliably improve accuracy. Of 176 corrected technique-versus-baseline contrasts across the 4 primary models, 52 were significantly better and 54 significantly worse; pooled bucket effects were +1.3 percentage points for single-pass System-2, −1.5 for multi-pass verification and −0.2 for integration, and the best technique added 2.9 points. The same 10-sample budget used for voting changed accuracy by +0.15 points, whereas the disagreement among those samples detected incorrect answers with AUROC 0.73 (95% CI 0.70–0.75; n=1559) on LongHealth, 0.69 (0.65–0.72; n=1040) on BioASQ, 0.58 (0.56–0.60; n=1990) on PubMedQA and 0.56 (0.53–0.58; n=1930) on MIMIC-IV-ED. Abstaining on the least confident half raised accuracy on the answered subset by 10.2, 10.3, 4.9 and 4.3 points respectively, each above a matched random-abstention baseline. Detectability tracked error arbitrariness: self-agreement for correct versus incorrect answers was 0.94 versus 0.77 on LongHealth but 0.83 versus 0.79 on MIMIC-IV-ED. An entailment-based estimator confirmed the free-text result (AUROC 0.692 entailment-based versus 0.686 rule-based). CONCLUSIONS In the open-weight, zero-shot, small-sample regime typical of local deployment, deliberate prompting is not a reliable safety lever, whereas the disagreement self-consistency discards is an informative and essentially free reliability signal. Its limitation is systematic: it detects arbitrary errors and is blind to confident, reproducible ones, the errors of greatest clinical concern. Sampling-based uncertainty should therefore serve as a triage filter, never as a certificate of correctness.