Christophe Ye, Cassie S. Mitchell. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2025.

This work introduces TrialSieve, a novel framework for biomedical information extraction that enhances clinical meta-analysis and drug repurposing. By extending traditional PICO (Patient, Intervention, Comparison, Outcome) methodologies, TrialSieve incorporates hierarchical, treatment group-based graphs, enabling more comprehensive and quantitative comparisons of clinical outcomes. TrialSieve was used to annotate 1609 PubMed abstracts, 170,557 annotations, and 52,638 final spans, incorporating 20 unique annotation categories that capture a diverse range of biomedical entities relevant to systematic reviews and meta-analyses. The performance (accuracy, precision, recall, F1-score) of four natural-language processing (NLP) models (BioLinkBERT, BioBERT, KRISSBERT, PubMedBERT) and the large language model (LLM), GPT-4o, was evaluated using the human-annotated TrialSieve dataset. BioLinkBERT had the best accuracy (0.875) and recall (0.679) for biomedical entity labeling, whereas PubMedBERT had the best precision (0.614) and F1-score (0.639). Error analysis showed that NLP models trained on noisy, human-annotated data can match or, in most cases, surpass human performance. This finding highlights the feasibility of fully automating biomedical information extraction, even when relying on imperfectly annotated datasets. An annotator user study (n = 39) revealed significant (p < 0.05) gains in efficiency and human annotation accuracy with the unique TrialSieve tree-based annotation approach. In summary, TrialSieve provides a foundation to improve automated biomedical information extraction for frontend clinical research.

Background/Objectives: Literature-derived knowledge graphs are attractive for drug repurposing because every edge retains its supporting publications. Their predicates, however, record which relation a sentence asserted, not whether the evidence supports a therapeutic direction. We asked whether reading those publications can align such a graph with the repurposing task. Methods: EviAlign re-interprets each edge of disease-centered SemMedDB subgraphs. A large language model (LLM) judge reads the supporting publications sentence by sentence, assigns a therapeutic direction or withdraws the edge, and weights each edge by evidence strength rather than publication count. Signed edges compose along drug–mediator–disease paths. Six diseases and 68,689 edges were evaluated against the original predicates read through a fixed mapping and trimmed to the same size, using curated drug–disease benchmarks, an internal sign-consistency test, and blinded rating by three domain experts. Results: EviAlign increased average precision by 63% over the size-matched fixed-mapping baseline. For candidates with no direct disease literature, Precision@10 increased approximately threefold (0.050 versus 0.018). Direct and mediator-derived signs agreed in 75.0% of evaluable cases, 16 percentage points above a shuffled-label control; the fixed mapping stayed at the level of its own control. Experts confirmed the re-interpreted direction in 34 of 36 reversal ratings and judged 80% of top-ranked mediator-only candidates worth investigating, versus 53% of low-ranked controls. Conclusions: Reading the evidence behind each edge aligns literature-derived knowledge graphs with drug repurposing and yields interpretable, mechanism-grounded hypotheses for further investigation.

Background: Mechanistic reporting from large-scale biomedical knowledge graphs remains challenging, particularly when integrating structured graph evidence with large language model (LLM)–based explanation in a reproducible and auditable manner. Existing approaches either rely on manual synthesis of graph-derived results or generate unconstrained narratives that lack traceability to underlying evidence. Methods: We present SemNet Explorer, an evidence-grounded knowledge graph–LLM unified framework for automated mechanistic reporting across biomedical domains using SemNet 2.0, a PubMed-scale heterogeneous knowledge graph. Given a set of target concepts and a selected semantic layer, the framework organizes graph-derived evidence into structured regions and generates two complementary report types: global reports for process-level mechanisms and anchor-centric reports for localized mediator-based explanations. A central methodological contribution is an ablation-derived adaptive grounding policy: we systematically compare alternative evidence-integration strategies across report types, semantic layers, and region structures, and use the resulting preferences to guide prompt selection in the deployed system. Results: SemNet Explorer produces stable region decompositions and interpretable report scaffolds across molecular (AAPP), disease-level (DSYN), and pharmacologic (PHSU) representations. For global reports, explicit evidence grounding improves expression quality more consistently than content accuracy, with benefits dependent on evidence density and semantic abstraction. In contrast, anchor-centric reports show consistent improvements in both content and expression under stronger, mediator-constrained prompting. These findings are supported by both pairwise ablation comparisons and absolute score analyses. Conclusions: SemNet Explorer establishes a generalizable unified framework and interactive platform for transforming knowledge graph evidence into reproducible mechanistic narratives across biomedical domains, including multimorbidity analysis, comparative pathophysiology, drug repurposing, and adverse event discovery. The results demonstrate that effective knowledge graph–LLM integration requires adaptive, context-dependent evidence grounding rather than fixed prompting strategies.