Karl Holub

Karl Holub

Auto-generated profile, based on publicly available information

Karl Holub

Nested Knowledge Saint Paul, MN United States

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
2 May 2022 in JMIR Medical Informatics
Verified
Journal Article
Open access
KCKathryn Cowie
Kathryn Cowie
ARAsad Rahmatullah
Asad Rahmatullah
Nicole Janell HardyNicole Janell Hardy
Nicole Janell Hardy
KHKarl Holub
Karl Holub
KKKevin Kallmes
Kevin Kallmes
Background: Systematic reviews (SRs) are central to evaluating therapies but have high costs in terms of both time and money. Many software tools exist to assist with SRs, but most tools do not support the full process, and transparency and replicability of SR depend on performing and presenting evidence according to established best practices. Objective: This study aims to provide a basis for comparing and selecting between web-based software tools that support SR, by conducting a feature-by-feature comparison of SR tools. Methods: We searched for SR tools by reviewing any such tool listed in the SR Toolbox, previous reviews of SR tools, and qualitative Google searching. We included all SR tools that were currently functional and required no coding, and excluded reference managers, desktop applications, and statistical software. The list of features to assess was populated by combining all features assessed in 4 previous reviews of SR tools; we also added 5 features (manual addition, screening automation, dual extraction, living review, and public outputs) that were independently noted as best practices or enhancements of transparency and replicability. Then, 2 reviewers assigned binary present or absent assessments to all SR tools with respect to all features, and a third reviewer adjudicated all disagreements. Results: Of the 53 SR tools found, 55% (29/53) were excluded, leaving 45% (24/53) for assessment. In total, 30 features were assessed across 6 classes, and the interobserver agreement was 86.46%. Giotto Compliance (27/30, 90%), DistillerSR (26/30, 87%), and Nested Knowledge (26/30, 87%) support the most features, followed by EPPI-Reviewer Web (25/30, 83%), LitStream (23/30, 77%), JBI SUMARI (21/30, 70%), and SRDB.PRO (VTS Software) (21/30, 70%). Fewer than half of all the features assessed are supported by 7 tools: RobotAnalyst (National Centre for Text Mining), SRDR (Agency for Healthcare Research and Quality), SyRF (Systematic Review Facility), Data Abstraction Assistant (Center for Evidence Synthesis in Health), SR Accelerator (Institute for Evidence-Based Healthcare), RobotReviewer (RobotReviewer), and COVID-NMA (COVID-NMA). Notably, of the 24 tools, only 10 (42%) support direct search, only 7 (29%) offer dual extraction, and only 13 (54%) offer living/updatable reviews. Conclusions: DistillerSR, Nested Knowledge, and EPPI-Reviewer Web each offer a high density of SR-focused web-based tools. By transparent comparison and discussion regarding SR tool functionality, the medical community can both choose among existing software offerings and note the areas of growth needed, most notably in the support of living reviews.

24 November 2021 in JMIR Formative Research
Verified
Journal Article
Open access
KHKarl Holub
Karl Holub
Nicole Janell HardyNicole Janell Hardy
Nicole Janell Hardy
KKKevin Kallmes
Kevin Kallmes
Background: Systematic reviews depend on time-consuming extraction of data from the PDFs of underlying studies. To date, automation efforts have focused on extracting data from the text, and no approach has yet succeeded in fully automating ingestion of quantitative evidence. However, the majority of relevant data is generally presented in tables, and the tabular structure is more amenable to automated extraction than free text. Objective: The purpose of this study was to classify the structure and format of descriptive statistics reported in tables in the comparative medical literature. Methods: We sampled 100 published randomized controlled trials from 2019 based on a search in PubMed; these results were imported to the AutoLit platform. Studies were excluded if they were nonclinical, noncomparative, not in English, protocols, or not available in full text. In AutoLit, tables reporting baseline or outcome data in all studies were characterized based on reporting practices. Measurement context, meaning the structure in which the interventions of interest, patient arm breakdown, measurement time points, and data element descriptions were presented, was classified based on the number of contextual pieces and metadata reported. The statistic formats for reported metrics (specific instances of reporting of data elements) were then classified by location and broken down into reporting strategies for continuous, dichotomous, and categorical metrics. Results: We included 78 of 100 sampled studies, one of which (1.3%) did not report data elements in tables. The remaining 77 studies reported baseline and outcome data in 174 tables, and 96% (69/72) of these tables broke down reporting by patient arms. Fifteen structures were found for the reporting of measurement context, which were broadly grouped into: 1×1 contexts, where two pieces of context are reported in total (eg, arms in columns, data elements in rows); 2×1 contexts, where two pieces of context are given on row headers (eg, time points in columns, arms nested in data elements on rows); and 1×2 contexts, where two pieces of context are given on column headers. The 1×1 contexts were present in 57% of tables (99/174), compared to 20% (34/174) for 2×1 contexts and 15% (26/174) for 1×2 contexts; the remaining 8% (15/174) used unique/other stratification methods. Statistic formats were reported in the headers or descriptions of 84% (65/74) of studies. Conclusions: In this cross-sectional pilot review, we found a high density of information in tables, but with major heterogeneity in presentation of measurement context. The highest-density studies reported both baseline and outcome measures in tables, with arm-level breakout, intervention labels, and arm sizes present, and reported both the statistic formats and units. The measurement context formats presented here, broadly classified into three classes that cover 92% (71/78) of studies, form a basis for understanding the frequency of different reporting styles, supporting automated detection of the data format for extraction of metrics.

23 November 2022 in JMIR Medical Informatics
Verified
Journal Article
Open access
KCKathryn Cowie
Kathryn Cowie
ARAsad Rahmatullah
Asad Rahmatullah
Nicole Janell HardyNicole Janell Hardy
Nicole Janell Hardy
KHKarl Holub
Karl Holub
KKKevin Kallmes
Kevin Kallmes
[This corrects the article DOI: 10.2196/33219.].

24 August 2021 in JMIR Preprints
Verified
Preprint
Open access
KHKarl Holub
Karl Holub
Nicole Janell HardyNicole Janell Hardy
Nicole Janell Hardy
KKKevin Kallmes
Kevin Kallmes
BACKGROUND Systematic reviews depend on time-consuming extraction of data from the PDFs of underlying studies. To date, automation efforts have focused on extracting data from the text, and no approach has yet succeeded in fully automating ingestion of quantitative evidence. However, the majority of relevant data is generally presented in tables, and the tabular structure is more amenable to automated extraction than free text. OBJECTIVE The purpose of this study was to classify the structure and format of descriptive statistics reported in tables in the comparative medical literature. METHODS We sampled 100 published randomized controlled trials from 2019 based on a search in PubMed; these results were imported to the AutoLit platform. Studies were excluded if they were nonclinical, noncomparative, not in English, protocols, or not available in full text. In AutoLit, tables reporting baseline or outcome data in all studies were characterized based on reporting practices. Measurement context, meaning the structure in which the interventions of interest, patient arm breakdown, measurement time points, and data element descriptions were presented, was classified based on the number of contextual pieces and metadata reported. The statistic formats for reported metrics (specific instances of reporting of data elements) were then classified by location and broken down into reporting strategies for continuous, dichotomous, and categorical metrics. RESULTS We included 78 of 100 sampled studies, one of which (1.3%) did not report data elements in tables. The remaining 77 studies reported baseline and outcome data in 174 tables, and 96% (69/72) of these tables broke down reporting by patient arms. Fifteen structures were found for the reporting of measurement context, which were broadly grouped into: 1×1 contexts, where two pieces of context are reported in total (eg, arms in columns, data elements in rows); 2×1 contexts, where two pieces of context are given on row headers (eg, time points in columns, arms nested in data elements on rows); and 1×2 contexts, where two pieces of context are given on column headers. The 1×1 contexts were present in 57% of tables (99/174), compared to 20% (34/174) for 2×1 contexts and 15% (26/174) for 1×2 contexts; the remaining 8% (15/174) used unique/other stratification methods. Statistic formats were reported in the headers or descriptions of 84% (65/74) of studies. CONCLUSIONS In this cross-sectional pilot review, we found a high density of information in tables, but with major heterogeneity in presentation of measurement context. The highest-density studies reported both baseline and outcome measures in tables, with arm-level breakout, intervention labels, and arm sizes present, and reported both the statistic formats and units. The measurement context formats presented here, broadly classified into three classes that cover 92% (71/78) of studies, form a basis for understanding the frequency of different reporting styles, supporting automated detection of the data format for extraction of metrics.

28 August 2021 in JMIR Preprints
Verified
Preprint
Open access
KCKathryn Cowie
Kathryn Cowie
ARAsad Rahmatullah
Asad Rahmatullah
Nicole Janell HardyNicole Janell Hardy
Nicole Janell Hardy
KHKarl Holub
Karl Holub
KKKevin Kallmes
Kevin Kallmes
BACKGROUND Systematic reviews (SRs) are central to evaluating therapies but have high costs in terms of both time and money. Many software tools exist to assist with SRs, but most tools do not support the full process, and transparency and replicability of SR depend on performing and presenting evidence according to established best practices. OBJECTIVE This study aims to provide a basis for comparing and selecting between web-based software tools that support SR, by conducting a feature-by-feature comparison of SR tools. METHODS We searched for SR tools by reviewing any such tool listed in the SR Toolbox, previous reviews of SR tools, and qualitative Google searching. We included all SR tools that were currently functional and required no coding, and excluded reference managers, desktop applications, and statistical software. The list of features to assess was populated by combining all features assessed in 4 previous reviews of SR tools; we also added 5 features (manual addition, screening automation, dual extraction, living review, and public outputs) that were independently noted as best practices or enhancements of transparency and replicability. Then, 2 reviewers assigned binary present or absent assessments to all SR tools with respect to all features, and a third reviewer adjudicated all disagreements. RESULTS Of the 53 SR tools found, 55% (29/53) were excluded, leaving 45% (24/53) for assessment. In total, 30 features were assessed across 6 classes, and the interobserver agreement was 86.46%. DistillerSR (Evidence Partners; 26/30, 87%), Nested Knowledge (Nested Knowledge; 25/30, 83%), and EPPI-Reviewer Web (EPPI-Centre; 24/30, 80%) support the most features followed by Giotto Compliance (Giotto Compliance; 23/30, 77%), LitStream (ICF), and SRDB.PRO (VTS Software). Fewer than half of all the features assessed are supported by 7 tools: RobotAnalyst (National Centre for Text Mining), SRDR (Agency for Healthcare Research and Quality), SyRF (Systematic Review Facility), Data Abstraction Assistant (Center for Evidence Synthesis in Health), SR Accelerator (Institute for Evidence-Based Healthcare), RobotReviewer (RobotReviewer), and COVID-NMA (COVID-NMA). Notably, of the 24 tools, only 10 (42%) support direct search, only 7 (29%) offer dual extraction, and only 13 (54%) offer living/updatable reviews. CONCLUSIONS DistillerSR, Nested Knowledge, and EPPI-Reviewer Web each offer a high density of SR-focused web-based tools. By transparent comparison and discussion regarding SR tool functionality, the medical community can both choose among existing software offerings and note the areas of growth needed, most notably in the support of living reviews.

2 June 2026 in JMIR Preprints
Verified
Preprint
Open access
KKKevin Kallmes
Kevin Kallmes
JTJade Thurnham
Jade Thurnham
MSMarius Sauca
Marius Sauca
RTRanita Tarchand
Ranita Tarchand
JTJoshua Twaites
Joshua Twaites
KHKarl J. Holub
Karl J. Holub
BACKGROUND While artificial intelligence (AI) tools have been utilized for individual stages within the systematic literature review (SLR) process, no tool has previously been shown to support each critical SLR step. In addition, expert oversight is required for responsible use of AI to ensure the quality of SLR findings. OBJECTIVE To describe a complete methodology for utilizing our AI SLR tool with human-in-the-loop curation workflows, as well as AI validations, time savings, and approaches to ensure compliance with best review practices. METHODS SLRs require completing search, screening, and extraction from relevant studies, with qualitative synthesis, meta-analysis, and critical appraisal as relevant. We present a full methodological framework for completing SLRs utilizing our AutoLit® software (Nested Knowledge). This system integrates AI models into the central steps in SLR: search strategy generation, dual screening of titles/abstracts and full texts, extraction of qualitative and quantitative evidence, critical appraisal, and insight drafting, with fully automated network meta-analysis. Validations comparing AI performance to experts are reported, and where relevant, time savings within the SLR workflow. RESULTS Across the SLR workflow, Nested Knowledge AI tools showed strongest evidence for search development, screening support, structured tagging, and extraction efficiency. Smart Search translated research questions into Boolean strings with 76.8% and 79.6% recall across two validation sets. For title/abstract screening, The Inclusion Prediction Model achieved 82%–97.1% recall, while criteria-based Smart Screener achieved approximately 95% accuracy, 80% time savings, and high recall in autonomous abstract/full-text screening. Core Smart Tags achieved F1=0.74 for PICO extraction and hierarchy building, with 74%, 78%, and 91% accuracy for study type, location, and study size, respectively. Adaptive Smart Tags demonstrated 68.4% expert overlap for version 1, while early version 2 evaluations showed 76% recall/69% precision overall and stronger performance for structured abstract-level concepts. Smart Meta-Analytical Extraction produced >95% time savings for quantitative extraction, and Smart Insights scored highest for citation integrity (4.9/5) and synthesis quality (4.4/5). Smart Critical Appraisal supports structured, framework-based appraisal, though validation evidence remains more limited. CONCLUSIONS AI-enabled SLR tools with human-in-the-loop can reduce reviewer burden and accelerate evidence-review workflows without replacing expert judgment. Transparent methods, reproducible outputs, traceable evidence links, and human oversight are essential for ensuring that efficiency gains do not compromise review quality.