To evaluate the diagnostic efficacy of artificial intelligence (AI) software in detecting incidental pulmonary embolism (IPE) at CT and shorten the time to diagnosis with use of radiologist reading worklist prioritization. In this study with historical controls and prospective evaluation, regulatory-cleared AI software was evaluated to prioritize IPE on routine chest CT scans with intravenous contrast agent in adult oncology patients. Diagnostic accuracy metrics were calculated, and temporal end points, including detection and notification times (DNTs), were assessed during three time periods (April 2019 to September 2020): routine workflow without AI, human triage without AI, and worklist prioritization with AI. In total, 11 736 CT scans in 6447 oncology patients (mean age, 63 years ± 12 [SD]; 3367 men) were included. Prevalence of IPE was 1.3% (51 of 3837 scans), 1.4% (54 of 3920 scans), and 1.0% (38 of 3979 scans) for the respective time periods. The AI software detected 131 true-positive, 12 false-negative, 31 false-positive, and 11 559 true-negative results, achieving 91.6% sensitivity, 99.7% specificity, 99.9% negative predictive value, and 80.9% positive predictive value. During prospective evaluation, AI-based worklist prioritization reduced the median DNT for IPE-positive examinations to 87 minutes (vs routine workflow of 7714 minutes and human triage of 4973 minutes). Radiologists’ missed rate of IPE was significantly reduced from 44.8% (47 of 105 scans) without AI to 2.6% (one of 38 scans) when assisted by the AI tool (P < .001). AI-assisted workflow prioritization of IPE on routine CT scans in oncology patients showed high diagnostic accuracy and significantly shortened the time to diagnosis in a setting with a backlog of examinations. Keywords: CT, Computer Applications, Detection, Diagnosis, Embolism, Thorax, Thrombosis Supplemental material is available for this article . © RSNA, 2023 See also the commentary by Elicker in this issue.

Radiomics is a method to extract detailed information from diagnostic images that cannot be perceived by the naked eye. Although radiomics research carries great potential to improve clinical decision-making, its inherent methodological complexities make it difficult to comprehend every step of the analysis, often causing reproducibility and generalizability issues that hinder clinical adoption. Critical steps in the radiomics analysis and model development pipeline—such as image, application of image filters, and selection of feature extraction parameters—can greatly affect the values of radiomic features. Moreover, common errors in data partitioning, model comparison, fine-tuning, assessment, and calibration can reduce reproducibility and impede clinical translation. Clinical adoption of radiomics also requires a deep understanding of model explainability and the development of intuitive interpretations of radiomic features. To address these challenges, it is essential for radiomics model developers and clinicians to be well-versed in current best practices. Proper knowledge and application of these practices is crucial for accurate radiomics feature extraction, robust model development, and thorough assessment, ultimately increasing reproducibility, generalizability, and the likelihood of successful clinical translation. In this article, we have provided researchers with our recommendations along with practical examples to facilitate good research practices in radiomics. Key Points: Radiomics’ inherent methodological complexity should be understood to ensure rigorous radiomic model development to improve clinical decision-making. Adherence to radiomics-specific checklists and quality assessment tools ensures methodological rigor. Use of standardized radiomics tools and best practices enhances clinical translation of radiomics models.

This article provides radiologists with practical recommendations for evaluating AI performance in radiology, ensuring alignment with clinical goals and patient safety. It outlines key performance metrics, including overlap metrics for segmentation, test-based metrics (e.g., sensitivity, specificity, and area under the receiver operating characteristic curve), and outcome-based metrics (e.g., precision, negative predictive value, F1-score, Matthews correlation coefficient, and area under the precision-recall curve). Key recommendations emphasize local validation using independent datasets, selecting task-specific metrics, and considering deployment context to ensure real-world performance matches claimed efficacy. Common pitfalls, such as overreliance on a single metric, misinterpretation in low-prevalence settings, and failure to account for clinical workflow, are addressed with mitigation strategies. Additional guidance is provided on threshold selection, prevalence-adjusted evaluation, and AI-generated image quality assessment. This guide equips radiologists to critically evaluate both commercially available and in-house developed AI tools, ensuring their safe and effective integration into clinical practice. Clinical relevance statement: This review provides guidance on selecting and interpreting AI performance metrics in radiology, ensuring clinically meaningful evaluation and safe deployment of AI tools. By addressing common pitfalls and promoting standardized reporting, it supports radiologists in making informed decisions, ultimately improving diagnostic accuracy and patient outcomes. Key Points: Radiologists must evaluate performance metrics as they reflect acceptable performance in specific datasets but do not guarantee clinical utility. Independent evaluation tailored to the clinical setting is essential . Performance metrics must align with the intended task of the AI application—segmentation, detection, or classification—and be selected based on domain knowledge and clinical context . Sensitivity, specificity, area under the ROC curve, and accuracy must be interpreted with prevalence-dependent metrics (e.g., precision, F1 score, and Matthew’s correlation coefficient) calculated for the target population to ensure safe and effective clinical use .

Objective: To provide a comprehensive framework for value assessment of artificial intelligence (AI) in radiology. Methods: This paper presents the RADAR framework, which has been adapted from Fryback and Thornbury’s imaging efficacy framework to facilitate the valuation of radiology AI from conception to local implementation. Local efficacy has been newly introduced to underscore the importance of appraising an AI technology within its local environment. Furthermore, the RADAR framework is illustrated through a myriad of study designs that help assess value. Results: RADAR presents a seven-level hierarchy, providing radiologists, researchers, and policymakers with a structured approach to the comprehensive assessment of value in radiology AI. RADAR is designed to be dynamic and meet the different valuation needs throughout the AI’s lifecycle. Initial phases like technical and diagnostic efficacy (RADAR-1 and RADAR-2) are assessed pre-clinical deployment via in silico clinical trials and cross-sectional studies. Subsequent stages, spanning from diagnostic thinking to patient outcome efficacy (RADAR-3 to RADAR-5), require clinical integration and are explored via randomized controlled trials and cohort studies. Cost-effectiveness efficacy (RADAR-6) takes a societal perspective on financial feasibility, addressed via health-economic evaluations. The final level, RADAR-7, determines how prior valuations translate locally, evaluated through budget impact analysis, multi-criteria decision analyses, and prospective monitoring. Conclusion: The RADAR framework offers a comprehensive framework for valuing radiology AI. Its layered, hierarchical structure, combined with a focus on local relevance, aligns RADAR seamlessly with the principles of value-based radiology. Critical relevance statement: The RADAR framework advances artificial intelligence in radiology by delineating a much-needed framework for comprehensive valuation. Keypoints: • Radiology artificial intelligence lacks a comprehensive approach to value assessment. • The RADAR framework provides a dynamic, hierarchical method for thorough valuation of radiology AI. • RADAR advances clinical radiology by bridging the artificial intelligence implementation gap.

Treatment planning of gastrointestinal stromal tumors (GISTs) includes distinguishing GISTs from other intra-abdominal tumors and GISTs’ molecular analysis. The aim of this study was to evaluate radiomics for distinguishing GISTs from other intra-abdominal tumors, and in GISTs, predict the c-KIT, PDGFRA, BRAF mutational status, and mitotic index (MI). Patients diagnosed at the Erasmus MC between 2004 and 2017, with GIST or non-GIST intra-abdominal tumors and a contrast-enhanced venous-phase CT, were retrospectively included. Tumors were segmented, from which 564 image features were extracted. Prediction models were constructed using a combination of machine learning approaches. The evaluation was performed in a 100 × random-split cross-validation. Model performance was compared to that of three radiologists. One hundred twenty-five GISTs and 122 non-GISTs were included. The GIST vs. non-GIST radiomics model had a mean area under the curve (AUC) of 0.77. Three radiologists had an AUC of 0.69, 0.76, and 0.84, respectively. The radiomics model had an AUC of 0.52 for c-KIT, 0.56 for c-KIT exon 11, and 0.52 for the MI. The numbers of PDGFRA, BRAF, and other c-KIT mutations were too low for analysis. Our radiomics model was able to distinguish GISTs from non-GISTs with a performance similar to three radiologists, but less observer dependent. Therefore, it may aid in the early diagnosis of GIST, facilitating rapid referral to specialized treatment centers. As the model was not able to predict any genetic or molecular features, it cannot aid in treatment planning yet.

Soft-tissue and bone tumours (STBT) are rare, diagnostically challenging lesions with variable clinical behaviours and treatment approaches. This systematic review aims to provide an overview of Artificial Intelligence (AI) methods using radiological imaging for diagnosis and prognosis of these tumours, highlighting challenges in clinical translation, and evaluating study alignment with the Checklist for AI in Medical Imaging (CLAIM) and the FUTURE-AI international consensus guidelines for trustworthy and deployable AI to promote the clinical translation of AI methods. The systematic review identified literature from several bibliographic databases, covering papers published before 17/07/2024. Original research published in peer-reviewed journals, focused on radiology-based AI for diagnosis or prognosis of primary STBT was included. Exclusion criteria were animal, cadaveric, or laboratory studies, and non-English papers. Abstracts were screened by two of three independent reviewers to determine eligibility. Included papers were assessed against the two guidelines by one of three independent reviewers. The review protocol was registered with PROSPERO (CRD42023467970). The search identified 15,015 abstracts, from which 325 articles were included for evaluation. Most studies performed moderately on CLAIM, averaging a score of 28.9 ± 7.5 out of 53, but poorly on FUTURE-AI, averaging 5.1 ± 2.1 out of 30. Imaging-AI tools for STBT remain at the proof-of-concept stage, indicating significant room for improvement. Future efforts by AI developers should focus on design (e.g. defining unmet clinical need, intended clinical setting and how AI would be integrated in clinical workflow), development (e.g. building on previous work, training with data that reflect real-world usage, explainability), evaluation (e.g. ensuring biases are evaluated and addressed, evaluating AI against current best practices), and the awareness of data reproducibility and availability (making documented code and data publicly available). Following these recommendations could improve clinical translation of AI methods. Hanarth Fonds, ICAI Lab, NIHR, EuCanImage.