“Just Accepted” papers have undergone full peer review and have been accepted for publication in Radiology: Artificial Intelligence. This article will undergo copyediting, layout, and proof review ...

Virtual Imaging Trials, known as VITs, provide a computational substitute for clinical trials. These traditional trials tend to be sluggish, costly, and frequently deficient in definitive evidence, all the while subjecting participants to ionizing radiation. Our VIT platform meticulously mimics essential components of the imaging process, encompassing everything from virtual patients and scanners to simulated readers. Within the scope of this intended research, we aim to authenticate our virtual imaging trial platform by duplicating the results of the National Lung Screening Trial (NLST) for lung cancer screening through the emulation of low-dose computed tomography (CT) and chest radiography (CXR) procedures. The methodology involves creating 66 unique computational phantoms, each with inserted simulated lung nodules. Replicating NLST CT imaging via Duke Legacy W20 scanner matched essential properties. Virtual imaging was done through DukeSim. A LUNA16-trained virtual reader, combining a 3D RetinaNet model (front-end) with a ResNet-10 false positive reduction model (back-end), evaluated the virtually imaged data, ensuring rigorous assessment. The back-end model achieved a sensitivity of over 95% at fewer than 3 false positives per scan for both the clinical and virtual imaged CTs. Notably, nodule diameter-based analysis showcases even higher sensitivity for nodules measuring 10 mm or more. In conclusion, the integration of diverse computational and imaging techniques, culminating in a virtual reader, demonstrates promising sensitivity. To capture both arms of the trial, future research will compare virtual reader performance on CT with CXR. This affirms the transformative potential of virtual imaging trials in advancing evidence-based medicine, offering an efficient and ethically conscious approach to medical research and development.

Research studies of artificial intelligence models in medical imaging have been hampered by poor generalization. This problem has been especially concerning over the last year with numerous applications of deep learning for COVID-19 diagnosis. Virtual imaging trials (VITs) could provide a solution for objective evaluation of these models. In this work utilizing the VITs, we created the CVIT-COVID dataset including 180 virtually imaged computed tomography (CT) images from simulated COVID-19 and normal phantom models under different COVID-19 morphology and imaging properties. We evaluated the performance of an open-source, deep-learning model from the University of Waterloo trained with multi-institutional data and an in-house model trained with the open clinical dataset called MosMed. We further validated the model's performance against open clinical data of 305 CT images to understand virtual vs. real clinical data performance. The open-source model was published with nearly perfect performance on the original Waterloo dataset but showed a consistent performance drop in external testing on another clinical dataset (AUC=0.77) and our simulated CVIT-COVID dataset (AUC=0.55). The in-house model achieved an AUC of 0.87 while testing on the internal test set (MosMed test set). However, performance dropped to an AUC of 0.65 and 0.69 when evaluated on clinical and our simulated CVIT-COVID dataset. The VIT framework offered control over imaging conditions, allowing us to show there was no change in performance as CT exposure was changed from 28.5 to 57 mAs. The VIT framework also provided voxel-level ground truth, revealing that performance of in-house model was much higher at AUC=0.87 for diffuse COVID-19 infection size <2.65% lung volume versus AUC=0.52 for focal disease with <2.65% volume. The virtual imaging framework enabled these uniquely rigorous analyses of model performance, which would be impracticable with real patients.