Research studies of artificial intelligence models in medical imaging have been hampered by poor generalization. This problem has been especially concerning over the last year with numerous applications of deep learning for COVID-19 diagnosis. Virtual imaging trials (VITs) could provide a solution for objective evaluation of these models. In this work utilizing the VITs, we created the CVIT-COVID dataset including 180 virtually imaged computed tomography (CT) images from simulated COVID-19 and normal phantom models under different COVID-19 morphology and imaging properties. We evaluated the performance of an open-source, deep-learning model from the University of Waterloo trained with multi-institutional data and an in-house model trained with the open clinical dataset called MosMed. We further validated the model's performance against open clinical data of 305 CT images to understand virtual vs. real clinical data performance. The open-source model was published with nearly perfect performance on the original Waterloo dataset but showed a consistent performance drop in external testing on another clinical dataset (AUC=0.77) and our simulated CVIT-COVID dataset (AUC=0.55). The in-house model achieved an AUC of 0.87 while testing on the internal test set (MosMed test set). However, performance dropped to an AUC of 0.65 and 0.69 when evaluated on clinical and our simulated CVIT-COVID dataset. The VIT framework offered control over imaging conditions, allowing us to show there was no change in performance as CT exposure was changed from 28.5 to 57 mAs. The VIT framework also provided voxel-level ground truth, revealing that performance of in-house model was much higher at AUC=0.87 for diffuse COVID-19 infection size <2.65% lung volume versus AUC=0.52 for focal disease with <2.65% volume. The virtual imaging framework enabled these uniquely rigorous analyses of model performance, which would be impracticable with real patients.
Since its initial identification in December 2019, the coronavirus SARS-CoV2 has disseminated internationally, impacting virtually every aspect of human activity. This virus causes Coronavirus Disease 2019, abbreviated as COVID-19: a highly contagious, high-mortality, acute respiratory illness. On March 11, 2020, the World Health Organization declared COVID-19 a pandemic following a 13-fold increase in cases outside China in two weeks.1 As of June 11, 2020, there have been 7,442,050 cases of COVID-19 confirmed globally, with 418,563 reported deaths due to the coronavirus disease.2 The extent and severity of the COVID-19 pandemic is unprecedented in modern times, pushing the need for fast and effective diagnosis to curtail the spread of this disease.
Skin cancers, notably melanoma, pose a significant health risk, with rising incidence rates and mortality rates. Early detection through screening is crucial, and the Department of Veterans Affairs has prioritized improved melanoma screening. Teledermatology, with digital dermatoscopy, offers a promising avenue for initial lesion assessment, prompting the exploration of algorithmic screening using deep learning. This paper investigates the efficacy of incorporating melanin index (MI) and erythema index (EI) as additional features, along with the conversion of images to the HSV color space, for use with deep learning models in melanoma classification. Considering the distinct clustering of human skin color in various color spaces, our study aims to explore the advantages of alternative color representations. Building on prior work and organized into two phases, our experiments utilize a diverse set of model architectures trained on images of varying sizes. Phase 1 aligns with the 2020 SIIM-ISIC Melanoma Classification Challenge, while Phase 2 involves an expanded dataset and more robust metrics. Despite optimistic outcomes in the earlier phase, our findings reveal no significant performance improvement when incorporating MI or EI in deep learning models for melanoma detection. This study contributes valuable insights for refining deep learning approaches in dermatoscopy, offering a cautionary note on the efficacy of specific features and of color space transformation.