A number of recent papers have shown experimental evidence that suggests it is possible to build highly accurate deep neural network models to detect COVID-19 from chest X-ray images. In this paper, we show that good generalization to unseen sources has not been achieved. Experiments with richer data sets than have previously been used show models have high accuracy on seen sources, but poor accuracy on unseen sources. The reason for the disparity is that the convolutional neural network model, which learns features, can focus on differences in X-ray machines or in positioning within the machines, for example. Any feature that a person would clearly rule out is called a confounding feature. Some of the models were trained on COVID-19 image data taken from publications, which may be different than raw images. Some data sets were of pediatric cases with pneumonia where COVID-19 chest X-rays are almost exclusively from adults, so lung size becomes a spurious feature that can be exploited. In this work, we have eliminated many confounding features by working with as close to raw data as possible. Still, deep learned models may leverage source specific confounders to differentiate COVID-19 from pneumonia preventing generalizing to new data sources (i.e. external sites). Our models have achieved an AUC of 1.00 on seen data sources but in the worst case only scored an AUC of 0.38 on unseen ones. This indicates that such models need further assessment/development before they can be broadly clinically deployed. An example of fine-tuning to improve performance at a new site is given.

Glioma is the most common type of primary malignant brain tumor. Accurate survival time prediction for glioma patients may positively impact treatment planning. In this paper, we develop an automatic survival time prediction tool for glioblastoma patients along with an effective solution to the limited availability of annotated medical imaging datasets. Ensembles of snapshots of three dimensional (3D) deep convolutional neural networks (CNN) are applied to Magnetic Resonance Image (MRI) data to predict survival time of high-grade glioma patients. Additionally, multi-sequence MRI images were used to enhance survival prediction performance. A novel way to leverage the potential of ensembles to overcome the limitation of labeled medical image availability is shown. This new classification method separates glioblastoma patients into long- and short-term survivors. The BraTS (Brain Tumor Image Segmentation) 2019 training dataset was used in this work. Each patient case consisted of three MRI sequences (T1CE, T2, and FLAIR). Our training set contained 163 cases while the test set included 46 cases. The best known prediction accuracy of 74% for this type of problem was achieved on the unseen test set.

In this paper, we explore the feasibility of leveraging the accelerometer and gyroscope sensors in modern smartphones to detect instances of distracted driving activities (e.g., calling, texting, and reading while driving). To do so, we conducted an experiment with 16 subjects on a realistic driving simulator. As discussed later, the simulator is equipped with a realistic steering wheel, acceleration/braking pedals, and a wide screen to visualize background vehicular traffic. It is also programmed to simulate multiple environmental conditions like daytime, nighttime, fog, and rain/snow. The subjects were instructed to drive the simulator while performing a randomized sequence of activities that included texting, calling, and reading from a phone while they were driving, during which the accelerometer and gyroscope in the phone were logging sensory data. By extracting features from this sensory data, we then implemented a machine learning technique based on random forests to detect distracted driving. Our technique achieves very good precision, recall, and F -measure across all environmental conditions we tested. We believe that our contributions in this paper can have a significant impact on enhancing road safety.

When a learned model has high accuracy under familiar settings (internal testing) and a big drop in accuracy under slightly different circumstances (external testing) we suspect it is using shortcuts to make decisions. This problem is known as shortcut learning. In medical imaging, shortcuts are undesired and unintended features that the model relies on to perform diagnosis. Shortcut-based solutions using medical images could lead to false diagnoses and have dangerous implications for patients. In the current COVID-19 era, a large set of papers have been published proposing the use of deep convolutional neural networks to perform diagnosis or triage of COVID-19 from chest X-rays (CXRs). These studies are reporting high accuracies which could be misleading and overestimated. To our knowledge, none of the currently published papers with high performance reported testing on samples from truly unseen data sources. Studies which did, have noticed a significant performance drop when testing on unseen sources indicating a failure to generalize. In this paper, we elucidate the generalization challenge of deep learning based models trained for disease diagnosis. We use the example of COVID-19 diagnosis from CXRs. Solutions that mitigate shortcut learning are introduced and experimentally shown to be effective. Our proposed methods enable the models to have a statistically significantly reduced performance drop-off on unseen data sources. Thus, lowering the performance drop to only 9% instead of 20%. The issues with convolutional neural networks addressed here generally apply to other imaging modalities and recognition problems, as shown.

Age of Big Data and Smart Cities: Privacy Trade-Off