Image classification is central to the big data revolution in medicine. Improved information processing methods for diagnosis and classification of digital medical images have shown to be successful via deep learning approaches. As this field is explored, there are limitations to the performance of traditional supervised classifiers. This paper outlines an approach that is different from the current medical image classification tasks that view the issue as multi-class classification. We performed a hierarchical classification using our Hierarchical Medical Image classification (HMIC) approach. HMIC uses stacks of deep learning models to give particular comprehension at each level of the clinical picture hierarchy. For testing our performance, we use biopsy of the small bowel images that contain three categories in the parent level (Celiac Disease, Environmental Enteropathy, and histologically normal controls). For the child level, Celiac Disease Severity is classified into 4 classes (I, IIIa, IIIb, and IIIC).

Celiac Disease (CD) is a chronic autoimmune disease that affects the small intestine in genetically predisposed children and adults. Gluten exposure triggers an inflammatory cascade which leads to compromised intestinal barrier function. If this enteropathy is unrecognized, this can lead to anemia, decreased bone density, and, in longstanding cases, intestinal cancer. The prevalence of the disorder is 1% in the United States. An intestinal (duodenal) biopsy is considered the “gold standard” for diagnosis. The mild CD might go unnoticed due to non-specific clinical symptoms or mild histologic features. In our current work, we trained a model based on deep residual networks to diagnose CD severity using a histological scoring system called the modified Marsh score. The proposed model was evaluated using an independent set of 120 whole slide images from 15 CD patients and achieved an AUC greater than 0.96 in all classes. These results demonstrate the capability of the proposed model for CD severity classification using histopathological images.

The gold standard of histopathology for the diagnosis of Barrett’s esophagus (BE) is hindered by inter-observer variability among gastrointestinal pathologists. Deep learning-based approaches have shown promising results in the analysis of whole-slide tissue histopathology images (WSIs). We performed a comparative study to elucidate the characteristics and behaviors of different deep learning-based feature representation approaches for the WSI-based diagnosis of diseased esophageal architectures, namely, dysplastic and non-dysplastic BE. The results showed that if appropriate settings are chosen, the unsupervised feature representation approach is capable of extracting more relevant image features from WSIs to classify and locate the precursors of esophageal cancer compared to weakly supervised and fully supervised approaches.

Intratumor heterogeneity drives disease progression and treatment resistance, which can lead to poor patient outcomes. Here, we present a computational approach for quantification of cancer cell diversity in routine hematoxylin and eosin (H&E)-stained histopathology images. We analyzed publicly available digitized whole slide H&E images for a total of 2000 patients. Four tumor types were included: lung, head and neck, colon and rectal cancers, representing major histology subtypes (adenocarcinomas and squamous cell carcinomas). We performed single-cell analysis on H&E images and trained a deep convolutional autoencoder to automatically learn feature representations of individual cancer nuclei. We then computed features of intra-nuclear variability and inter-nuclear diversity to quantify tumor heterogeneity. Finally, we used these features to build a machine learning model to predict patient prognosis. A total of 68 million cancer cells were segmented and analyzed for nuclear image features. We discovered multiple morphological subtypes of cancer cells (range: 15-20) that co-exist within the same tumor, each with distinct phenotypic characteristics. Moreover, we showed that a higher morphological diversity is associated with chromosome instability and genomic aneuploidy. A machine learning model based on morphological diversity demonstrated independent prognostic values across tumor types (hazard ratio range: 1.62-3.23, P < 0.035) in validation cohorts and further improved prognostication when combined with clinical risk factors. Our study provides a practical approach for quantifying intratumor heterogeneity based on routine histopathology images. The cancer cell diversity score can be used to refine risk stratification and inform personalized treatment strategies.

Aneuploidy is a hallmark of aggressive malignancies associated with therapeutic resistance and poor survival. Measuring aneuploidy requires expensive specialized techniques that are not clinically applicable. Deep learning analysis of routine histopathology slides has revealed associations with genetic mutations. However, existing studies focus on image tiles, and there is no prior work that predicts aneuploidy using single-cell analysis. Here, we present a single-cell heterogeneity-aware and transformer-guided deep learning framework to predict aneuploidy from whole slide histopathology images. First, we perform nuclei segmentation and classification to obtain individual cancer cells, which are clustered into multiple subtypes. The cell subtype distributions are computed to measure cancer cell heterogeneity. Additionally, morphological features of different cell subtypes are extracted. Further, we leverage a multiple instance learning module with Transformer, which encourages the network to focus on the most informative cancer cells. Lastly, a hybrid network is built to unify cell heterogeneity, morphology, and deep features for aneuploidy prediction. We train and validate our method on two public datasets from TCGA: lung adenocarcinoma (LUAD) and head and neck squamous cell carcinoma (HNSC), with 339 and 245 patients. Our model achieves promising performance with AUC of 0.818 (95% CI: 0.718–0.919) and 0.827 (95% CI: 0.704–0.949) on the LUAD and HNSC test sets, respectively. Through extensive ablation and comparison studies, we demonstrate the effectiveness of each component of the model and superior performance over alternative networks. In conclusion, we present a novel deep learning approach to predict aneuploidy from histopathology images, which could inform personalized cancer treatment.