1 February 2024 in Medical Image Analysis
Determining early-stage prognostic markers and stratifying patients for effective treatment are two key challenges for improving outcomes for melanoma patients. Previous studies have used tumour transcriptome data to stratify patients into immune subgroups, which were associated with differential melanoma specific survival and potential predictive biomarkers. However, acquiring transcriptome data is a time-consuming and costly process. Moreover, it is not routinely used in the current clinical workflow. Here, we attempt to overcome this by developing deep learning models to classify gigapixel haematoxylin and eosin (H&E) stained pathology slides, which are well established in clinical workflows, into these immune subgroups. We systematically assess six different multiple instance learning (MIL) frameworks, using five different image resolutions and three different feature extraction methods. We show that pathology-specific self-supervised models using 10x resolution patches generate superior representations for the classification of immune subtypes. In addition, in a primary melanoma dataset, we achieve a mean area under the receiver operating characteristic curve (AUC) of 0.80 for classifying histopathology images into 'high' or 'low immune' subgroups and a mean AUC of 0.82 in an independent TCGA melanoma dataset. Furthermore, we show that these models are able to stratify patients into 'high' and 'low immune' subgroups with significantly different melanoma specific survival outcomes (log rank test, P< 0.005). We anticipate that MIL methods will allow us to find new biomarkers of high importance, act as a tool for clinicians to infer the immune landscape of tumours and stratify patients, without needing to carry out additional expensive genetic tests.
11 May 2026 in Cancer Informatics
This study systematically reviews and evaluates published research on machine learning models that integrate histopathology whole slide images and high-throughput -omic data to predict overall survival in cancer. A comprehensive search of PubMed, EMBASE, and Cochrane CENTRAL was conducted through August 12, 2024, with citation screening for additional studies. Eligible studies applied machine learning or deep learning methods to multimodal data combining pathology images and -omics. Data extraction followed the CHARMS checklist, and risk of bias was assessed using the PROBAST + AI tool. Narrative synthesis was conducted in line with PRISMA 2020 guidelines. Forty-eight studies published since 2017 met inclusion criteria, spanning 19 cancer types. All relied on The Cancer Genome Atlas dataset. Modelling approaches included regularised Cox regression (n = 4), classical machine learning (n = 13), and deep learning (n = 31). Reported concordance indices ranged from 0.550 to 0.857, with most multimodal models outperforming unimodal counterparts. However, all studies were assessed as having high or unclear risk of bias—most often due to limited external validation, insufficient reporting, and minimal assessment of clinical utility. This review highlights a rapidly evolving yet methodologically underdeveloped field. While model performance is promising, improvements in data standardisation, reporting practices, and real-world contextualisation are critical for clinical translation. This work was funded by the National Pathology Imaging Cooperative (NPIC), supported by UK Research and Innovation (Project no. 104687).