29 March 2023 in IEEE Transactions on Medical Imaging
The rapid advances in deep learning-based computational pathology and radiology have demonstrated the promise of using whole slide images (WSIs) and radiology images for survival prediction in cancer patients. However, most image-based survival prediction methods are limited to using either histology or radiology alone, leaving integrated approaches across histology and radiology relatively underdeveloped. There are two main challenges in integrating WSIs and radiology images: (1) the gigapixel nature of WSIs and (2) the vast difference in spatial scales between WSIs and radiology images. To address these challenges, in this work, we propose an interpretable, weaklysupervised, multimodal learning framework, called Hierarchical Multimodal Co-Attention Transformer (HMCAT), to integrate WSIs and radiology images for survival prediction. Our approach first uses hierarchical feature extractors to capture various information including cellular features, cellular organization, and tissue phenotypes in WSIs. Then the hierarchical radiology-guided co-attention (HRCA) in HMCAT characterizes the multimodal interactions between hierarchical histology-based visual concepts and radiology features and learns hierarchical co-attention mappings for two modalities. Finally, HMCAT combines their complementary information into a multimodal risk score and discovers prognostic features from two modalities by multimodal interpretability. We apply our approach to two cancer datasets (365 WSIs with matched magnetic resonance [MR] images and 213 WSIs with matched computed tomography [CT] images). Our results demonstrate that the proposed HMCAT consistently achieves superior performance over the unimodal approaches trained on either histology or radiology data alone, as well as other state-of-the-art methods.
5 January 2026 in Nature Medicine
Spatial proteomics enables high-resolution mapping of protein expression and can transform our understanding of biology and disease. However, major challenges remain for clinical translation, including cost, complexity and scalability. Here we present H&E to protein expression (HEX), an AI model designed to computationally generate spatial proteomics profiles from standard histopathology slides. Trained and validated on 819,000 histopathology image tiles with matched protein expression from 382 tumor samples, HEX accurately predicts the expression of 40 biomarkers encompassing immune, structural and functional programs. HEX demonstrates substantial performance gains over alternative methods for protein expression prediction from H&E images. We develop a multimodal data integration approach that combines the original H&E image and AI-derived virtual spatial proteomics to enhance outcome prediction. Applied to six independent non-small-cell lung cancer cohorts totaling 2,298 patients, HEX-enabled multimodal integration improved prognostic accuracy by 22% and immunotherapy response prediction by 24–39% compared with conventional clinicopathological and molecular biomarkers. Biological interpretation revealed spatially organized tumor–immune niches predictive of therapeutic response, including the co-localization of T helper cells and cytotoxic T cells in responders, and immunosuppressive tumor-associated macrophage and neutrophil aggregates in non-responders. HEX provides a low-cost and scalable approach to study spatial biology and enables the discovery and clinical translation of interpretable biomarkers for precision medicine.
29 April 2025 in Cancer Discovery
The complementarity and clinical utility of combining liquid biopsies and radiomic image analysis has not been demonstrated. ctDNA minimal residual disease after chemoradiotherapy (CRT) for non–small cell lung cancer (NSCLC) is highly prognostic, but on-treatment biomarkers are needed to enable response-adapted therapies. In this study, we analyzed 418 patients with NSCLC undergoing CRT to develop and validate a novel dynamic risk model that accurately predicts ultimate progression-free survival during treatment. We optimize tissue-free variant calling from plasma samples to facilitate ctDNA monitoring and demonstrate the importance of accounting for persistent clonal hematopoiesis variants. We show that mid-CRT ctDNA concentration is prognostic for disease progression and integrate additional pre-CRT risk factors, including radiomics, into a combined model that improves outcome prediction. Our results suggest that tumor features, radiomics, and mid-CRT ctDNA analysis are complementary and can identify patients at high and low risk of progression to potentially enable response-adapted therapies. Significance: This study demonstrates that combining tumor features, radiomics, and ctDNA analysis improves outcome prediction in NSCLC treated with CRT therapy. Our integrated model could enable personalized and response-adapted therapies to reduce toxicity and improve outcomes in patients.
7 July 2025 in Nature Communications
SMSeyed Hossein Mirjahanmardi
Seyed Hossein Mirjahanmardi
+4
Histopathology with hematoxylin and eosin (H&E) staining is routinely employed for clinical diagnoses. Single-cell analysis of histopathology provides a powerful tool for understanding the intricate cellular interactions underlying disease progression and therapeutic response. However, existing efforts are hampered by inefficient and error-prone human annotations. Here, we present an experimental and computational approach for automated cell annotation and classification on H&E-stained images. Instead of human annotations, we use multiplexed immunofluorescence (mIF) to define cell types based on cell lineage protein markers. By co-registering H&E images with mIF of the same tissue section at the single-cell level, we create a dataset of 1,127,252 cells with high-quality annotations on tissue microarray cores. A deep learning model combining self-supervised learning with domain adaptation is trained to classify four cell types on H&E images with an overall accuracy of 86%-89%, and the cell classification model is applicable to whole slide images. Further, we show that spatial interactions among specific immune cells in the tumor microenvironment are linked to patient survival and response to immune checkpoint inhibitors. Our work provides a scalable approach for single-cell analysis of standard histopathology and may enable discovery of novel spatial biomarkers for precision oncology.
1 December 2017 in Neurocomputing
1 December 2025 in IEEE Journal of Biomedical and Health Informatics
Computational pathology has emerged as a transformative paradigm by leveraging artificial intelligence to automate and enhance diagnostic procedures. However, existing models often target narrow tasks or specific tumor types, missing opportunities to unify diverse datasets and tasks through joint learning. In this work, we introduce PathBot, a foundation model tailored for comprehensive pathological image analysis. Central to PathBot is a ViTGiant encoder with one billion parameters, the largest model to date trained on publicly available pathological data. We pre-train this encoder using a novel Masked Distillation Network (MDN) and an integrated learning strategy that combines contrastive and generative objectives. The pretraining leverages over 30 million image patches derived from 11,765 whole slide images (WSIs) across 32 cancer types in the Cancer Genome Atlas (TCGA). To evaluate its versatility, we pair the encoder with task-specific decoders for segmentation, detection, classification, and regression. Extensive experiments across 20 downstream tasks demonstrate that PathBot achieves state-of-the-art performance in most cases, showcasing its robustness and generalizability. Code and models will be released to support further research.