Molecular profiling is central in cancer precision medicine but remains costly and is based on tumor average profiles. Morphologic patterns observable in histopathology sections from tumors are determined by the underlying molecular phenotype and therefore have the potential to be exploited for prediction of molecular phenotypes. We report here the first transcriptome-wide expression–morphology (EMO) analysis in breast cancer, where individual deep convolutional neural networks were optimized and validated for prediction of mRNA expression in 17,695 genes from hematoxylin and eosin–stained whole slide images. Predicted expressions in 9,334 (52.75%) genes were significantly associated with RNA sequencing estimates. We also demonstrated successful prediction of an mRNA-based proliferation score with established clinical value. The results were validated in independent internal and external test datasets. Predicted spatial intratumor variabilities in expression were validated through spatial transcriptomics profiling. These results suggest that EMO provides a cost-efficient and scalable approach to predict both tumor average and intratumor spatial expression from histopathology images.Significance: Transcriptome-wide expression morphology deep learning analysis enables prediction of mRNA expression and proliferation markers from routine histopathology whole slide images in breast cancer.

In pathology and biomedical research, histology is the cornerstone method for tissue analysis. Currently, the histological workflow consumes plenty of chemicals, water, and time for staining procedures. Deep learning is now enabling digital replacement of parts of the histological staining procedure. In virtual staining, histological stains are created by training neural networks to produce stained images from an unstained tissue image, or through transferring information from one stain to another. These technical innovations provide more sustainable, rapid, and cost-effective alternatives to traditional histological pipelines, but their development is in an early phase and requires rigorous validation. In this review we cover the basic concepts of virtual staining for histology and provide future insights into the utilization of artificial intelligence (AI)-enabled virtual histology.

Several algorithms have been proposed for detecting fluorescently labeled subcellular objects in microscope images. Many of these algorithms have been designed for specific tasks and validated with limited image data. But despite the potential of using extensive comparisons between algorithms to provide useful information to guide method selection and thus more accurate results, relatively few studies have been performed.

Immunohistochemistry (IHC) of ER, PR, and Ki-67 are routinely used assays in breast cancer diagnostics. Determination of the proportion of stained cells (labeling index) should be restricted on malignant epithelial cells, carefully avoiding tumor infiltrating stroma and inflammatory cells. Here, we developed a deep learning based digital mask for automated epithelial cell detection using fluoro-chromogenic cytokeratin-Ki-67 double staining and sequential hematoxylin-IHC staining as training material. A partially pre-trained deep convolutional neural network was fine-tuned using image batches from 152 patient samples of invasive breast tumors. Validity of the trained digital epithelial cell masks was studied with 366 images captured from 98 unseen samples, by comparing the epithelial cell masks to cytokeratin images and by visual evaluation of the brightfield images performed by two pathologists. A good discrimination of epithelial cells was achieved (AUC of mean ROC = 0.93; defined as the area under mean receiver operating characteristics), and well in concordance with pathologists’ visual assessment (4.01/5 and 4.67/5). The effect of epithelial cell masking on the Ki-67 labeling index was substantial. 52 tumor images initially classified as low proliferation (Ki-67<14%) without epithelial cell masking were re-classified as high proliferation (Ki-67≥14%) after applying the deep learning based epithelial cell mask. The digital epithelial cell masks were found applicable also to IHC of ER and PR. We conclude that deep learning can be applied to detect carcinoma cells in breast cancer samples stained with conventional brightfield IHC.

Digital pathology enables new approaches that expand beyond storage, visualization or analysis of histological samples in digital format. One novel opportunity is 3D histology, where a three-dimensional reconstruction of the sample is formed computationally based on serial tissue sections. This allows examining tissue architecture in 3D, for example, for diagnostic purposes. Importantly, 3D histology enables joint mapping of cellular morphology with spatially resolved omics data in the true 3D context of the tissue at microscopic resolution. Several algorithms have been proposed for the reconstruction task, but a quantitative comparison of their accuracy is lacking. We developed a benchmarking framework to evaluate the accuracy of several free and commercial 3D reconstruction methods using two whole slide image datasets. The results provide a solid basis for further development and application of 3D histology algorithms and indicate that methods capable of compensating for local tissue deformation are superior to simpler approaches. Code: https://github.com/BioimageInformaticsTampere/RegBenchmark. Whole slide image datasets: http://urn.fi/urn: nbn: fi: csc-kata20170705131652639702. Supplementary data are available at Bioinformatics online.