Histopathologic diagnosis relies on simultaneous integration of information from a broad range of scales, ranging from nuclear aberrations (≈O(0.1μm)) through cellular structures (≈O(10μm)) to the global tissue architecture (⪆O(1mm)). To explicitly mimic how human pathologists combine multi-scale information, we introduce a family of multi-encoder fully-convolutional neural networks with deep fusion. We present a simple block for merging model paths with differing spatial scales in a spatial relationship-preserving fashion, which can readily be included in standard encoder-decoder networks. Additionally, a context classification gate block is proposed as an alternative for the incorporation of global context. Our experiments were performed on three publicly available whole-slide images of recent challenges (PAIP 2019: hepatocellular carcinoma segmentation; BACH 2020: breast cancer segmentation; CAMELYON 2016: metastasis detection in lymph nodes). The multi-scale architectures consistently outperformed the baseline single-scale U-Nets by a large margin. They benefit from local as well as global context and particularly a combination of both. If feature maps from different scales are fused, doing so in a manner preserving spatial relationships was found to be beneficial. Deep guidance by a context classification loss appeared to improve model training at low computational costs. All multi-scale models had a reduced GPU memory footprint compared to ensembles of individual U-Nets trained on different image scales. Additional path fusions were shown to be possible at low computational cost, opening up possibilities for further, systematic and task-specific architecture optimisation. The findings demonstrate the potential of the presented family of human-inspired, end-to-end trainable, multi-scale multi-encoder fully-convolutional neural networks to improve deep histopathologic diagnosis by extensive integration of largely different spatial scales.

Is self-supervised deep learning (DL) for medical image analysis already a serious alternative to the de facto standard of end-to-end trained supervised DL? We tackle this question for medical image classification, with a particular focus on one of the currently most limiting factor of the field: the (non-)availability of labeled data. Based on three common medical imaging modalities (bone marrow microscopy, gastrointestinal endoscopy, dermoscopy) and publicly available data sets, we analyze the performance of self-supervised DL within the self-distillation with no labels (DINO) framework. After learning an image representation without use of image labels, conventional machine learning classifiers are applied. The classifiers are fit using a systematically varied number of labeled data (1–1000 samples per class). Exploiting the learned image representation, we achieve state-of-the-art classification performance for all three imaging modalities and data sets with only a fraction of between 1% and 10% of the available labeled data and about 100 labeled samples per class.

Background and Purpose: Mechanical thrombectomy is an established procedure for treatment of acute ischemic stroke. Mechanical thrombectomy success is commonly assessed by the Thrombolysis in Cerebral Infarction (TICI) score, assigned by visual inspection of X-ray digital subtraction angiography data. However, expert-based TICI scoring is highly observer-dependent. This represents a major obstacle for mechanical thrombectomy outcome comparison in, for instance, multicentric clinical studies. Focusing on occlusions of the M1 segment of the middle cerebral artery, the present study aimed to develop a deep learning (DL) solution to automated and, therefore, objective TICI scoring, to evaluate the agreement of DL- and expert-based scoring, and to compare corresponding numbers to published scoring variability of clinical experts. Methods: The study comprises 2 independent datasets. For DL system training and initial evaluation, an in-house dataset of 491 digital subtraction angiography series and modified TICI scores of 236 patients with M1 occlusions was collected. To test the model generalization capability, an independent external dataset with 95 digital subtraction angiography series was analyzed. Characteristics of the DL system were modeling TICI scoring as ordinal regression, explicit consideration of the temporal image information, integration of physiological knowledge, and modeling of inherent TICI scoring uncertainties. Results: For the in-house dataset, the DL system yields Cohen’s kappa, overall accuracy, and specific agreement values of 0.61, 71%, and 63% to 84%, respectively, compared with the gold standard: the expert rating. Values slightly drop to 0.52/64%/43% to 87% when the model is, without changes, applied to the external dataset. After model updating, they increase to 0.65/74%/60% to 90%. Literature Cohen’s kappa values for expert-based TICI scoring agreement are in the order of 0.6. Conclusions: The agreement of DL- and expert-based modified TICI scores in the range of published interobserver variability of clinical experts highlights the potential of the proposed DL solution to automated TICI scoring.

Background and Objective: Classification of peripheral blood and bone marrow cells is critical in the diagnosis and monitoring of hematological disorders. The development of robust and reliable automatic classification systems is hampered by data scarcity and limited model generalizability across laboratories. The present study proposes the integration of self-supervised learning (SSL) into cell classification pipelines to address these challenges. Methods: The experiments are based on four public hematological single cell image datasets: one bone marrow and three peripheral blood datasets. The cell classification pipeline consists of two parts: (1) SSL-based image feature extraction without the use of image annotations, and (2) a lightweight machine learning classifier applied to the SSL features and trained on only a small number of annotated images. Results: Direct transfer of SSL models trained on bone marrow data to peripheral blood data resulted in higher balanced classification accuracy than the transfer of supervised deep learning counterparts for all blood datasets. After adaptation of the lightweight machine learning classifier with 50 labeled samples per class of the new dataset, the SSL pipeline surpasses supervised deep learning classification performance for one dataset and classes with rare or atypical cell types and performs similarly on the other datasets. Conclusions: The results demonstrate that SSL enables (1) extraction of meaningful cell image features without the use of cell class information; (2) efficient transfer of knowledge between bone marrow and peripheral blood cell domains; and (3) efficient model adaptation to new datasets using only a few labeled data samples.

Missing data present a major challenge for deep learning, and various imputation techniques exist. However, imputation quality generally decreases as missing data rates increase. In time-series data from electronic health records, missing rates for individual parameters up to 99% can be observed, posing a problem for mere imputation. In addition, underlying patterns between missing and observed data can contain valuable information that can be learned by deep learning models. In this work, we propose a strategy called missing adjusted normalization and nullity encoding representation strategy (MANNERS), which can be applied to training pipelines for representation learning. MANNERS encodes missingness, masks loss evaluation at missing data points, and applies rebalancing such that the signal from variables with high missing rates is not lost. We evaluated MANNERS on reconstruction and downstream classification, regression, and synthetic data generation tasks. We showed performance improvements in the presence of very high missing rates compared with state-of-the-art imputation-only techniques.