Spatially resolved transcriptomics enable comprehensive measurement of gene expression at subcellular resolution while preserving the spatial context of the tissue microenvironment. While deep learning has shown promise in analyzing SCST datasets, most efforts have focused on sequence data and spatial localization, with limited emphasis on leveraging rich histopathological insights from staining images. We introduce GIST, a deep learning-enabled gene expression and histology integration for spatial cellular profiling. GIST employs histopathology foundation models pretrained on millions of histology images to enhance feature extraction and a hybrid graph transformer model to integrate them with transcriptome features. Validated with datasets from human lung, breast, and colorectal cancers, GIST effectively reveals spatial domains and substantially improves the accuracy of segmenting the microenvironment after denoising transcriptomics data. This enhancement enables more accurate gene expression analysis and aids in identifying prognostic marker genes, outperforming state-of-the-art deep learning methods with a total improvement of up to 49.72%. GIST provides a generalizable framework for integrating histology with spatial transcriptome analysis, revealing novel insights into spatial organization and functional dynamics.

To avoid the irregularities during the level set evolution, a fractional distance regularized variational model is proposed for image segmentation. We first define a fractional distance regularization term which punishes the deviation of the level set function (LSF) and the signed distance function. Since the fractional derivative of the constant value function outside the starting point is nonzero, the fractional gradient modular of the LSF does not approach infinity where the integer order gradient modular is close to 0. This prevents the sharp reverse diffusion of LSF in flat areas and ensures the smooth evolution of LSF. Then, we use the Gr|nwald-Letnikov (G-L) fractional derivative to derive the discrete forms of the conjugate of fractional derivatives and fractional divergence. To facilitate the calculation of fractional derivatives and their conjugates, we designed their covering templates. Finally, a numerical solution to the minimization of the energy functional is obtained from these discrete forms and covering templates. Numerical experiments of medical images with different modalities show that the model in this paper can well segment weak images and intensity inhomogeneity images.

Applying deep learning to predict patient prognostic survival outcomes using histological whole-slide images (WSIs) and genomic data is challenging due to the morphological and transcriptomic heterogeneity present in the tumor microenvironment. Existing deep learning-enabled methods often exhibit learning biases, primarily because the genomic knowledge used to guide directional feature extraction from WSIs may be irrelevant or incomplete. This results in a suboptimal and sometimes myopic understanding of the overall pathological landscape, potentially overlooking crucial histological insights. To tackle these challenges, we propose the CounterFactual Bidirectional Co-Attention Transformer framework. By integrating a bidirectional co-attention layer, our framework fosters effective feature interactions between the genomic and histology modalities and ensures consistent identification of prognostic features from WSIs. Using counterfactual reasoning, our model utilizes causality to model unimodal and multimodal knowledge for cancer risk stratification. This approach directly addresses and reduces bias, enables the exploration of ’what-if' scenarios, and offers a deeper understanding of how different features influence survival outcomes. Our framework, validated across eight diverse cancer benchmark datasets from The Cancer Genome Atlas (TCGA), represents a major improvement over current histology-genomic model learning methods. It shows an average 2.5% improvement in c-index performance over 18 state-of-the-art models in predicting patient prognoses across eight cancer types. Our code is released at https://github.com/BusyJzy599/CFBCT-main.

Deep learning-based pavement cracks detection methods often require large-scale labels with detailed crack location information to learn accurate predictions. In practice, however, crack locations are very difficult to be manually annotated due to various visual patterns of pavement crack. In this paper, we propose a Deep Domain Adaptation-based Crack Detection Network (DDACDN), which learns domain invariant features by taking advantage of the source domain knowledge to predict the multi-category crack location information in the target domain, where only image-level labels are available. Specifically, DDACDN first extracts crack features from both the source and target domain by a two-branch weights-shared backbone network. And in an effort to achieve the cross-domain adaptation, an intermediate domain is constructed by aggregating the three-scale features from the feature space of each domain to adapt the crack features from the source domain to the target domain. Finally, the network involves the knowledge of both domains and is trained to recognize and localize pavement cracks. To facilitate accurate training and validation for domain adaptation, we use two challenging pavement crack datasets CQU-BPDD and RDD2020. Furthermore, we construct a new large-scale Bituminous Pavement Multi-label Disease Dataset named CQU-BPMDD, which contains 38994 high-resolution pavement disease images to further evaluate the robustness of our model. Extensive experiments demonstrate that DDACDN outperforms state-of-the-art pavement crack detection methods in predicting the crack location on the target domain.

Semi-supervised learning based on consistency learning offers significant promise for enhancing medical image segmentation. Current approaches use copy-paste as an effective data perturbation technique to facilitate weak-to-strong consistency learning. However, these techniques often lead to a decrease in the accuracy of synthetic labels corresponding to the synthetic data and introduce excessive perturbations to the distribution of the training data. Such over-perturbation causes the data distribution to stray from its true distribution, thereby impairing the model’s generalization capabilities as it learns the decision boundaries. We propose a weak-to-strong consistency learning framework that integrally addresses these issues with two primary designs: ① it emphasizes the use of highly reliable data to enhance the quality of labels in synthetic datasets through cross-copy-pasting between labeled and unlabeled datasets; ② it employs uncertainty estimation and foreground region constraints to meticulously filter the regions for copy-pasting, thus the copy-paste technique implemented introduces a beneficial perturbation to the training data distribution. Our framework expands the copy-paste method by addressing its inherent limitations, and amplifying the potential of data perturbations for consistency learning. We extensively validated our model using six publicly available medical image segmentation datasets across different diagnostic tasks, including the segmentation of cardiac structures, prostate structures, brain structures, skin lesions, and gastrointestinal polyps. The results demonstrate that our method significantly outperforms state-of-the-art models. For instance, on the PROMISE12 dataset for the prostate structure segmentation task, using only 10% labeled data, our method achieves a 15.31% higher Dice score compared to the baseline models. Our experimental code will be made publicly available at https://github.com/slhuang24/RCP4CL.