Purpose: To enable magnetic resonance imaging (MRI)-guided radiotherapy with real-time adaptation, motion must be quickly estimated with low latency. The motion estimate is used to adapt the radiation beam to the current anatomy, yielding a more conformal dose distribution. As the MR acquisition is the largest component of latency, deep learning (DL) may reduce the total latency by enabling much higher undersampling factors compared to conventional reconstruction and motion estimation methods. The benefit of DL on image reconstruction and motion estimation was investigated for obtaining accurate deformation vector fields (DVFs) with high temporal resolution and minimal latency.Methods: 2D cine MRI acquired at 1.5T from 135 abdominal cancer patients were retrospectively included in this study. Undersampled radial golden angle acquisitions were retrospectively simulated. DVFs were computed using different combinations of conventional- and DL-based methods for image reconstruction and motion estimation, allowing a comparison of four approaches to achieve real-time motion estimation. The four approaches were evaluated based on the end-point-error and root-mean-square error compared to a ground-truth optical flow estimate on fully-sampled images, the structural similarity (SSIM) after registration and time necessary to acquire k-space, reconstruct an image and estimate motion.Results: The lowest DVF error and highest SSIM were obtained using conventional methods up to R≤10. For undersampling factors R>10, the lowest DVF error and highest SSIM were obtained using conventional image reconstruction and DL-based motion estimation. We have found that, with this combination, accurate DVFs can be obtained up to R=25 with an average root-mean-square error up to 1 millimeter and an SSIM greater than 0.8 after registration, taking 60 milliseconds.Conclusion: High-quality 2D DVFs from highly undersampled k-space can be obtained with a high temporal resolution with conventional image reconstruction and a deep learning-based motion estimation approach for real-time adaptive MRI-guided radiotherapy.
Purpose: To enable real‐time adaptive magnetic resonance imaging–guided radiotherapy (MRIgRT) by obtaining time‐resolved three‐dimensional (3D) deformation vector fields (DVFs) with high spatiotemporal resolution and low latency ( ms).Theory and Methods: Respiratory‐resolved ‐weighted 4D‐MRI of 27 patients with lung cancer were acquired using a golden‐angle radial stack‐of‐stars readout. A multiresolution convolutional neural network (CNN) called TEMPEST was trained on up to 32 retrospectively undersampled MRI of 17 patients, reconstructed with a nonuniform fast Fourier transform, to learn optical flow DVFs. TEMPEST was validated using 4D respiratory‐resolved MRI, a digital phantom, and a physical motion phantom. The time‐resolved motion estimation was evaluated in‐vivo using two volunteer scans, acquired on a hybrid MR‐scanner with integrated linear accelerator. Finally, we evaluated the model robustness on a publicly‐available four‐dimensional computed tomography (4D‐CT) dataset.Results: TEMPEST produced accurate DVFs on respiratory‐resolved MRI at 20‐fold acceleration, with the average end‐point‐error mm, both on respiratory‐sorted MRI and on a digital phantom. TEMPEST estimated accurate time‐resolved DVFs on MRI of a motion phantom, with an error mm at 28 undersampling. On two volunteer scans, TEMPEST accurately estimated motion compared to the self‐navigation signal using 50 spokes per dynamic (366 undersampling). At this undersampling factor, DVFs were estimated within 200 ms, including MRI acquisition. On fully sampled CT data, we achieved a target registration error of mm without retraining the model.Conclusion: A CNN trained on undersampled MRI produced accurate 3D DVFs with high spatiotemporal resolution for MRIgRT.
Convolutional neural networks (CNNs) are increasingly adopted in medical imaging, e.g., to reconstruct high-quality images from undersampled magnetic resonance imaging (MRI) acquisitions or estimate subject motion during an examination. MRI is naturally acquired in the complex domain C, obtaining magnitude and phase information in k-space. However, CNNs in complex regression tasks are almost exclusively trained to minimize the L2 loss or maximizing the magnitude structural similarity (SSIM), which are possibly not optimal as they do not take full advantage of the magnitude and phase information present in the complex domain. This work identifies that minimizing the L2 loss in the complex field has an asymmetry in the magnitude/phase loss landscape and is biased, underestimating the reconstructed magnitude. To resolve this, we propose a new loss function for regression in the complex domain called ⊥-loss, which adds a novel phase term to established magnitude loss functions, e.g., L2 or SSIM. We show ⊥-loss is symmetric in the magnitude/phase domain and has favourable properties when applied to regression in the complex domain. Specifically, we evaluate the ⊥+ℓ2-loss and ⊥+SSIM-loss for complex undersampled MR image reconstruction tasks and MR image registration tasks. We show that training a model to minimize the ⊥+ℓ2-loss outperforms models trained to minimize the L2 loss and results in similar performance compared to models trained to maximize the magnitude SSIM while offering high-quality phase reconstruction. Moreover, ⊥-loss is defined in Rn, and we apply the loss function to the R2 domain by learning 2D deformation vector fields for image registration. We show that a model trained to minimize the ⊥+ℓ2-loss outperforms models trained to minimize the end-point error loss.
Background: Respiratory‐resolved four‐dimensional magnetic resonance imaging (4D‐MRI) provides essential motion information for accurate radiation treatments of mobile tumors. However, obtaining high‐quality 4D‐MRI suffers from long acquisition and reconstruction times.Purpose: To develop a deep learning architecture to quickly acquire and reconstruct high‐quality 4D‐MRI, enabling accurate motion quantification for MRI‐guided radiotherapy (MRIgRT).Methods: A small convolutional neural network called MODEST is proposed to reconstruct 4D‐MRI by performing a spatial and temporal decomposition, omitting the need for 4D convolutions to use all the spatio‐temporal information present in 4D‐MRI. This network is trained on undersampled 4D‐MRI after respiratory binning to reconstruct high‐quality 4D‐MRI obtained by compressed sensing reconstruction. The network is trained, validated, and tested on 4D‐MRI of 28 lung cancer patients acquired with a T1‐weighted golden‐angle radial stack‐of‐stars (GA‐SOS) sequence. The 4D‐MRI of 18, 5, and 5 patients were used for training, validation, and testing. Network performances are evaluated on image quality measured by the structural similarity index (SSIM) and motion consistency by comparing the position of the lung‐liver interface on undersampled 4D‐MRI before and after respiratory binning. The network is compared to conventional architectures such as a U‐Net, which has 30 times more trainable parameters.Results: MODEST can reconstruct high‐quality 4D‐MRI with higher image quality than a U‐Net, despite a thirty‐fold reduction in trainable parameters. High‐quality 4D‐MRI can be obtained using MODEST in approximately 2.5 min, including acquisition, processing, and reconstruction.Conclusion: High‐quality accelerated 4D‐MRI can be obtained using MODEST, which is particularly interesting for MRIgRT.
Skeletons are well-known descriptors used for analysis and processing of 2D binary images. Recently, dense skeletons have been proposed as an extension of classical skeletons as a dual encoding for 2D grayscale and color images. Yet, their encoding power, measured by the quality and size of the encoded image, and how these metrics depend on selected encoding parameters, has not been formally evaluated. In this paper, we fill this gap with two main contributions. First, we improve the encoding power of dense skeletons by effective layer selection heuristics, a refined skeleton pixel-chain encoding, and a postprocessing compression scheme. Secondly, we propose a benchmark to assess the encoding power of dense skeletons for a wide set of natural and synthetic color and grayscale images. We use this benchmark to derive optimal parameters for dense skeletons. Our method, called Compressing Dense Medial Descriptors (CDMD), achieves higher-compression ratios at similar quality to the well-known JPEG technique and, thereby, shows that skeletons can be an interesting option for lossy image encoding.
Microscopy-based phenotypic profiling relies increasingly on autonomous, unsupervised feature extraction, yet no existing method explicitly separates shape from texture into dedicated and independent latent subspaces by architectural design. Therefore texture, encoding critical biological information such as protein distribution and intracellular organisation, remains inaccessible as an independent feature domain in standard unsupervised approaches. This represents a fundamental limitation that prevents unbiased phenotypic analysis across biological scales. Here we introduce UDIST (Unsupervised Disentanglement of Shape and Texture), a sequential dual variational autoencoder (VAE) framework that tackles this fundamental limitation by explicitly decoupling shape from texture into independent, non-overlapping latent subspaces at the single-object level. By training two VICReg-regularised VAEs on principal-axis-aligned objects, UDIST separates binary shape from continuous texture information into rotation-invariant feature spaces, enabling separate downstream analysis of both domains. We validated UDIST across biological scales, from nuclei and single cells to patient-derived intestinal organoids, using both fluorescence and brightfield imaging, revealing phenotypic differences previously hidden by morphological variation and enabling the independent analysis of shape and texture in downstream analyses including clustering and similarity measurements. UDIST provides a versatile, label-free, and unsupervised tool for multi-scale phenotypic profiling in high-content microscopy and screening.