To investigate the chest radiograph classification performance of vision transformers (ViTs) and interpretability of attention-based saliency maps, using the example of pneumothorax classification. In this retrospective study, ViTs were fine-tuned for lung disease classification using four public datasets: CheXpert, Chest X-Ray 14, MIMIC CXR, and VinBigData. Saliency maps were generated using transformer multimodal explainability and gradient-weighted class activation mapping (GradCAM). Classification performance was evaluated on the Chest X-Ray 14, VinBigData, and Society for Imaging Informatics in Medicine–American College of Radiology (SIIM-ACR) Pneumothorax Segmentation datasets using the area under the receiver operating characteristic curve (AUC) analysis and compared with convolutional neural networks (CNNs). The explainability methods were evaluated with positive and negative perturbation, sensitivity-n, effective heat ratio, intra-architecture repeatability, and interarchitecture reproducibility. In the user study, three radiologists classified 160 chest radiographs with and without saliency maps for pneumothorax and rated their usefulness. ViTs had comparable chest radiograph classification AUCs compared with state-of-the-art CNNs: 0.95 (95% CI: 0.94, 0.95) versus 0.83 (95%, CI 0.83, 0.84) on Chest X-Ray 14, 0.84 (95% CI: 0.77, 0.91) versus 0.83 (95% CI: 0.76, 0.90) on VinBigData, and 0.85 (95% CI: 0.85, 0.86) versus 0.87 (95% CI: 0.87, 0.88) on SIIM-ACR. Both saliency map methods unveiled a strong bias toward pneumothorax tubes in the models. Radiologists found 47% of the attention-based and 39% of the GradCAM saliency maps useful. The attention-based methods outperformed GradCAM on all metrics. ViTs performed similarly to CNNs in chest radiograph classification, and their attention-based saliency maps were more useful to radiologists and outperformed GradCAM. Keywords: Conventional Radiography, Thorax, Diagnosis, Supervised Learning, Convolutional Neural Network (CNN) Online supplemental material is available for this article. © RSNA, 2023

Background: Automated segmentation of spinal magnetic resonance imaging (MRI) plays a vital role both scientifically and clinically. However, accurately delineating posterior spine structures is challenging. Methods: This retrospective study, approved by the ethical committee, involved translating T1-weighted and T2-weighted images into computed tomography (CT) images in a total of 263 pairs of CT/MR series. Landmark-based registration was performed to align image pairs. We compared two-dimensional (2D) paired — Pix2Pix, denoising diffusion implicit models (DDIM) image mode, DDIM noise mode — and unpaired (SynDiff, contrastive unpaired translation) image-to-image translation using “peak signal-to-noise ratio” as quality measure. A publicly available segmentation network segmented the synthesized CT datasets, and Dice similarity coefficients (DSC) were evaluated on in-house test sets and the “MRSpineSeg Challenge” volumes. The 2D findings were extended to three-dimensional (3D) Pix2Pix and DDIM. Results: 2D paired methods and SynDiff exhibited similar translation performance and DCS on paired data. DDIM image mode achieved the highest image quality. SynDiff, Pix2Pix, and DDIM image mode demonstrated similar DSC (0.77). For craniocaudal axis rotations, at least two landmarks per vertebra were required for registration. The 3D translation outperformed the 2D approach, resulting in improved DSC (0.80) and anatomically accurate segmentations with higher spatial resolution than that of the original MRI series. Conclusions: Two landmarks per vertebra registration enabled paired image-to-image translation from MRI to CT and outperformed all unpaired approaches. The 3D techniques provided anatomically correct segmentations, avoiding underprediction of small structures like the spinous process. Relevance statement: This study addresses the unresolved issue of translating spinal MRI to CT, making CT-based tools usable for MRI data. It generates whole spine segmentation, previously unavailable in MRI, a prerequisite for biomechanical modeling and feature extraction for clinical applications. Key points: • Unpaired image translation lacks in converting spine MRI to CT effectively. • Paired translation needs registration with two landmarks per vertebra at least. • Paired image-to-image enables segmentation transfer to other domains. • 3D translation enables super resolution from MRI to CT. • 3D translation prevents underprediction of small structures. Graphical

Objectives: To present a publicly available deep learning-based torso segmentation model that provides comprehensive voxel-wise coverage, including delineations that extend to the boundaries of anatomical compartments. Materials and methods: We extracted preliminary segmentations from TotalSegmentator, spine, and body composition models for magnetic resonance tomography (MR) images, then improved them iteratively and retrained an nnUNet model. Using a random retrospective subset of German National Cohort (NAKO), UK Biobank, internal MR and computed tomography (CT) data (Training: 2897 series from 626 subjects, 290 female; mean age 53 ± 16; 3-fold-cross validation (20% hold-out). Internal testing 36 series from 12 subjects, 6 male; mean age 60 ± 11), we segmented 71 structures in torso MR and 72 in CT images: 20 organs, 10 muscles, 19 vessels, 16 bones, ribs in CT, intervertebral discs, spinal cord, spinal canal and body composition (subcutaneous fat, unclassified muscles and visceral fat). For external validation, we used existing automatic organ segmentations, independent ground truth segmentations on gradient echo images, and the Amos data. We used non-parametric bootstrapping for confidence intervals and the Wilcoxon rank-sum test for computing statistical significance. Results: We achieved an average Dice score of 0.90 ± 0.06 on our internal gradient echo test set, which included 71 semantic segmentation labels. Our model ties with the best model on Amos with a Dice of 0,81 ± 0.14, while having a larger field of view and a considerably higher number of structures included. Conclusion: Our work presents a publicly available full-torso segmentation model for MRI and CT images that classifies almost all subject voxels to date. Key Points: Question No completed MRI segmentation model exists that delineates the true transition boundaries of the anatomical structures of bone and muscles . Findings We provide a simple-to-use model that automatically segments MRI images, that can be utilized as a backbone for computer-aided automatic analysis . Clinical relevance Our segmentation model enables accurate and detailed full-torso segmentation on MRI and CT, improving automated analysis in large-scale epidemiological studies and facilitating more precise body composition and organ assessments for clinical and research applications . Graphical

Counterfactual explanations (CEs) aim to enhance the interpretability of machine learning models by illustrating how alterations in input features would affect the resulting predictions. Common CE approaches require an additional model and are typically constrained to binary counterfactuals. In contrast, we propose a novel method that operates directly on the latent space of a generative model, specifically a Diffusion Autoencoder (DAE). This approach offers inherent interpretability by enabling the generation of CEs and the continuous visualization of the model’s internal representation across decision boundaries. Our method leverages the DAE’s ability to encode images into a semantically rich latent space in an unsupervised manner, eliminating the need for labeled data or separate feature extraction models. We show that these latent representations are helpful for medical condition classification and the ordinal regression of severity pathologies, such as vertebral compression fractures (VCF) and diabetic retinopathy (DR). Beyond binary CEs, our method supports the visualization of ordinal CEs using a linear model, providing deeper insights into the model’s decision-making process and enhancing interpretability. Experiments across various medical imaging datasets demonstrate the method’s advantages in interpretability and versatility. The linear manifold of the DAE’s latent space allows for meaningful interpolation and manipulation, making it a powerful tool for exploring medical image properties. Our code is available at https://github.com/matanat/dae_counterfactual

Objectives: To generate sagittal T1-weighted fast spin echo (T1w FSE) and short tau inversion recovery (STIR) images from sagittal T2-weighted (T2w) FSE and axial T1w gradient echo Dixon technique (T1w-Dixon) sequences. Materials and methods: This retrospective study used three existing datasets: “Study of Health in Pomerania” (SHIP, 3142 subjects, 1.5 Tesla), “German National Cohort” (NAKO, 2000 subjects, 3 Tesla), and an internal dataset (157 patients 1.5/3 Tesla). We generated synthetic sagittal T1w FSE and STIR images from sagittal T2w FSE and low-resolution axial T1w-Dixon sequences based on two successively applied 3D Pix2Pix deep learning models. “Peak signal-to-noise ratio” (PSNR) and “structural similarity index metric” (SSIM) were used to evaluate the generated image quality on an ablations test. A Turing test, where seven radiologists rated 240 images as either natively acquired or generated, was evaluated using misclassification rate and Fleiss kappa interrater agreement. Results: Including axial T1w-Dixon or T1w FSE images resulted in higher image quality in generated T1w FSE (PSNR = 26.942, SSIM = 0.965) and STIR (PSNR = 28.86, SSIM = 0.948) images compared to using only single T2w images as input (PSNR = 23.076/24.677 SSIM = 0.952/0.928). Radiologists had difficulty identifying generated images (misclassification rate: 0.39 ± 0.09 for T1w FSE, 0.42 ± 0.18 for STIR) and showed low interrater agreement on suspicious images (Fleiss kappa: 0.09 for T1w/STIR). Conclusions: Axial T1w-Dixon and sagittal T2w FSE images contain sufficient information to generate sagittal T1w FSE and STIR images. Clinical relevance statement: T1w fast spin echo and short tau inversion recovery can be retroactively added to existing datasets, saving MRI time and enabling retrospective analysis, such as evaluating bone marrow pathologies. Key Points: Sagittal T2-weighted images alone were insufficient for differentiating fat and water and to generate T1-weighted images. Axial T1w Dixon technique, together with a T2-weighted sequence, produced realistic sagittal T1-weighted images. Our approach can be used to retrospectively generate STIR and T1-weighted fast spin echo sequences. Graphical