Rheumatoid arthritis (RA) is a chronic autoimmune disease that can cause severe joint damage and functional impairment. Ultrasound imaging has shown promise in providing real-time assessment of synovium inflammation associated with the early stages of RA. Accurate segmentation of the synovium region and quantification of inflammation-specific imaging biomarkers are crucial for assessing and grading RA. However, automatic segmentation of the synovium in 3D ultrasound is challenging due to ambiguous boundaries, variability in synovium shape, and inhomogeneous intensity distribution. In this work, we introduce a novel network architecture, Swin Transformers with Deep Attentive Features for 3D segmentation (SwinDAF3D), which integrates Swin Transformers into a Deep Attentive Features framework. The developed architecture leverages the hierarchical structure and shifted windows of Swin Transformers to capture rich, multi-scale and attentive contextual information, improving the modeling of long-range dependencies and spatial hierarchies in 3D ultrasound images. In a six-fold cross-validation study with 3D ultrasound images of RA patients’ finger joints (n = 72), our SwinDAF3D model achieved the highest performance with a Dice Score (DSC) of 0.838 ± 0.013, an Intersection over Union (IoU) of 0.719 ± 0.019, and Surface Dice Score (SDSC) of 0.852 ± 0.020, compared to 3D UNet (DSC: 0.742 ± 0.025; IoU: 0.589 ± 0.031; SDSC: 0.661 ± 0.029), DAF3D (DSC: 0.813 ± 0.017; IoU: 0.689 ± 0.022; SDSC: 0.817 ± 0.013), Swin UNETR (DSC: 0.808 ± 0.025; IoU: 0.678 ± 0.032; SDSC: 0.822 ± 0.039), UNETR++ (DSC: 0.810 ± 0.014; IoU: 0.684 ± 0.018; SDSC: 0.829 ± 0.027) and TransUNet (DSC: 0.818 ± 0.013; IoU: 0.692 ± 0.017; SDSC: 0.815 ± 0.016) models. This ablation study demonstrates the effectiveness of combining a Swin Transformers feature pyramid with a deep attention mechanism, improving the segmentation accuracy of the synovium in 3D ultrasound. This advancement shows great promise in enabling more efficient and standardized RA screening using ultrasound imaging.
Aiming at an intelligent point-of-care imaging technology for rheumatology clinics, a fully automatic 3D photoacoustic (PA) and ultrasound (US) dual-modality system driven by a robot and powered by deep learning (DL)-based image processing was developed. Automated scanning of volumetric images from patient joints, plus DL-based tissue segmentation and quantification of imaging biomarkers, ensures that the measurements from this system are objective and reproducible. Clinical validation was conducted via a longitudinal study on 43 finger joints from patients affected by inflammatory arthritis. Using manual segmentations as the gold standard, our DL algorithm utilizing the 3D Deep Attentive Feature (DAF3D) model showed satisfactory performance in automatic segmentation of joint space and synovial region and achieved a Dice score of 0.77±0.03 and an IoU of 0.64±0.03. Based on the tissue segmentations facilitated by the DAF3D model, six volumetric imaging biomarkers reflecting the activity of arthritis and its change in response to the treatment were quantified, including hyperemia, blood oxygenation, US power Doppler, joint space echogenicity, joint space volume, and synovial volume. The imaging biomarkers quantified from DL-based segmentation and manual segmentation showed moderate to strong correlations (: 0.41−0.86). Hyperemia quantified from PA imaging has the strongest association with the disease activity indicated by Clinical Assessment Questionnaire (CAQ) scores, with =0.41. Linear models combining the two biomarkers from PA imaging, the four biomarkers from US imaging, and all six imaging biomarkers rendered moderate to very strong associations with the disease activity scores, with of 0.41, 0.26, and 0.52, respectively.