Skin lesion segmentation from dermoscopy images is of great significance in the quantitative analysis of skin cancers, which is yet challenging even for dermatologists due to the inherent issues, i.e., considerable size, shape and color variation, and ambiguous boundaries. Recent vision transformers have shown promising performance in handling the variation through global context modeling. Still, they have not thoroughly solved the problem of ambiguous boundaries as they ignore the complementary usage of the boundary knowledge and global contexts. In this paper, we propose a novel cross-scale boundary-aware transformer, XBound-Former, to simultaneously address the variation and boundary problems of skin lesion segmentation. XBound-Former is a purely attention-based network and catches boundary knowledge via three specially designed learners. First, we propose an implicit boundary learner (im-Bound) to constrain the network attention on the points with noticeable boundary variation, enhancing the local context modeling while maintaining the global context. Second, we propose an explicit boundary learner (ex-Bound) to extract the boundary knowledge at multiple scales and convert it into embeddings explicitly. Third, based on the learned multi-scale boundary embeddings, we propose a cross-scale boundary learner (X-Bound) to simultaneously address the problem of ambiguous and multi-scale boundaries by using learned boundary embedding from one scale to guide the boundary-aware attention on the other scales. We evaluate the model on two skin lesion datasets and one polyp lesion dataset, where our model consistently outperforms other convolution- and transformer-based models, especially on the boundary-wise metrics. All resources could be found in https://github.com/jcwang123/xboundformer.

Personalized federated learning (PFL) addresses the data heterogeneity challenge faced by general federated learning (GFL). Rather than learning a single global model, with PFL a collection of models are adapted to the unique feature distribution of each site. However, current PFL methods rarely consider self-attention networks which can handle data heterogeneity by long-range dependency modeling and they do not utilize prediction inconsistencies in local models as an indicator of site uniqueness. In this paper, we propose FedDP , a novel federated learning scheme with dual personalization, which improves model personalization from both feature and prediction aspects to boost image segmentation results. We leverage long-range dependencies by designing a local query (LQ) that decouples the query embedding layer out of each local model, whose parameters are trained privately to better adapt to the respective feature distribution of the site. We then propose inconsistency-guided calibration (IGC), which exploits the inter-site prediction inconsistencies to accommodate the model learning concentration. By encouraging a model to penalize pixels with larger inconsistencies, we better tailor prediction-level patterns to each local site. Experimentally, we compare FedDP with the state-of-the-art PFL methods on two popular medical image segmentation tasks with different modalities, where our results consistently outperform others on both tasks. Our code and models will be available at https://github.com/jcwang123/PFL-Seg-Trans.

Medical image segmentation is crucial in clinical diagnosis, helping physicians identify and analyze medical conditions. However, this task is often accompanied by challenges like sensitive data, privacy concerns, and expensive annotations. Current research focuses on personalized collaborative training of medical segmentation systems, ignoring that obtaining segmentation annotations is time-consuming and laborious. Achieving a perfect balance between annotation cost and segmentation performance while ensuring local model personalization has become a valuable direction. Therefore, this study introduces a novel Model-Heterogeneous Semi-Supervised Federated (HSSF) Learning framework. It proposes Regularity Condensation and Regularity Fusion to transfer autonomously selective knowledge to ensure the personalization between sites. In addition, to efficiently utilize unlabeled data and reduce the annotation burden, it proposes a Self-Assessment (SA) module and a Reliable Pseudo-Label Generation (RPG) module. The SA module generates self-assessment confidence in real-time based on model performance, and the RPG module generates reliable pseudo-label based on SA confidence. We evaluate our model separately on the Skin Lesion and Polyp Lesion datasets. The results show that our model performs better than other methods characterized by heterogeneity. Moreover, it exhibits highly commendable performance even in homogeneous designs, most notably in region-based metrics. The full range of resources can be readily accessed through the designated repository located at HSSF(github.com) on the platform of GitHub.

Automated detecting lung infections from computed tomography (CT) data plays an important role for combating coronavirus 2019 (COVID-19). However, there are still some challenges for developing AI system: 1) most current COVID-19 infection segmentation methods mainly relied on 2-D CT images, which lack 3-D sequential constraint; 2) existing 3-D CT segmentation methods focus on single-scale representations, which do not achieve the multiple level receptive field sizes on 3-D volume; and 3) the emergent breaking out of COVID-19 makes it hard to annotate sufficient CT volumes for training deep model. To address these issues, we first build a multiple dimensional-attention convolutional neural network (MDA-CNN) to aggregate multiscale information along different dimension of input feature maps and impose supervision on multiple predictions from different convolutional neural networks (CNNs) layers. Second, we assign this MDA-CNN as a basic network into a novel dual multiscale mean teacher network (DM ${^{2}}$ T-Net) for semi-supervised COVID-19 lung infection segmentation on CT volumes by leveraging unlabeled data and exploring the multiscale information. Our DM ${^{2}}$ T-Net encourages multiple predictions at different CNN layers from the student and teacher networks to be consistent for computing a multiscale consistency loss on unlabeled data, which is then added to the supervised loss on the labeled data from multiple predictions of MDA-CNN. Third, we collect two COVID-19 segmentation datasets to evaluate our method. The experimental results show that our network consistently outperforms the compared state-of-the-art methods.

Multimodal Federated Learning (MFL) has emerged as a collaborative paradigm for training models across decentralized devices, harnessing various data modalities to facilitate effective learning while respecting data ownership. In this realm, notably, a pivotal shift from homogeneous to heterogeneous MFL has taken place. While the former assumes uniformity in input modalities across clients, the latter accommodates modality-incongruous setups, which is often the case in practical situations. For example, while some advanced medical institutions have the luxury of utilizing both MRI and CT for disease diagnosis, remote hospitals often find themselves constrained to employ CT exclusively due to its cost-effectiveness. Although heterogeneous MFL can apply to a broader scenario, it introduces a new challenge: modality-heterogeneous client drift, arising from diverse modality-coupled local optimization. To address this, we introduce FedMM, a simple yet effective approach. During local optimization, FedMM employs modality dropout, randomly masking available modalities, and promoting weight alignment while preserving model expressivity on its original modality combination. To enhance the modality dropout process, FedMM incorporates a task-specific inter- and intra-modal regularizer, which acts as an additional constraint, forcing that weight distribution remains more consistent across diverse input modalities and therefore eases the optimization process with modality dropout enabled. By combining them, our approach holistically addresses client drift. It fosters convergence among client models while considering each client’s unique input modalities, enhancing heterogeneous MFL performance. Comprehensive evaluations in three medical image segmentation datasets demonstrate FedMM’s superiority over state-of-the-art heterogeneous MFL methods.