Interactive segmentation has recently been explored to effectively and efficiently harvest high-quality segmentation masks by iteratively incorporating user hints. While iterative in nature, most existing interactive segmentation methods tend to ignore the dynamics of successive interactions and take each interaction independently. We here propose to model iterative interactive image segmentation with a Markov decision process (MDP) and solve it with reinforcement learning (RL) where each voxel is treated as an agent. Considering the large exploration space for voxel-wise prediction and the dependence among neighboring voxels for the segmentation tasks, multi-agent reinforcement learning is adopted, where the voxel-level policy is shared among agents. Considering that boundary voxels are more important for segmentation, we further introduce a boundary-aware reward, which consists of a global reward in the form of relative cross-entropy gain, to update the policy in a constrained direction, and a boundary reward in the form of relative weight, to emphasize the correctness of boundary predictions. To combine the advantages of different types of interactions, i.e., simple and efficient for point-clicking, and stable and robust for scribbles, we propose a supervoxel-clicking based interaction design. Experimental results on four benchmark datasets have shown that the proposed method significantly outperforms the state-of-the-arts, with the advantage of fewer interactions, higher accuracy, and enhanced robustness.
12 July 2024 in International Journal of Computer Vision
Motivation: Chest X-ray (CXR) is a routine diagnostic X-ray examination for checking and screening various diseases. Automatically localizing and classifying diseases from CXR as a detection task is of much significance for subsequent diagnosis and treatment. Due to the fact that samples of some diseases are difficult to acquire, CXR detection datasets often present a long-tail distribution over different diseases. Objective: The detection performance of tail classes is very poor due to the limited number and diversity of samples in the training dataset and should be improved. Method: In this paper, motivated by a correspondence-based tracking system, we build a pipeline named RaTrack, leveraging a diffusion model to alleviate the tail class degradation problem by aligning the generation process of the tail to the head class. Then, the samples of rare classes are generated to extend the number and diversity of rare samples. In addition, we propose a filtering strategy to control the quality of the generated samples. Results: Extensive experiments on public datasets, Vindr-CXR and RSNA, demonstrate the effectiveness of the proposed method, especially for rare diseases.
Learning from a large corpus of data, pre-trained models have achieved impressive progress nowadays. As popular generative pre-training, diffusion models capture both low-level visual knowledge and high-level semantic relations. In this paper, we propose to exploit such knowledgeable diffusion models for mainstream discriminative tasks, i.e., unsupervised object discovery: saliency segmentation and object localization. However, the challenges exist as there is one structural difference between generative and discriminative models, which limits the direct use. Besides, the lack of explicitly labeled data significantly limits performance in unsupervised settings. To tackle these issues, we introduce DiffusionSeg, one novel synthesis-exploitation framework containing two-stage strategies. To alleviate data insufficiency, we synthesize abundant images, and propose a novel training-free AttentionCut to obtain masks in the first synthesis stage. In the second exploitation stage, to bridge the structural gap, we use the inversion technique, to map the given image back to diffusion features. These features can be directly used by downstream architectures. Extensive experiments and ablation studies demonstrate the superiority of adapting diffusion for unsupervised object discovery.