Shaoan Xie

Shaoan Xie

Auto-generated profile, based on publicly available information

Shaoan Xie

Carnegie Mellon University

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
9 December 2022 in arXiv
Verified
Preprint
Open access
PDF file
SXShaoan Xie
Shaoan Xie
ZZZhifei Zhang
Zhifei Zhang
ZLZhe Lin
Zhe Lin
THTobias Hinz
Tobias Hinz
KZKun Zhang
Kun Zhang
Generic image inpainting aims to complete a corrupted image by borrowing surrounding information, which barely generates novel content. By contrast, multi-modal inpainting provides more flexible and useful controls on the inpainted content, \eg, a text prompt can be used to describe an object with richer attributes, and a mask can be used to constrain the shape of the inpainted object rather than being only considered as a missing area. We propose a new diffusion-based model named SmartBrush for completing a missing region with an object using both text and shape-guidance. While previous work such as DALLE-2 and Stable Diffusion can do text-guided inapinting they do not support shape guidance and tend to modify background texture surrounding the generated object. Our model incorporates both text and shape guidance with precision control. To preserve the background better, we propose a novel training and sampling strategy by augmenting the diffusion U-net with object-mask prediction. Lastly, we introduce a multi-task training strategy by jointly training inpainting with text-to-image generation to leverage more training data. We conduct extensive experiments showing that our model outperforms all baselines in terms of visual quality, mask controllability, and background preservation.

5 December 2023 in arXiv
Verified
Preprint
Open access
PDF file
SXShaoan Xie
Shaoan Xie
YZYang Zhao
Yang Zhao
ZXZhisheng Xiao
Zhisheng Xiao
KCKelvin C. K. Chan
Kelvin C. K. Chan
YLYandong Li
Yandong Li
THTingbo Hou
Tingbo Hou
This study introduces Text-Guided Subject-Driven Image Inpainting, a novel task that combines text and exemplar images for image inpainting. While both text and exemplar images have been used independently in previous efforts, their combined utilization remains unexplored. Simultaneously accommodating both conditions poses a significant challenge due to the inherent balance required between editability and subject fidelity. To tackle this challenge, we propose a two-step approach DreamInpainter. First, we compute dense subject features to ensure accurate subject replication. Then, we employ a discriminative token selection module to eliminate redundant subject details, preserving the subject's identity while allowing changes according to other conditions such as mask shape and text prompts. Additionally, we introduce a decoupling regularization technique to enhance text control in the presence of exemplar images. Our extensive experiments demonstrate the superior performance of our method in terms of visual quality, identity preservation, and text control, showcasing its effectiveness in the context of text-guided subject-driven image inpainting.

9 September 2026
Verified
Book Chapter
Closed access
PDF file
PZPeiyuan Zhu
Peiyuan Zhu
SXShaoan Xie
Shaoan Xie
ZLZijian Li
Zijian Li
YSYifan Shen
Yifan Shen
NDNamrata Deka
Namrata Deka
KZKun Zhang
Kun Zhang

14 September 2026
Verified
Book Chapter
Closed access
PDF file
LKLingjing Kong
Lingjing Kong
SXShaoan Xie
Shaoan Xie
GCGuangyi Chen
Guangyi Chen
YSYuewen Sun
Yuewen Sun
XSXiangchen Song
Xiangchen Song
KZKun Zhang
Kun Zhang

10 October 2023 in arXiv
Verified
Preprint
Open access
PDF file
YXYanwu Xu
Yanwu Xu
MGMingming Gong
Mingming Gong
SXShaoan Xie
Shaoan Xie
WWWei Wei
Wei Wei
MGMatthias Grundmann
Matthias Grundmann
THTingbo Hou
Tingbo Hou
Despite the proliferation of generative models, achieving fast sampling during inference without compromising sample diversity and quality remains challenging. Existing models such as Denoising Diffusion Probabilistic Models (DDPM) deliver high-quality, diverse samples but are slowed by an inherently high number of iterative steps. The Denoising Diffusion Generative Adversarial Networks (DDGAN) attempted to circumvent this limitation by integrating a GAN model for larger jumps in the diffusion process. However, DDGAN encountered scalability limitations when applied to large datasets. To address these limitations, we introduce a novel approach that tackles the problem by matching implicit and explicit factors. More specifically, our approach involves utilizing an implicit model to match the marginal distributions of noisy data and the explicit conditional distribution of the forward diffusion. This combination allows us to effectively match the joint denoising distributions. Unlike DDPM but similar to DDGAN, we do not enforce a parametric distribution for the reverse step, enabling us to take large steps during inference. Similar to the DDPM but unlike DDGAN, we take advantage of the exact form of the diffusion process. We demonstrate that our proposed method obtains comparable generative performance to diffusion-based models and vastly superior results to models with a small number of sampling steps.