Xiaodong Cun

Xiaodong Cun

Auto-generated profile, based on publicly available information

Xiaodong Cun

Great Bay University, Guangdong, China

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
28 February 2020 in IEEE Transactions on Image Processing
Verified
Journal Article
Closed access
XCXiaodong Cun
Xiaodong Cun
Chi-Man PunChi-Man Pun
Chi-Man Pun
Image composition is one of the most important applications in image processing. However, the inharmonious appearance between the spliced region and background degrade the quality of the image. Thus, we address the problem of Image Harmonization: Given a spliced image and the mask of the spliced region, we try to harmonize the “style” of the pasted region with the background (non-spliced region). Previous approaches have been focusing on learning directly by the neural network. In this work, we start from an empirical observation: the differences can only be found in the spliced region between the spliced image and the harmonized result while they share the same semantic information and the appearance in the nonspliced region. Thus, in order to learn the feature map in the masked region and the others individually, we propose a novel attention module named Spatial-Separated Attention Module (S2AM). Furthermore, we design a novel image harmonization framework by inserting the S2AM in the coarser low-level features of the Unet structure by two different ways. Besides image harmonization, we make a big step for harmonizing the composite image without the specific mask under previous observation. The experiments show that the proposed S2AM performs better than other state-of-the-art attention modules in our task. Moreover, we demonstrate the advantages of our model against other state-of-the-art image harmonization methods via criteria from multiple points of view.

14 February 2024 in IEEE Transactions on Visualization and Computer Graphics
Verified
Journal Article
Open access
JXJinbo Xing
Jinbo Xing
MXMenghan Xia
Menghan Xia
YLYuxin Liu
Yuxin Liu
YZYuechen Zhang
Yuechen Zhang
YZYong Zhang
Yong Zhang
TWTien-Tsin Wong
Tien-Tsin Wong
Creating a vivid video from the event or scenario in our imagination is a truly fascinating experience. Recent advancements in text-to-video synthesis have unveiled the potential to achieve this with prompts only. While text is convenient in conveying the overall scene context, it may be insufficient to control precisely. In this paper, we explore customized video generation by utilizing text as context description and motion structure (e.g., frame-wise depth) as concrete guidance. Our method, dubbed Make-Your-Video, involves joint-conditional video generation using a Latent Diffusion Model that is pre-trained for still image synthesis and then promoted for video generation with the introduction of temporal modules. This two-stage learning scheme not only reduces the computing resources required, but also improves the performance by transferring the rich concepts available in image datasets solely into video generation. Moreover, we use a simple yet effective causal attention mask strategy to enable longer video synthesis, which mitigates the potential quality degradation effectively. Experimental results show the superiority of our method over existing baselines, particularly in terms of temporal coherence and fidelity to users’ guidance. In addition, our model enables several intriguing applications that demonstrate potential for practical usage.

26 June 2023 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
WLWeihuang Liu
Weihuang Liu
XCXiaodong Cun
Xiaodong Cun
Chi-Man PunChi-Man Pun
Chi-Man Pun
MXMenghan Xia
Menghan Xia
YZYong Zhang
Yong Zhang
JWJue Wang
Jue Wang
Image inpainting aims to fill the missing hole of the input. It is hard to solve this task efficiently when facing high-resolution images due to two reasons: (1) Large reception field needs to be handled for high-resolution image inpainting. (2) The general encoder and decoder network synthesizes many background pixels synchronously due to the form of the image matrix. In this paper, we try to break the above limitations for the first time thanks to the recent development of continuous implicit representation. In detail, we down-sample and encode the degraded image to produce the spatial-adaptive parameters for each spatial patch via an attentional Fast Fourier Convolution (FFC)-based parameter generation network. Then, we take these parameters as the weights and biases of a series of multi-layer perceptron (MLP), where the input is the encoded continuous coordinates and the output is the synthesized color value. Thanks to the proposed structure, we only encode the high-resolution image in a relatively low resolution for larger reception field capturing. Then, the continuous position encoding will be helpful to synthesize the photo-realistic high-frequency textures by re-sampling the coordinate in a higher resolution. Also, our framework enables us to query the coordinates of missing pixels only in parallel, yielding a more efficient solution than the previous methods. Experiments show that the proposed method achieves real-time performance on the 2048X2048 images using a single GTX 2080 Ti GPU and can handle 4096X4096 images, with much better performance than existing state-of-the-art methods visually and numerically. The code is available at: https://github.com/NiFangBaAGe/CoordFill.

6 December 2024
Verified
Book Chapter
Closed access
PDF file
MNMuyao Niu
Muyao Niu
XCXiaodong Cun
Xiaodong Cun
XWXintao Wang
Xintao Wang
YZYong Zhang
Yong Zhang
YSYing Shan
Ying Shan
YZYinqiang Zheng
Yinqiang Zheng

10 December 2023
Verified
Conference Paper
Open access
YGYuan Gong
Yuan Gong
YPYouxin Pang
Youxin Pang
XCXiaodong Cun
Xiaodong Cun
MXMenghan Xia
Menghan Xia
YHYingqing He
Yingqing He
YYYujiu Yang
Yujiu Yang
Accurate Story visualization requires several necessary elements, such as identity consistency across frames, the alignment between plain text and visual content, and a reasonable layout of objects in images. Most previous works endeavor to meet these requirements by fitting a text-to-image (T2I) model on a set of videos in the same style and with the same characters, e.g., the FlintstonesSV dataset. However, the learned T2I models typically struggle to adapt to new characters, scenes, and styles, and often lack the flexibility to revise the layout of the synthesized images. This paper proposes a system for generic interactive story visualization, capable of handling multiple novel characters and supporting the editing of layout and local structure. It is developed by leveraging the prior knowledge of large language and T2I models, trained on massive corpora. The system comprises four interconnected components: story-to-prompt generation (S2P), text-to-layout generation (T2L), controllable text-to-image generation (C-T2I), and image-to-video animation (I2V). First, the S2P module converts concise story information into detailed prompts required for subsequent stages. Next, T2L generates diverse and reasonable layouts based on the prompts, offering users the ability to adjust and refine the layout to their preferences. The core component, C-T2I, enables the creation of images guided by layouts, sketches, and actor-specific identifiers to maintain consistency and detail across visualizations. Finally, I2V enriches the visualization process by animating the generated images. Extensive experiments and a user study are conducted to validate the effectiveness and flexibility of interactive editing of the proposed system.

25 October 2024
Verified
Book Chapter
Closed access
PDF file
LGLanqing Guo
Lanqing Guo
YHYingqing He
Yingqing He
HCHaoxin Chen
Haoxin Chen
MXMenghan Xia
Menghan Xia
XCXiaodong Cun
Xiaodong Cun
BWBihan Wen
Bihan Wen