Xintao Wang

Xintao Wang

Auto-generated profile, based on publicly available information

Xintao Wang

Kuaishou Technology

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 October 2024
Verified
Book Chapter
Closed access
PDF file
JXJinbo Xing
Jinbo Xing
MXMenghan Xia
Menghan Xia
YZYong Zhang
Yong Zhang
HCHaoxin Chen
Haoxin Chen
WYWangbo Yu
Wangbo Yu
TWTien-Tsin Wong
Tien-Tsin Wong

13 July 2024
Verified
Conference Paper
Open access
ZWZhouxia Wang
Zhouxia Wang
ZYZiyang Yuan
Ziyang Yuan
XWXintao Wang
Xintao Wang
YLYaowei Li
Yaowei Li
TCTianshui Chen
Tianshui Chen
YSYing Shan
Ying Shan
Motions in a video primarily consist of camera motion, induced by camera movement, and object motion, resulting from object movement. Accurate control of both camera and object motion is essential for video generation. However, existing works either mainly focus on one type of motion or do not clearly distinguish between the two, limiting their control capabilities and diversity. Therefore, this paper presents MotionCtrl, a unified and flexible motion controller for video generation designed to effectively and independently control camera and object motion. The architecture and training strategy of MotionCtrl are carefully devised, taking into account the inherent properties of camera motion, object motion, and imperfect training data. Compared to previous methods, MotionCtrl offers three main advantages: 1) It effectively and independently controls camera motion and object motion, enabling more fine-grained motion control and facilitating flexible and diverse combinations of both types of motion. 2) Its motion conditions are determined by camera poses and trajectories, which are appearance-free and minimally impact the appearance or shape of objects in generated videos. 3) It is a relatively generalizable model that can adapt to a wide array of camera poses and trajectories once trained. Extensive qualitative and quantitative experiments have been conducted to demonstrate the superiority of MotionCtrl over existing methods. Project page: https://wzhouxiff.github.io/projects/MotionCtrl/ .

14 February 2024 in IEEE Transactions on Visualization and Computer Graphics
Verified
Journal Article
Open access
JXJinbo Xing
Jinbo Xing
MXMenghan Xia
Menghan Xia
YLYuxin Liu
Yuxin Liu
YZYuechen Zhang
Yuechen Zhang
YZYong Zhang
Yong Zhang
TWTien-Tsin Wong
Tien-Tsin Wong
Creating a vivid video from the event or scenario in our imagination is a truly fascinating experience. Recent advancements in text-to-video synthesis have unveiled the potential to achieve this with prompts only. While text is convenient in conveying the overall scene context, it may be insufficient to control precisely. In this paper, we explore customized video generation by utilizing text as context description and motion structure (e.g., frame-wise depth) as concrete guidance. Our method, dubbed Make-Your-Video, involves joint-conditional video generation using a Latent Diffusion Model that is pre-trained for still image synthesis and then promoted for video generation with the introduction of temporal modules. This two-stage learning scheme not only reduces the computing resources required, but also improves the performance by transferring the rich concepts available in image datasets solely into video generation. Moreover, we use a simple yet effective causal attention mask strategy to enable longer video synthesis, which mitigates the potential quality degradation effectively. Experimental results show the superiority of our method over existing baselines, particularly in terms of temporal coherence and fidelity to users’ guidance. In addition, our model enables several intriguing applications that demonstrate potential for practical usage.

3 November 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
XCXiangyu Chen
Xiangyu Chen
XWXintao Wang
Xintao Wang
WZWenlong Zhang
Wenlong Zhang
XKXiangtao Kong
Xiangtao Kong
YQYu Qiao
Yu Qiao
CDChao Dong
Chao Dong
Transformer-based methods have shown impressive performance in image restoration tasks, such as image super-resolution and denoising. However, we find that these networks can only utilize a limited spatial range of input information through attribution analysis. This implies that the potential of Transformer is still not fully exploited in existing networks. In order to activate more input pixels for better restoration, we propose a new Hybrid Attention Transformer (HAT). It combines both channel attention and window-based self-attention schemes, thus making use of their complementary advantages. Moreover, to better aggregate the cross-window information, we introduce an overlapping cross-attention module to enhance the interaction between neighboring window features. In the training stage, we additionally adopt a same-task pre-training strategy to further exploit the potential of the model for further improvement. Extensive experiments have demonstrated the effectiveness of the proposed modules. We further scale up the model to show that the performance of the SR task can be greatly improved. Besides, we extend HAT to more image restoration applications, including real-world image super-resolution, Gaussian image denoising and image compression artifacts reduction. Experiments on benchmark and real-world datasets demonstrate that our HAT achieves state-of-the-art performance both quantitatively and qualitatively.

19 November 2024 in ACM Transactions on Graphics
Verified
Journal Article
Open access
JXJinbo Xing
Jinbo Xing
HLHanyuan Liu
Hanyuan Liu
MXMenghan Xia
Menghan Xia
YZYong Zhang
Yong Zhang
XWXintao Wang
Xintao Wang
TWTien-Tsin Wong
Tien-Tsin Wong
We introduce ToonCrafter, a novel approach that transcends traditional correspondence-based cartoon video interpolation, paving the way for generative interpolation. Traditional methods, that implicitly assume linear motion and the absence of complicated phenomena like dis-occlusion, often struggle with the exaggerated non-linear and large motions with occlusion commonly found in cartoons, resulting in implausible or even failed interpolation results. To overcome these limitations, we explore the potential of adapting live-action video priors to better suit cartoon interpolation within a generative framework. ToonCrafter effectively addresses the challenges faced when applying live-action video motion priors to generative cartoon interpolation. First, we design a toon rectification learning strategy that seamlessly adapts live-action video priors to the cartoon domain, resolving the domain gap and content leakage issues. Next, we introduce a dual-reference-based 3D decoder to compensate for lost details due to the highly compressed latent prior spaces, ensuring the preservation of fine details in interpolation results. Finally, we design a flexible sketch encoder that empowers users with interactive control over the interpolation results. Experimental results demonstrate that our proposed method not only produces visually convincing and more natural dynamics, but also effectively handles dis-occlusion. The comparative evaluation demonstrates the notable superiority of our approach over existing competitors. Code and model weights are available at https://doubiiu.github.io/projects/ToonCrafter

14 December 2025
Verified
Conference Paper
Open access
JYJiwen Yu
Jiwen Yu
JBJianhong Bai
Jianhong Bai
YQYiran Qin
Yiran Qin
QLQuande Liu
Quande Liu
XWXintao Wang
Xintao Wang
XLXihui Liu
Xihui Liu