Weikang Bian

Weikang Bian

Auto-generated profile, based on publicly available information

Weikang Bian

The Chinese University of Hong Kong, Hong Kong SAR, Hong Kong

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
13 July 2024
Verified
Conference Paper
Open access
XSXiaoyu Shi
Xiaoyu Shi
ZHZhaoyang Huang
Zhaoyang Huang
FWFu-Yun Wang
Fu-Yun Wang
WBWeikang Bian
Weikang Bian
DLDasong Li
Dasong Li
HLHongsheng Li
Hongsheng Li
We introduce Motion-I2V, a novel framework for consistent and controllable text-guided image-to-video generation (I2V). In contrast to previous methods that directly learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into two stages with explicit motion modeling. For the first stage, we propose a diffusion-based motion field predictor, which focuses on deducing the trajectories of the reference image’s pixels. For the second stage, we propose motion-augmented temporal attention to enhance the limited 1-D temporal attention in video latent diffusion models. This module can effectively propagate reference image features to synthesized frames with the guidance of predicted trajectories from the first stage. Compared with existing methods, Motion-I2V can generate more consistent videos even in the presence of large motion and viewpoint variation. By training a sparse trajectory ControlNet for the first stage, Motion-I2V can support users to precisely control motion trajectories and motion regions with sparse trajectory and region. This offers more controllability of the I2V process than solely relying on textual instructions. Additionally, Motion-I2V’s second stage naturally supports zero-shot video-to-video translation. Both qualitative and quantitative comparisons demonstrate the advantages of Motion-I2V over prior approaches in consistent and controllable image-to-video generation. Please see our project page at https://xiaoyushi97.github.io/Motion-I2V/.

19 November 2024
Verified
Conference Paper
Open access
FWFu-Yun Wang
Fu-Yun Wang
ZHZhaoyang Huang
Zhaoyang Huang
WBWeikang Bian
Weikang Bian
XSXiaoyu Shi
Xiaoyu Shi
KSKeqiang Sun
Keqiang Sun
HLHongsheng Li
Hongsheng Li

24 November 2024
Verified
Book Chapter
Closed access
PDF file
FWFu-Yun Wang
Fu-Yun Wang
ZHZhaoyang Huang
Zhaoyang Huang
QMQiang Ma
Qiang Ma
GSGuanglu Song
Guanglu Song
XLXudong Lu
Xudong Lu
HLHongsheng Li
Hongsheng Li