Fu-Yun Wang

Fu-Yun Wang

Auto-generated profile, based on publicly available information

Fu-Yun Wang

MMLab CUHK Hong Kong

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
20 October 2022
Verified
Book Chapter
Closed access
PDF file
FWFu-Yun Wang
Fu-Yun Wang
DZDa-Wei Zhou
Da-Wei Zhou
HYHan-Jia Ye
Han-Jia Ye
DZDe-Chuan Zhan
De-Chuan Zhan

13 July 2024
Verified
Conference Paper
Open access
XSXiaoyu Shi
Xiaoyu Shi
ZHZhaoyang Huang
Zhaoyang Huang
FWFu-Yun Wang
Fu-Yun Wang
WBWeikang Bian
Weikang Bian
DLDasong Li
Dasong Li
HLHongsheng Li
Hongsheng Li
We introduce Motion-I2V, a novel framework for consistent and controllable text-guided image-to-video generation (I2V). In contrast to previous methods that directly learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into two stages with explicit motion modeling. For the first stage, we propose a diffusion-based motion field predictor, which focuses on deducing the trajectories of the reference image’s pixels. For the second stage, we propose motion-augmented temporal attention to enhance the limited 1-D temporal attention in video latent diffusion models. This module can effectively propagate reference image features to synthesized frames with the guidance of predicted trajectories from the first stage. Compared with existing methods, Motion-I2V can generate more consistent videos even in the presence of large motion and viewpoint variation. By training a sparse trajectory ControlNet for the first stage, Motion-I2V can support users to precisely control motion trajectories and motion regions with sparse trajectory and region. This offers more controllability of the I2V process than solely relying on textual instructions. Additionally, Motion-I2V’s second stage naturally supports zero-shot video-to-video translation. Both qualitative and quantitative comparisons demonstrate the advantages of Motion-I2V over prior approaches in consistent and controllable image-to-video generation. Please see our project page at https://xiaoyushi97.github.io/Motion-I2V/.

30 September 2024
Verified
Book Chapter
Closed access
PDF file
FWFu-Yun Wang
Fu-Yun Wang
XWXiaoshi Wu
Xiaoshi Wu
ZHZhaoyang Huang
Zhaoyang Huang
XSXiaoyu Shi
Xiaoyu Shi
DSDazhong Shen
Dazhong Shen
HLHongsheng Li
Hongsheng Li

19 November 2024
Verified
Conference Paper
Open access
FWFu-Yun Wang
Fu-Yun Wang
ZHZhaoyang Huang
Zhaoyang Huang
WBWeikang Bian
Weikang Bian
XSXiaoyu Shi
Xiaoyu Shi
KSKeqiang Sun
Keqiang Sun
HLHongsheng Li
Hongsheng Li

24 November 2024
Verified
Book Chapter
Closed access
PDF file
FWFu-Yun Wang
Fu-Yun Wang
ZHZhaoyang Huang
Zhaoyang Huang
QMQiang Ma
Qiang Ma
GSGuanglu Song
Guanglu Song
XLXudong Lu
Xudong Lu
HLHongsheng Li
Hongsheng Li

16 May 2025 in arXiv
Verified
Preprint
Open access
PDF file
FWFu-Yun Wang
Fu-Yun Wang
YSYunhao Shui
Yunhao Shui
JPJingtan Piao
Jingtan Piao
KSKeqiang Sun
Keqiang Sun
HLHongsheng Li
Hongsheng Li
Diffusion models have made substantial advances in image generation, yet models trained on large, unfiltered datasets often yield outputs misaligned with human preferences. Numerous methods have been proposed to fine-tune pre-trained diffusion models, achieving notable improvements in aligning generated outputs with human preferences. However, we argue that existing preference alignment methods neglect the critical role of handling unconditional/negative-conditional outputs, leading to a diminished capacity to avoid generating undesirable outcomes. This oversight limits the efficacy of classifier-free guidance~(CFG), which relies on the contrast between conditional generation and unconditional/negative-conditional generation to optimize output quality. In response, we propose a straightforward but versatile effective approach that involves training a model specifically attuned to negative preferences. This method does not require new training strategies or datasets but rather involves minor modifications to existing techniques. Our approach integrates seamlessly with models such as SD1.5, SDXL, video diffusion models and models that have undergone preference optimization, consistently enhancing their alignment with human preferences.