Wei Cao

Wei Cao

Auto-generated profile, based on publicly available information

Wei Cao

University of Illinois Urbana-Champaign, USA

Research Focus & Metrics

Academic Career Length

1Year (2026-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
19 July 2026
Verified
Conference Paper
Open access
WCWei Cao
Wei Cao
HZHao Zhang
Hao Zhang
FTFengrui Tian
Fengrui Tian
YWYulun Wu
Yulun Wu
YLYingying Li
Yingying Li
YLYaoyao Liu
Yaoyao Liu
Camera redirection aims to replay a dynamic scene from a single monocular video under a user-specified camera trajectory. However, large-angle redirection is inherently ill-posed: a monocular video captures only a narrow spatio-temporal view of a dynamic 3D scene, providing severely limited observations of the underlying 4D world. The key challenge is therefore to recover a complete and coherent plenoptic representation from this limited input, with consistent geometry and coherent motion. While recent diffusion-based methods achieve impressive visual generation quality, they often break down under large-angle viewpoint changes far from the original trajectory, where missing visual grounding leads to severe geometric ambiguity and temporal inconsistency. To address this, we present FreeOrbit4D, an effective training-free framework that tackles this geometric ambiguity by recovering a foreground-complete 4D proxy as structural grounding for video generation. We obtain this proxy by decoupling foreground and background reconstructions: we unproject the monocular video into a static background and partial foreground point clouds in a unified global space, then leverage an object-centric multi-view diffusion model to synthesize multi-view images and reconstruct complete foreground point clouds in canonical object space. By aligning the canonical foreground point cloud to the global scene space via dense pixel-synchronized 3D–3D correspondences and projecting the foreground-complete 4D proxy onto target camera viewpoints, we provide geometric scaffolds (e.g., depth/visibility cues) that guide a conditional video diffusion model. Extensive experiments show that FreeOrbit4D produces more faithful and temporally coherent redirected videos under challenging large-angle trajectories, and our foreground-complete 4D proxy further opens a potential avenue for practical applications such as edit propagation and 4D data generation. Project page: https://freeorbit4d.vision.ischool.illinois.edu/

1 August 2026 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Open access
JTJiapeng Tang
Jiapeng Tang
WCWei Cao
Wei Cao
BZBiao Zhang
Biao Zhang
CLChang Luo
Chang Luo
YLYaoyao Liu
Yaoyao Liu
MNMatthias Nießner
Matthias Nießner
We introduce Motion2VecSets, a 4D diffusion model for dynamic surface mesh generation from various ambiguous observations, including a sequence of RGB images, sparse and partial point clouds, and low-resolution voxel grids. While recent methods using neural field representations have shown success in modeling non-rigid objects, conventional feed-forward architectures struggle with noisy, partial, or sparse observations due to their deterministic nature. To address the inherent one-to-many mapping problem, we introduce a diffusion model that explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of compressed latent representations. The diffusion-based priors provide more plausible and diverse reconstructions under ambiguous conditions. Instead of relying on global latent codes, we represent 4D dynamics using latent sets. This novel 4D representation captures local shape and deformation patterns, leading to more accurate non-linear motion capture and significantly improving generalization capacity to unseen motions and identities. For temporally coherent tracking, we jointly denoise latent sets across frames and enable cross-frame information exchange. To reduce computational cost, we design an interleaved spatial-temporal attention block that alternately aggregates deformation latents along spatial and temporal dimensions. Extensive experiments on datasets of humans, animals, and articulated objects demonstrate that Motion2VecSets outperforms prior methods in reconstructing and tracking non-rigid deformations from various imperfect observations. Our implementation is available at https://vveicao.github.io/projects/Motion2VecSets/.

15 September 2026
Verified
Book Chapter
Closed access
PDF file
YWYulun Wu
Yulun Wu
RZRuyi Zha
Ruyi Zha
WCWei Cao
Wei Cao
YLYingying Li
Yingying Li
YCYuanhao Cai
Yuanhao Cai
YLYaoyao Liu
Yaoyao Liu

19 October 2026 in Preprints.org
Verified
Preprint
Open access
PDF file
YSYifan Shen
Yifan Shen
ZZZhuoqing Zhong
Zhuoqing Zhong
XLXinzhuo Li
Xinzhuo Li
JLJiateng Liu
Jiateng Liu
BLBingxuan Li
Bingxuan Li
ILIsmini Lourentzou
Ismini Lourentzou
The rapid progress of large language models has established strong foundations for reasoning, knowledge, and generation, while the integration of vision, audio, video, and other modalities has extended these capabilities toward increasingly unified intelligence across modalities, which we refer to as Omni Intelligence. Yet, real-world assistants must do more than process completed multimodal prompts. They must perceive evolving streams, respond while new evidence is arriving, handle hesitation, overlap, interruption, and changing intent, and coordinate reasoning or tool use that may proceed asynchronously. We refer to this broader capability as \( \textbf{Interactive Omni Intelligence} \): the ability to maintain and exercise omni intelligence under the temporal and adaptive demands of real-time interaction. The question guiding this paper is: How can foundation-model capabilities be carried into real-time interaction while preserving both intelligence and interactivity?We develop a conceptual framework connecting two development paths. One adapts pretrained language or omni models through streaming interfaces, modality modules, post-training, and orchestration. The other develops model-native systems that learn temporal structure, continuous multimodal processing, generation, and interaction control. Across both paths, we compare representations, architectures, fusion, data, training objectives, serving, and evaluation. Their comparison reveals an Intelligence--Interactivity Trade-off between preserving reasoning and knowledge, and meeting the timing, coordination, and resource demands of continuous operation. We examine Multi-Rate Interactive Intelligence as a cross-cutting direction, including foreground--background designs that coordinate fast interaction with slower reasoning, retrieval, memory, and tools. Finally, we identify open problems in evolving state, persistent and proactive interaction, joint evaluation, and safe open-world operation. The resulting framework connects capability preservation, interaction control, and multi-rate coordination.