Xin Tong

Xin Tong

Auto-generated profile, based on publicly available information

Xin Tong

Microsoft Research Asia, Beijing, China

Research Focus & Metrics

Academic Career Length

25Years (2002-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
26 July 2023 in ACM Transactions on Graphics
Verified
Journal Article
Open access
XZXin-Yang Zheng
Xin-Yang Zheng
HPHao Pan
Hao Pan
PWPeng-Shuai Wang
Peng-Shuai Wang
XTXin Tong
Xin Tong
YLYang Liu
Yang Liu
HSHeung-Yeung Shum
Heung-Yeung Shum
Although the recent rapid evolution of 3D generative neural networks greatly improves 3D shape generation, it is still not convenient for ordinary users to create 3D shapes and control the local geometry of generated shapes. To address these challenges, we propose a diffusion-based 3D generation framework --- locally attentional SDF diffusion , to model plausible 3D shapes, via 2D sketch image input. Our method is built on a two-stage diffusion model. The first stage, named occupancy-diffusion , aims to generate a low-resolution occupancy field to approximate the shape shell. The second stage, named SDF-diffusion , synthesizes a high-resolution signed distance field within the occupied voxels determined by the first stage to extract fine geometry. Our model is empowered by a novel view-aware local attention mechanism for image-conditioned shape generation, which takes advantage of 2D image patch features to guide 3D voxel feature learning, greatly improving local controllability and model generalizability. Through extensive experiments in sketch-conditioned and category-conditioned 3D shape generation tasks, we validate and demonstrate the ability of our method to provide plausible and diverse 3D shapes, as well as its superior controllability and generalizability over existing work.

1 August 2022 in Computer Graphics Forum
Verified
Journal Article
Closed access
XZX. Zheng
X. Zheng
YLY. Liu
Y. Liu
PWP. Wang
P. Wang
XTX. Tong
X. Tong

12 September 2022 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
XGXiangjun Gao
Xiangjun Gao
JYJiaolong Yang
Jiaolong Yang
JKJongyoo Kim
Jongyoo Kim
SPSida Peng
Sida Peng
ZLZicheng Liu
Zicheng Liu
XTXin Tong
Xin Tong
There has been rapid progress recently on 3D human rendering, including novel view synthesis and pose animation, based on the advances of neural radiance fields (NeRF). However, most existing methods focus on person-specific training and their training typically requires multi-view videos. This article deals with a new challenging task – rendering novel views and novel poses for a person unseen in training, using only multiview still images as input without videos. For this task, we propose a simple yet surprisingly effective method to train a generalizable NeRF with multiview images as conditional input. The key ingredient is a dedicated representation combining a canonical NeRF and a volume deformation scheme. Using a canonical space enables our method to learn shared properties of human and easily generalize to different people. Volume deformation is used to connect the canonical space with input and target images and query image features for radiance and density prediction. We leverage the parametric 3D human model fitted on the input images to derive the deformation, which works quite well in practice when combined with our canonical NeRF. The experiments on both real and synthetic data with the novel view synthesis and pose animation tasks collectively demonstrate the efficacy of our method.

13 July 2024
Verified
Conference Paper
Open access
CZChong Zeng
Chong Zeng
YDYue Dong
Yue Dong
PPPieter Peers
Pieter Peers
YKYoukang Kong
Youkang Kong
HWHongzhi Wu
Hongzhi Wu
XTXin Tong
Xin Tong
This paper presents a novel method for exerting fine-grained lighting control during text-driven diffusion-based image generation. While existing diffusion models already have the ability to generate images under any lighting condition, without additional guidance these models tend to correlate image content and lighting. Moreover, text prompts lack the necessary expressional power to describe detailed lighting setups. To provide the content creator with fine-grained control over the lighting during image generation, we augment the text-prompt with detailed lighting information in the form of radiance hints, i.e., visualizations of the scene geometry with a homogeneous canonical material under the target lighting. However, the scene geometry needed to produce the radiance hints is unknown. Our key observation is that we only need to guide the diffusion process, hence exact radiance hints are not necessary; we only need to point the diffusion model in the right direction. Based on this observation, we introduce a three stage method for controlling the lighting during image generation. In the first stage, we leverage a standard pretrained diffusion model to generate a provisional image under uncontrolled lighting. Next, in the second stage, we resynthesize and refine the foreground object in the generated image by passing the target lighting to a refined diffusion model, named DiLightNet, using radiance hints computed on a coarse shape of the foreground object inferred from the provisional image. To retain the texture details, we multiply the radiance hints with a neural encoding of the provisional synthesized image before passing it to DiLightNet. Finally, in the third stage, we resynthesize the background to be consistent with the lighting on the foreground object. We demonstrate and validate our lighting controlled diffusion model on a variety of text prompts and lighting conditions.

13 December 2013 in International Journal of Computer Vision
Verified
Journal Article
Closed access
PDF file
CMChenguang Ma
Chenguang Ma
XCXun Cao
Xun Cao
XTXin Tong
Xin Tong
QDQionghai Dai
Qionghai Dai
SLStephen Lin
Stephen Lin

8 June 2021 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
JTJiapeng Tang
Jiapeng Tang
XHXiaoguang Han
Xiaoguang Han
MTMingkui Tan
Mingkui Tan
XTXin Tong
Xin Tong
KJKui Jia
Kui Jia
This paper focuses on the challenging task of learning 3D object surface reconstructions from RGB images. Existing methods achieve varying degrees of success by using different surface representations. However, they all have their own drawbacks, and cannot properly reconstruct the surface shapes of complex topologies, arguably due to a lack of constraints on the topological structures in their learning frameworks. To this end, we propose to learn and use the topology-preserved, skeletal shape representation to assist the downstream task of object surface reconstruction from RGB images. Technically, we propose the novel SkeletonNet design that learns a volumetric representation of a skeleton via a bridged learning of a skeletal point set, where we use parallel decoders each responsible for the learning of points on 1D skeletal curves and 2D skeletal sheets, as well as an efficient module of globally guided subvolume synthesis for a refined, high-resolution skeletal volume; we present a differentiable Point2Voxel layer to make SkeletonNet end-to-end and trainable. With the learned skeletal volumes, we propose two models, the Skeleton-Based Graph Convolutional Neural Network (SkeGCNN) and the Skeleton-Regularized Deep Implicit Surface Network (SkeDISN), which respectively build upon and improve over the existing frameworks of explicit mesh deformation and implicit field learning for the downstream surface reconstruction task. We conduct thorough experiments that verify the efficacy of our proposed SkeletonNet. SkeGCNN and SkeDISN outperform existing methods as well, and they have their own merits when measured by different metrics. Additional results in generalized task settings further demonstrate the usefulness of our proposed methods. We have made our implementation code publicly available at https://github.com/tangjiapeng/SkeletonNet .