Zhixin Shu

Zhixin Shu

Auto-generated profile, based on publicly available information

Zhixin Shu

Adobe Research

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
31 October 2022
Verified
Book Chapter
Closed access
PDF file
YLYuchen Liu
Yuchen Liu
ZSZhixin Shu
Zhixin Shu
YLYijun Li
Yijun Li
ZLZhe Lin
Zhe Lin
RZRichard Zhang
Richard Zhang
SKS. Y. Kung
S. Y. Kung

19 November 2024 in ACM Transactions on Graphics
Verified
Journal Article
Open access
JYJae Shin Yoon
Jae Shin Yoon
ZSZhixin Shu
Zhixin Shu
MRMengwei Ren
Mengwei Ren
CZCecilia Zhang
Cecilia Zhang
YHYannick Hold-Geoffroy
Yannick Hold-Geoffroy
HZHe Zhang
He Zhang
We introduce a high-fidelity portrait shadow removal model that can effectively enhance the image of a portrait by predicting its appearance under disturbing shadows and highlights. Portrait shadow removal is a highly ill-posed problem where multiple plausible solutions can be found based on a single image. For example, disentangling complex environmental lighting from original skin color is a non-trivial problem. While existing works have solved this problem by predicting the appearance residuals that can propagate local shadow distribution, such methods are often incomplete and lead to unnatural predictions, especially for portraits with hard shadows. We overcome the limitations of existing local propagation methods by formulating the removal problem as a generation task where a diffusion model learns to globally rebuild the human appearance from scratch as a condition of an input portrait image. For robust and natural shadow removal, we propose to train the diffusion model with a compositional repurposing framework: a pre-trained text-guided image generation model is first fine-tuned to harmonize the lighting and color of the foreground with a background scene by using a background harmonization dataset; and then the model is further fine-tuned to generate a shadow-free portrait image via a shadow-paired dataset. To overcome the limitation of losing fine details in the latent diffusion model, we propose a guided-upsampling network to restore the original high-frequency details (e.g. , wrinkles and dots) from the input image. To enable our compositional training framework, we construct a high-fidelity and large-scale dataset using a lightstage capturing system and synthetic graphics simulation. Our generative framework effectively removes shadows caused by both self and external occlusions while maintaining original lighting distribution and high-frequency details. Our method also demonstrates robustness to diverse subjects captured in real environments.

24 November 2024
Verified
Book Chapter
Closed access
PDF file
AHAndrew Hou
Andrew Hou
ZSZhixin Shu
Zhixin Shu
XZXuaner Zhang
Xuaner Zhang
HZHe Zhang
He Zhang
YHYannick Hold-Geoffroy
Yannick Hold-Geoffroy
XLXiaoming Liu
Xiaoming Liu

3 December 2024
Verified
Conference Paper
Open access
CHChengan He
Chengan He
JHJorge Alejandro Amador Herrera
Jorge Alejandro Amador Herrera
YZYi Zhou
Yi Zhou
ZSZhixin Shu
Zhixin Shu
XSXin Sun
Xin Sun
HRHOLLY RUSHMEIER
HOLLY RUSHMEIER
We introduce Digital Salon, a novel approach to 3D hair grooming and simulation by integrating advanced AI and physics-based algorithms. This tool enables users to create detailed hairstyles through natural language descriptions, seamlessly blending text-driven hair generation, interactive editing, and high-fidelity rendering within a cohesive workflow. With its innovative real-time simulation capabilities, Digital Salon supports dynamic hair interactions, accommodating 10,000 to 80,000 strands, thus making sophisticated hair design accessible to a wide range of users. This tool significantly enhances the creative process in digital media by providing an intuitive, versatile, and efficient solution for hair modeling and animation.

21 September 2023 in arXiv
Verified
Preprint
Open access
PDF file
SAShahRukh Athar
ShahRukh Athar
ZSZhixin Shu
Zhixin Shu
ZXZexiang Xu
Zexiang Xu
FLFujun Luan
Fujun Luan
SBSai Bi
Sai Bi
DSDimitris Samaras
Dimitris Samaras
Recent advances in Neural Radiance Fields (NeRFs) have made it possible to reconstruct and reanimate dynamic portrait scenes with control over head-pose, facial expressions and viewing direction. However, training such models assumes photometric consistency over the deformed region e.g. the face must be evenly lit as it deforms with changing head-pose and facial expression. Such photometric consistency across frames of a video is hard to maintain, even in studio environments, thus making the created reanimatable neural portraits prone to artifacts during reanimation. In this work, we propose CoDyNeRF, a system that enables the creation of fully controllable 3D portraits in real-world capture conditions. CoDyNeRF learns to approximate illumination dependent effects via a dynamic appearance model in the canonical space that is conditioned on predicted surface normals and the facial expressions and head-pose deformations. The surface normals prediction is guided using 3DMM normals that act as a coarse prior for the normals of the human head, where direct prediction of normals is hard due to rigid and non-rigid deformations induced by head-pose and facial expression changes. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls, and realistic lighting effects. The project page can be found here: http://shahrukhathar.github.io/2023/08/22/CoDyNeRF.html

4 July 2023 in arXiv
Verified
Preprint
Open access
PDF file
ZZZhen Zhu
Zhen Zhu
YLYijun Li
Yijun Li
WLWeijie Lyu
Weijie Lyu
KSKrishna Kumar Singh
Krishna Kumar Singh
ZSZhixin Shu
Zhixin Shu
DHDerek Hoiem
Derek Hoiem
We investigate how to generate multimodal image outputs, such as RGB, depth, and surface normals, with a single generative model. The challenge is to produce outputs that are realistic, and also consistent with each other. Our solution builds on the StyleGAN3 architecture, with a shared backbone and modality-specific branches in the last layers of the synthesis network, and we propose per-modality fidelity discriminators and a cross-modality consistency discriminator. In experiments on the Stanford2D3D dataset, we demonstrate realistic and consistent generation of RGB, depth, and normal images. We also show a training recipe to easily extend our pretrained model on a new domain, even with a few pairwise data. We further evaluate the use of synthetically generated RGB and depth pairs for training or fine-tuning depth estimators. Code will be available at https://github.com/jessemelpolio/MultimodalGAN.