Tianyu He

Tianyu He

Auto-generated profile, based on publicly available information

Tianyu He

Microsoft Research Asia, Beijing, China

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
11 January 2019 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Closed access
ZCZhibo Chen
Zhibo Chen
THTianyu He
Tianyu He
XJXin Jin
Xin Jin
FWFeng Wu
Feng Wu

26 October 2023
Verified
Conference Paper
Open access
CDChenpeng Du
Chenpeng Du
QCQi Chen
Qi Chen
THTianyu He
Tianyu He
XTXu Tan
Xu Tan
XCXie Chen
Xie Chen
JBJiang Bian
Jiang Bian
While recent research has made significant progress in speech-driven talking face generation, the quality of the generated video still lags behind that of real recordings. One reason for this is the use of handcrafted intermediate representations like facial landmarks and 3DMM coefficients, which are designed based on human knowledge and are insufficient to precisely describe facial movements. Additionally, these methods require an external pretrained model for extracting these representations, whose performance sets an upper bound on talking face generation. To address these limitations, we propose a novel method called DAE-Talker that leverages data-driven latent representations obtained from a diffusion autoencoder (DAE). DAE contains an image encoder that encodes an image into a latent vector and a DDIM-based image decoder that reconstructs the image from it. We train our DAE on talking face video frames and then extract their latent representations as the training target for a Conformer-based speech2latent model. During inference, DAE-Talker first predicts the latents from speech and then generates the video frames with the image decoder in DAE from the predicted latents. This allows DAE-Talker to synthesize full video frames and produce natural head movements that align with the content of speech, rather than relying on a predetermined head pose from a template video. We also introduce pose modelling in speech2latent for pose controllability. Additionally, we propose a novel method for generating continuous video frames with the DDIM-based image decoder trained on individual frames, eliminating the need for modelling the joint distribution of consecutive frames directly. Our experiments show that DAE-Talker outperforms existing popular methods in lip-sync, video fidelity, and pose naturalness. We also conduct ablation studies to analyze the effectiveness of the proposed techniques and demonstrate the pose controllability of DAE-Talker.

27 October 2025
Verified
Conference Paper
Open access
JBJianhong Bai
Jianhong Bai
THTianyu He
Tianyu He
YWYuchi Wang
Yuchi Wang
JGJunliang Guo
Junliang Guo
Hao HuHao Hu
Hao Hu
JBJiang Bian
Jiang Bian

4 June 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
ATAnni Tang
Anni Tang
THTianyu He
Tianyu He
XTXu Tan
Xu Tan
JLJun Ling
Jun Ling
RLRunnan Li
Runnan Li
LSLi Song
Li Song
Talking face generation aims at generating photo-realistic video portraits of a target person driven by input audio. According to the nature of audio to lip motions mapping, the same speech content may have different appearances even for the same person at different occasions. Such one-to-many mapping problem brings ambiguity during training and thus causes inferior visual results. Although this one-to-many mapping could be alleviated in part by a two-stage framework (i.e., an audio-to-expression model followed by a neural-rendering model), it is still insufficient since the prediction is produced without enough information (e.g., emotions, wrinkles, etc.). In this paper, we propose MemFace to complement the missing information with an implicit memory and an explicit memory that follow the sense of the two stages respectively. More specifically, the implicit memory is employed in the audio-to-expression model to capture high-level semantics in the audio-expression shared space, while the explicit memory is employed in the neural-rendering model to help synthesize pixel-level details. Our experimental results show that our proposed MemFace surpasses all the state-of-the-art results across multiple scenarios consistently and significantly.

12 October 2023 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
ZWZhengning Wu
Zhengning Wu
THTianyu He
Tianyu He
XXXiaobo Xia
Xiaobo Xia
YJYu Jun
Yu Jun
XSXu Shen
Xu Shen
TLTongliang Liu
Tongliang Liu
Consistency regularization has achieved great successes in Semi-Supervised Single-Label Image Classification (SS-SLC) with deep learning models, while few effort has been devoted to Semi-Supervised Multi-Label Image Classification (SS-MLC) with deep learning models. One intuitive solution for introducing consistency regularization to SS-MLC is to regularize model predictions to be invariant to different augmented data of the same input image. However, the solution lacks the consideration of label relations, which are key elements in multi-label image classification. In this paper, we go beyond the consistency regularization for multi-view input images, and propose Conditional Consistency Regularization (CCR) that is tailored for SS-MLC. Specifically, for two augmented input images, we make the two model predictions conditioned on different label states (i.e., positive, negative, or unknown for each class). By encouraging the two predictions to be consistent, the model is able to build relations between the given two different label states, which helps to make use of label relations for boosting image classification. The experiments on large-scale real-world SS-MLC benchmarks demonstrate that the proposed method can surpass state-of-the-art methods by a large margin.

9 January 2019 in arXiv
Verified
Preprint
Open access
PDF file
ZCZhibo Chen
Zhibo Chen
THTianyu He
Tianyu He
XJXin Jin
Xin Jin
FWFeng Wu
Feng Wu
One key challenge to learning-based video compression is that motion predictive coding, a very effective tool for video compression, can hardly be trained into a neural network. In this paper we propose the concept of PixelMotionCNN (PMCNN) which includes motion extension and hybrid prediction networks. PMCNN can model spatiotemporal coherence to effectively perform predictive coding inside the learning network. On the basis of PMCNN, we further explore a learning-based framework for video compression with additional components of iterative analysis/synthesis, binarization, etc. Experimental results demonstrate the effectiveness of the proposed scheme. Although entropy coding and complex configurations are not employed in this paper, we still demonstrate superior performance compared with MPEG-2 and achieve comparable results with H.264 codec. The proposed learning-based scheme provides a possible new direction to further improve compression efficiency and functionalities of future video coding.