Tao Liu

Tao Liu

Auto-generated profile, based on publicly available information

Tao Liu

X-LANCE Lab, MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
28 October 2024
Verified
Conference Paper
Open access
TLTao Liu
Tao Liu
FCFeilong Chen
Feilong Chen
SFShuai Fan
Shuai Fan
CDChenpeng Du
Chenpeng Du
QCQi Chen
Qi Chen
Kai YuKai Yu
Kai Yu
The paper introduces AniTalker, an innovative framework designed to generate lifelike talking faces from a single portrait. Unlike existing models that primarily focus on verbal cues such as lip synchronization and fail to capture the complex dynamics of facial expressions and nonverbal cues, AniTalker employs a universal motion representation. This innovative representation effectively captures a wide range of facial dynamics, including subtle expressions and head movements. AniTalker enhances motion depiction through two self-supervised learning strategies: the first involves reconstructing target video frames from source frames within the same identity to learn subtle motion representations, and the second develops an identity encoder using metric learning while actively minimizing mutual information between the identity and motion encoders. This approach ensures that the motion representation is dynamic and devoid of identity-specific details, significantly reducing the need for labeled data. Additionally, the integration of a diffusion model with a variance adapter allows for the generation of diverse and controllable facial animations. This method not only demonstrates AniTalker's capability to create detailed and realistic facial movements but also underscores its potential in crafting dynamic avatars for real-world applications. Synthetic results can be viewed at https://github.com/X-LANCE/AniTalker.

15 June 2022 in IEEE Access
Verified
Journal Article
Open access
NYNing Yang
Ning Yang
DLDe Feng Liu
De Feng Liu
TLTao Liu
Tao Liu
THTianyuan Han
Tianyuan Han
PZPingyue Zhang
Pingyue Zhang
JZJianguo Zhang
Jianguo Zhang
Parkinson’s disease (PD) is a nervous disorder that can cause motor impairment. PD severity assessment based on a series of motor movements illustrated in Unified Parkinson’s Disease Rating Scale (UPDRS) is an important part of clinical PD diagnosis. However, the current quantifying method heavily relies on human observation, which is time-consuming and subjective. Therefore, automatic severity estimation stemming from machine learning methods is receiving an increasing amount of research attention. However, these advances are still limited by data availability and interpretability. In this paper, we release a large PD motor dataset of over 300 real PD patients collected under doctors’ instructions and propose a pipeline to automatically quantify the motor severity of PD in finger tapping and postural stability. These two selected movements are representative of local and global motor control, exhibiting great clinical importance. The pipeline contains three-stage: pose estimation, domain knowledge extraction, and classification stage. The pose estimation uses deep-learning-based methods to extract 21 and 17 key points for finger tapping and postural stability respectively. The domain knowledge extraction stage extracts several explicit features pre-defined by experienced neuro-physicians. Finally, a classifier is trained to infer PD severity under MDS-UPDRS. To combine deep-learning-based features from pose estimation and domain features from the expert, the pipeline achieves a better trade-off between the model efficiency and clinical interpretability. Experiments show that our method achieves a micro average f1-score of 88%, 84%, and 84%, respectively on left finger tapping, right finger tapping, and postural stability, outperforming previous methods by a large margin. In addition, involving expert knowledge in the feature extraction stage greatly improves our model’s interpretability, which is essential in automatic PD detection.