Takaaki Shiratori

Takaaki Shiratori

Auto-generated profile, based on publicly available information

Takaaki Shiratori

Codec Avatars Lab, Meta

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
17 November 2024
Verified
Book Chapter
Closed access
PDF file
GMGyeongsik Moon
Gyeongsik Moon
TSTakaaki Shiratori
Takaaki Shiratori
SSShunsuke Saito
Shunsuke Saito

26 July 2023 in ACM Transactions on Graphics
Verified
Journal Article
Open access
KPKunkun Pang
Kunkun Pang
DQDafei Qin
Dafei Qin
YFYingruo Fan
Yingruo Fan
JHJulian Habekost
Julian Habekost
TSTakaaki Shiratori
Takaaki Shiratori
TKTaku Komura
Taku Komura
Automatic gesture synthesis from speech is a topic that has attracted researchers for applications in remote communication, video games and Metaverse. Learning the mapping between speech and 3D full-body gestures is difficult due to the stochastic nature of the problem and the lack of a rich cross-modal dataset that is needed for training. In this paper, we propose a novel transformer-based framework for automatic 3D body gesture synthesis from speech. To learn the stochastic nature of the body gesture during speech, we propose a variational transformer to effectively model a probabilistic distribution over gestures, which can produce diverse gestures during inference. Furthermore, we introduce a mode positional embedding layer to capture the different motion speeds in different speaking modes. To cope with the scarcity of data, we design an intra-modal pre-training scheme that can learn the complex mapping between the speech and the 3D gesture from a limited amount of data. Our system is trained with either the Trinity speech-gesture dataset or the Talking With Hands 16.2M dataset. The results show that our system can produce more realistic, appropriate, and diverse body gestures compared to existing state-of-the-art approaches.

19 December 2025 in ACM Transactions on Graphics
Verified
Journal Article
Open access
JLJintao Lu
Jintao Lu
HZHe Zhang
He Zhang
YYYuting Ye
Yuting Ye
TSTakaaki Shiratori
Takaaki Shiratori
SSSebastian Starke
Sebastian Starke
TKTaku Komura
Taku Komura
Animating human-scene interactions such as picking and placing a wide range of objects with different geometries is a challenging task, especially in a cluttered environment where interactions with complex articulated containers are involved. The main difficulty lies in the sparsity of the motion data compared to the wide variation of the objects and environments, as well as the poor availability of transition motions between different actions, increasing the complexity of the generalization to arbitrary conditions. To cope with this issue, we develop a system that tackles the interaction synthesis problem as a hierarchical goal-driven task. Firstly, we develop a bimanual scheduler that plans a set of keyframes for simultaneously controlling the two hands to efficiently achieve the pick-and-place task from an abstract goal signal such as the target object selected by the user. Next, we develop a neural implicit planner that generates hand trajectories to guide reaching and leaving motions across diverse object shapes/types and obstacle layouts. Finally, we propose a linear dynamic model for our DeepPhase controller that incorporates a Kalman filter to enable smooth transitions in the frequency domain, resulting in a more realistic and effective multi-objective control of the character. Our system can synthesize a rich variety of natural pick-and-place movements that adapt to different object geometries, container articulations, and scene layouts.

21 November 2024
Verified
Book Chapter
Closed access
PDF file
JPJoonkyu Park
Joonkyu Park
GMGyeongsik Moon
Gyeongsik Moon
WXWeipeng Xu
Weipeng Xu
EKEvan Kaseman
Evan Kaseman
TSTakaaki Shiratori
Takaaki Shiratori
KLKyoung Mu Lee
Kyoung Mu Lee

31 July 2024 in arXiv
Verified
Preprint
Open access
PDF file
GMGyeongsik Moon
Gyeongsik Moon
TSTakaaki Shiratori
Takaaki Shiratori
SSShunsuke Saito
Shunsuke Saito
Facial expression and hand motions are necessary to express our emotions and interact with the world. Nevertheless, most of the 3D human avatars modeled from a casually captured video only support body motions without facial expressions and hand motions.In this work, we present ExAvatar, an expressive whole-body 3D human avatar learned from a short monocular video. We design ExAvatar as a combination of the whole-body parametric mesh model (SMPL-X) and 3D Gaussian Splatting (3DGS). The main challenges are 1) a limited diversity of facial expressions and poses in the video and 2) the absence of 3D observations, such as 3D scans and RGBD images. The limited diversity in the video makes animations with novel facial expressions and poses non-trivial. In addition, the absence of 3D observations could cause significant ambiguity in human parts that are not observed in the video, which can result in noticeable artifacts under novel motions. To address them, we introduce our hybrid representation of the mesh and 3D Gaussians. Our hybrid representation treats each 3D Gaussian as a vertex on the surface with pre-defined connectivity information (i.e., triangle faces) between them following the mesh topology of SMPL-X. It makes our ExAvatar animatable with novel facial expressions by driven by the facial expression space of SMPL-X. In addition, by using connectivity-based regularizers, we significantly reduce artifacts in novel facial expressions and poses.

26 October 2023 in arXiv
Verified
Preprint
Open access
PDF file
GMGyeongsik Moon
Gyeongsik Moon
SSShunsuke Saito
Shunsuke Saito
WXWeipeng Xu
Weipeng Xu
RJRohan Joshi
Rohan Joshi
JBJulia Buffalini
Julia Buffalini
TSTakaaki Shiratori
Takaaki Shiratori
The two-hand interaction is one of the most challenging signals to analyze due to the self-similarity, complicated articulations, and occlusions of hands. Although several datasets have been proposed for the two-hand interaction analysis, all of them do not achieve 1) diverse and realistic image appearances and 2) diverse and large-scale groundtruth (GT) 3D poses at the same time. In this work, we propose Re:InterHand, a dataset of relighted 3D interacting hands that achieve the two goals. To this end, we employ a state-of-the-art hand relighting network with our accurately tracked two-hand 3D poses. We compare our Re:InterHand with existing 3D interacting hands datasets and show the benefit of it. Our Re:InterHand is available in https://mks0601.github.io/ReInterHand/.