Soshi Shimada

Soshi Shimada

Auto-generated profile, based on publicly available information

Soshi Shimada

Max Planck Institute for Informatics, Saarland Informatics Campus

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
2 November 2021 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
JMJAMEEL MALIK
JAMEEL MALIK
SSSoshi Shimada
Soshi Shimada
AEAhmed Elhayek
Ahmed Elhayek
SASk Aziz Ali
Sk Aziz Ali
CTChristian Theobalt
Christian Theobalt
DSDidier Stricker
Didier Stricker
3D hand shape and pose estimation from a single depth map is a new and challenging computer vision problem with many applications. Existing methods addressing it directly regress hand meshes via 2D convolutional neural networks, which leads to artifacts due to perspective distortions in the images. To address the limitations of the existing methods, we develop HandVoxNet++, i.e., a voxel-based deep network with 3D and graph convolutions trained in a fully supervised manner. The input to our network is a 3D voxelized-depth-map-based on the truncated signed distance function (TSDF). HandVoxNet++ relies on two hand shape representations. The first one is the 3D voxelized grid of hand shape, which does not preserve the mesh topology and which is the most accurate representation. The second representation is the hand surface that preserves the mesh topology. We combine the advantages of both representations by aligning the hand surface to the voxelized hand shape either with a new neural Graph-Convolutions-based Mesh Registration (GCN-MeshReg) or classical segment-wise Non-Rigid Gravitational Approach (NRGA++) which does not rely on training data. In extensive evaluations on three public benchmarks, i.e., SynHand5M, depth-based HANDS19 challenge and HO-3D, the proposed HandVoxNet++ achieves the state-of-the-art performance. In this journal extension of our previous approach presented at CVPR 2020, we gain $41.09\%$ and $13.7\%$ higher shape alignment accuracy on SynHand5M and HANDS19 datasets, respectively. Our method is ranked first on the HANDS19 challenge dataset ( Task 1: Depth-Based 3D Hand Pose Estimation ) at the moment of the submission of our results to the portal in August 2020.

23 October 2022
Verified
Book Chapter
Closed access
PDF file
SSSoshi Shimada
Soshi Shimada
VGVladislav Golyanik
Vladislav Golyanik
ZLZhi Li
Zhi Li
PPPatrick Pérez
Patrick Pérez
WXWeipeng Xu
Weipeng Xu
CTChristian Theobalt
Christian Theobalt

13 October 2023 in arXiv
Verified
Preprint
Open access
PDF file
SSSoshi Shimada
Soshi Shimada
VGVladislav Golyanik
Vladislav Golyanik
PPPatrick Pérez
Patrick Pérez
CTChristian Theobalt
Christian Theobalt
Existing methods for 3D tracking from monocular RGB videos predominantly consider articulated and rigid objects. Modelling dense non-rigid object deformations in this setting remained largely unaddressed so far, although such effects can improve the realism of the downstream applications such as AR/VR and avatar communications. This is due to the severe ill-posedness of the monocular view setting and the associated challenges. While it is possible to naively track multiple non-rigid objects independently using 3D templates or parametric 3D models, such an approach would suffer from multiple artefacts in the resulting 3D estimates such as depth ambiguity, unnatural intra-object collisions and missing or implausible deformations. Hence, this paper introduces the first method that addresses the fundamental challenges depicted above and that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system. As a pivotal step in its creation, we process the reconstructed raw 3D shapes with position-based dynamics and an approach for non-uniform stiffness estimation of the head tissues, which results in plausible annotations of the surface deformations, hand-face contact regions and head-hand positions. At the core of our neural approach are a variational auto-encoder supplying the hand-face depth prior and modules that guide the 3D tracking by estimating the contacts and the deformations. Our final 3D hand and face reconstructions are realistic and more plausible compared to several baselines applicable in our setting, both quantitatively and qualitatively. https://vcai.mpi-inf.mpg.de/projects/Decaf