19 November 2024 in ACM Transactions on Graphics
We tackle the problem of highly-accurate, holistic performance capture for the face, body and hands simultaneously. Motion-capture technologies used in film and game production typically focus only on face, body or hand capture independently, involve complex and expensive hardware and a high degree of manual intervention from skilled operators. While machine-learning-based approaches exist to overcome these problems, they usually only support a single camera, often operate on a single part of the body, do not produce precise world-space results, and rarely generalize outside specific contexts. In this work, we introduce the first technique for markerfree, high-quality reconstruction of the complete human body, including eyes and tongue, without requiring any calibration, manual intervention or custom hardware. Our approach produces stable world-space results from arbitrary camera rigs as well as supporting varied capture environments and clothing. We achieve this through a hybrid approach that leverages machine learning models trained exclusively on synthetic data and powerful parametric models of human shape and motion. We evaluate our method on a number of body, face and hand reconstruction benchmarks and demonstrate state-of-the-art results that generalize on diverse datasets.
16 August 2023 in Proceedings of the ACM on Computer Graphics and Interactive Techniques
Virtual reality (VR) allows us to immerse ourselves in alternative worlds in which we can embody avatars to take on new identities. Usually, these avatars are humanoid or possess very strong anthropomorphic qualities. Allowing users of VR to embody non-humanoid virtual characters or animals presents additional challenges. Extreme morphological differences and the complexities of different characters' motions can make the construction of a real-time mapping between input human motion and target character motion a difficult challenge. Previous animal embodiment work has focused on direct mapping of human motion to the target animal via inverse kinematics. This can lead to the target animal moving in a way which is inappropriate or unnatural for the animal type. We present a novel real-time method, incorporating two neural networks, for mapping human motion to realistic quadruped motion. Crucially, the output quadruped motions are realistic, while also being faithful to the input user motions. We incorporate our mapping into a VR embodiment system in which users can embody a virtual quadruped from a first person perspective. Further, we evaluate our system via a perceptual experiment in which we investigate the quality of the synthesised motion, the system's response to user input and the sense of embodiment experienced by users. The main findings of the study are that the system responds as well as traditional embodiment systems to user input, produces higher quality motion and users experience a higher sense of body ownership when compared to a baseline method in which the human to quadruped motion mapping relies solely on inverse kinematics. Finally, our embodiment system relies solely on consumer-grade hardware, thus making it appropriate for use in applications such as VR gaming or VR social platforms.