With the popularity of virtual reality (VR) games and devices, demand is increasing for estimating and displaying user motion in VR applications. Most pose estimation methods for VR avatars exploit inverse kinematics (IK) and online motion capture methods. In contrast to existing approaches, we aim for a stable process with less computation, usable in a small space. Therefore, our strategy has minimum latency for VR device users, from high-performance to low-performance, in multi-user applications over the network. In this study, we estimate the upper body pose of a VR user in real time using a deep learning method. We propose a novel method inspired by a classical regression model and trained with 3D motion capture data. Thus, our design uses a convolutional neural network (CNN)-based architecture from the joint information of motion capture data and modifies the network input and output to obtain input from a head and both hands. After feeding the model with properly normalized inputs, a head-mounted display (HMD), and two controllers, we render the user’s corresponding avatar in VR applications. We used our proposed pose estimation method to build single-user and multi-user applications, measure their performance, conduct a user study, and compare the results with previous methods for VR avatars.

For embodied mixed reality(MR) experiences, it is crucial to accurately render the user’s full body in the virtual environment. Conventional MR systems provide sparse trackers such as a headset and two hand-held controllers. Recent studies have intensively investigated the learning methods to regress the untracked joints from the sparse trackers and produced plausible poses in real time for MR applications. However, most studies have assumed that they either know the position of the root joint or constrain it, yielding stiff pelvis motions. This paper presents the first geometry-incorporated learning method to generate the position and rotation of all joints, including the root joint, from the head and hands information for a wide range of motions. We split the problem into finding a reference frame and a pose inference with respect to a new reference frame. Our method defines an avatar frame by setting a non-joint as an origin and transforms joint data in a world coordinate system into the avatar coordinate system. Our learning builds on a propagating long short-term memory network exploiting prior knowledge of the kinematic chains and previous time domain. The learned joints are transformed back to obtain the positions with respect to the world frame. In our experiments, our method achieves competitive accuracy and robustness with the state-of-the-art speed of about 130 fps on motion capture datasets and the wild tracking data obtained from commercial MR devices. Our experiments confirm that the proposed method is practically applicable to MR systems.

Redirected walking (RDW) alters the relationship between physical locomotion and visual feedback to guide users along virtual paths that vary from their real-world trajectories. Over time these changes can lead to adaptation of the user and produce novel sensorimotor settings that users might retain across sessions and apply directly once they enter virtual reality (VR). Although short-term adaptation to altered sensorimotor contingencies is well established, it remains unclear whether such adaptation is retained across multiple days. We investigated long-term retention of RDW adaptation by repeatedly exposing ten participants to a fixed rightward curvature gain of $\pi$/30 across nine sessions over two weeks. Each session consisted of 200 walking repetitions with gain applied. Adaptation was assessed before and after each session using blind walking and a pointing task. Moreover, perceptual detection thresholds for curvature gains were measured before the first session (Baseline), a day after the ninth session (Final), and once in-between before the 5th adaptation session. Results showed clear retention of adapted locomotor behavior: during blind walking at the beginning of the sessions, when instructed to walk straight, participants consistently exhibited curved trajectories, which indicates that newly acquired sensorimotor contingencies can be retained over days and are immediately available upon re-entering VR. At the same time, pointing accuracy remained stable throughout the experiment and detection thresholds showed no consistent changes across sessions. In summary, our study provides evidence that adaptation to altered sensorimotor contingencies in VR can be retained across multiple sessions and days and can be available as soon as the user enters VR. This may be useful for many scenarios in which users repeatedly use VR tools over a long period of time. The complete data set, all supplemental materials and the preprint of the manuscript are available at https://osf.io/z973w.