We envision a telepresence system that enhances remote work by facilitating both physical and immersive visual interactions between individuals. However, during robot teleoperation, communication often lacks realism, as users see the robot’s body rather than the remote individual. To address this, we propose a method for overlaying a digital human model onto a humanoid robot using XR visualization, enabling an immersive 3D telepresence experience. Our approach employs a learning-based method to estimate the 2D poses of the humanoid robot from head-worn stereo views, leveraging a newly collected dataset of full-body poses for humanoid robots. The stereo 2D poses and sparse inertial measurements from the remote operator are optimized to compute 3D poses over time. The digital human is localized from the perspective of a continuously moving observer, utilizing the estimated 3D pose of the humanoid robot. Our moving camera-based pose estimation method does not rely on any markers or external knowledge of the robot’s status, effectively overcoming challenges such as marker occlusion, calibration issues, and dependencies on headset tracking errors. We demonstrate the system in a remote physical training scenario, achieving real-time performance at 40 fps, which enables simultaneous immersive and physical interactions. Experimental results show that our learning-based 3D pose estimation method, which operates without prior knowledge of the robot, significantly outperforms alternative approaches requiring the robot’s global pose, particularly during rapid headset movements, achieving markerless digital human augmentation from head-worn views.

We present a novel dataset for humanoid robot pose estimation from images, addressing the critical need for accurate pose estimation to enhance human–robot interaction in extended reality (XR) applications. Despite the importance of this task, large-scale pose datasets for diverse humanoid robots remain scarce. To overcome this limitation, we collected sparse pose datasets for commercially available humanoid robots and augmented them through various synthetic data generation techniques, including AI-assisted image synthesis, foreground removal, and 3D character simulations. Our dataset is the first to provide full-body pose annotations for a wide range of humanoid robots exhibiting diverse motions, including side and back movements, in real-world scenarios. Furthermore, we introduce a new benchmark method for real-time full-body 2D keypoint estimation from a single image. Extensive experiments demonstrate that our extended dataset-based pose estimation approach achieves over 33.9% improvement in accuracy compared to using only sparse datasets. Additionally, our method demonstrates the real-time capability of 42 frames per second (FPS) and maintains full-body pose estimation consistency in side and back motions across 11 differently shaped humanoid robots, utilizing approximately 350 training images per robot.

Expressive human mesh recovery reconstructs the body, hands, and face from a single image. Whole-body models such as SMPL-X offer only coarse control over the face, whereas face models such as FLAME are head-only. Combining them requires maintaining two models at inference and re-annotating training data. Complete eye closure and lip articulation are perceptually important yet difficult to represent, supervise, and evaluate. Existing formulations leave residual gaps or artifacts, and no public dataset, to our knowledge, characterizes eye and lip aperture. We present SMFLIX, a unified body–head model that merges both bases into a single linear block matrix formulation. It eliminates external registration, yields a watertight neck seam, and makes these articulations representable. Parameter compatibility lets us build SMFLIX-Set without full re-annotation, yielding 11M instances with joint whole-body and facial supervision. The dataset includes 536K face-valid test samples with continuous eye and lip aperture ratios, enabling direct evaluation of articulations obscured by mesh-averaged metrics. We train SMFLIX-Net, a one-stage dual-decoder network regressing body and facial parameters separately. Compared with state-of-the-art whole-body, face-only, and combined methods, the network achieves the lowest whole-body, body-part, and aperture errors on SMFLIX-Set. It remains competitive on external benchmarks, runs at 48 FPS on a consumer GPU, and supports a live webcam application.