We present an AI-mediated 3D video conferencing system that can reconstruct and autostereoscopically display a life-sized talking head using consumer-grade compute resources and minimal capture equipment. Our 3D capture uses a novel 3D lifting method that encodes a given 2D input into an efficient triplanar neural representation of the user, which can be rendered from novel viewpoints in real-time. Our AI-based techniques drastically reduce the cost for 3D capture, while providing a high-fidelity 3D representation on the receiver’s end at the cost of traditional 2D video streaming. Additional advantages of our AI-based approach include the ability to accommodate both photorealistic and stylized avatars, and the ability to enable mutual eye contact in multi-directional video conferencing. We demonstrate our system using a tracked stereo display for a personal viewing experience as well as a lightfield display for a room-scale multi-viewer experience.
By manipulating light as a wavefront, holographic displays have the potential to revolutionize virtual reality (VR) and augmented reality (AR) systems. These displays support 3D focus cues for visual comfort, vision correcting capabilities, and high light efficiency. However, despite their incredible promise, holographic displays have consistently been hampered by poor image quality. Recently, artificial intelligence–driven computer-generated holography (CGH) algorithms have emerged as a solution to this obstacle. On a prototype holographic display, we demonstrate how the progress of recent state-of-the-art Neural Holography algorithms can produce high-quality dynamic 3D holograms with accurate focus cues. The advances demonstrated in this work aim to provide a glimpse into a future where our displays can fully reproduce three-dimensional virtual content.
We introduce a flexible computer-generated holography (CGH) framework with fast, MEMS-based, heavily-quantized SLMs. Our framework supports various types of content, from 2D images to 4D light-fields. We demonstrate state-of-the-art results with each content type.