29 August 2023 in ACM Transactions on Graphics
This article presents object-oriented neural light transfer (NeLT), a novel neural representation of the dynamic light transportation between an object and the environment. Our method disentangles the global illumination of a scene into individual objects’ light transportation represented via neural networks, then composes them explicitly. It therefore enables flexible rendering with dynamic lighting, cameras, materials, and objects. Our rendering features various important global illumination effects, such as diffuse illumination, glossy illumination, dynamic shadowing, and indirect illumination, which completes the capability of existing neural object representation. Experiments show that NeLT does not require path tracing or shading results as input but achieves rendering quality comparable to state-of-the-art rendering frameworks, including the recent deep learning based denoisers.
27 July 2025 in ACM Transactions on Graphics
Recent advances in cloud rendering have brought us a promising alternative for interactive photorealistic rendering on lightweight devices, which used to be only available on high-end platforms equipped with powerful graphic cards. This technique enables users to perform rendering-related creative tasks, such as 3D product visualization and lighting design, from the comfort of any location using handheld devices, rather than being confined to the front of a noisy heat-generating workstation. However, existing large-scale cloud rendering systems that stream path-traced frames from the server to the client present extremely high rendering costs and transmission bandwidth requirements, even with advanced path-tracing acceleration and video compression techniques. To alleviate these problems, we propose a novel streaming-aware rendering framework that is able to learn a joint optimal model integrating two path-tracing acceleration techniques (adaptive sampling and denoising) and video compression technique. Our joint model can fully exploit the inherent connections between these techniques and thus achieve substantially reduced rendering costs and enhanced compression quality. We also introduce the collaboration of client rendering ability to assist the frame decoding by rendering G-buffers as the shared side information. We demonstrate that appropriately incorporating the geometry and material priors from G-buffers into a neural compression pipeline can significantly reduce the streaming bandwidth in a cloud rendering system, and lighten the compression module design for computation efficiency. Our experiments show that our method delivers the best quality at various bitrates compared to existing Monte Carlo rendering streaming schemes, while remaining lightweight and efficient for cross-platform thin clients, including mobiles and tablets.