Over the last years, many face analysis tasks have accomplished astounding performance, with applications including face generation and 3D face reconstruction from a single ‘`in-the-wild’' image. Nevertheless, to the best of our knowledge, there is no method which can produce render-ready high-resolution 3D faces from ‘`in-the-wild’' images and this can be attributed to the: (a) scarcity of available data for training, and (b) lack of robust methodologies that can successfully be applied on very high-resolution data. In this work, we introduce the first method that is able to reconstruct photorealistic render-ready 3D facial geometry and BRDF from a single ‘`in-the-wild’' image. We capture a large dataset of facial shape and reflectance, which we have made public. We define a fast facial photorealistic differentiable rendering methodology with accurate facial skin diffuse and specular reflection, self-occlusion and subsurface scattering approximation. With this, we train a network that disentangles the facial diffuse and specular BRDF components from a shape and texture with baked illumination, reconstructed with a state-of-the-art 3DMM fitting method. Our method outperforms the existing arts by a significant margin and reconstructs high-resolution 3D faces from a single low-resolution image, that can be rendered in various applications, and bridge the uncanny valley.
We present two novel mobile reflectometry approaches for acquiring detailed spatially varying isotropic surface reflectance and mesostructure of a planar material sample using commodity mobile devices. The first approach relies on the integrated camera and flash pair present on typical mobile devices to support free‐form handheld acquisition of spatially varying rough specular material samples. The second approach, suited for highly specular samples, uses the LCD panel to illuminate the sample with polarized second‐order gradient illumination. To address the limited overlap of the front facing camera's view and the LCD illumination (and thus limited sample size), we propose a novel appearance transfer method that combines controlled reflectance measurement of a small exemplar section with uncontrolled reflectance measurements of the full sample under natural lighting. Finally, we introduce a novel surface detail enhancement method that adds fine scale surface mesostructure from close‐up observations under uncontrolled natural lighting. We demonstrate the accuracy and versatility of the proposed mobile reflectometry methods on a wide variety of spatially varying materials.
We present a novel computational photography technique for single-shot separation of diffuse/specular reflectance, as well as novel angular domain separation of layered reflectance. We present two imaging solutions for this purpose: two-way polarized light-field (TPLF) imaging and four-way polarized light-field (FPLF) imaging. TPLF imaging consists of a polarized light-field camera, which simultaneously captures two orthogonal states of polarization. A single photograph of a subject acquired with the TPLF camera under polarized illumination then enables standard separation of diffuse (depolarizing) and polarization preserving specular reflectance using light-field sampling. We further demonstrate that the acquired data also enable novel angular separation of layered reflectance including separation of specular reflectance and single scattering in the polarization preserving component, as well as separation of shallow scattering from deep scattering in the depolarizing component. FPLF imaging further generalized the functionality of TPLF imaging under uncontrolled unpolarized or partially polarized illumination such as outdoors. We apply our approach for efficient acquisition of facial reflectance including diffuse and specular normal maps and novel separation of photometric normals into layered reflectance normals for layered facial renderings. We validate our proposed single-shot layered reflectance separation under various imaging conditions and demonstrate it to be comparable to an existing multi-shot technique that relies on structured lighting while achieving separation results under a variety of illumination conditions.