Mohamed Elgharib

Mohamed Elgharib

Auto-generated profile, based on publicly available information

Mohamed Elgharib

Max Planck Institute for Informatics, Saarbrücken, Germany

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
19 November 2024 in ACM Transactions on Graphics
Verified
Journal Article
Open access
KTKartik Teotia
Kartik Teotia
HKHyeongwoo Kim
Hyeongwoo Kim
PGPablo Garrido
Pablo Garrido
MHMarc Habermann
Marc Habermann
MEMohamed Elgharib
Mohamed Elgharib
CTChristian Theobalt
Christian Theobalt
Real-time rendering of human head avatars is a cornerstone of many computer graphics applications, such as augmented reality, video games, and films, to name a few. Recent approaches address this challenge with computationally efficient geometry primitives in a carefully calibrated multi-view setup. Albeit producing photorealistic head renderings, they often fail to represent complex motion changes, such as the mouth interior and strongly varying head poses. We propose a new method to generate highly dynamic and deformable human head avatars from multi-view imagery in real time. At the core of our method is a hierarchical representation of head models that can capture the complex dynamics of facial expressions and head movements. First, with rich facial features extracted from raw input frames, we learn to deform the coarse facial geometry of the template mesh. We then initialize 3D Gaussians on the deformed surface and refine their positions in a fine step. We train this coarse-to-fine facial avatar model along with the head pose as learnable parameters in an end-to-end framework. This enables not only controllable facial animation via video inputs but also high-fidelity novel view synthesis of challenging facial expressions, such as tongue deformations and fine-grained teeth structure under large motion changes. Moreover, it encourages the learned head avatar to generalize towards new facial expressions and head poses at inference time. We demonstrate the performance of our method with comparisons against the related methods on different datasets, spanning challenging facial expression sequences across multiple identities. We also show the potential application of our approach by demonstrating a cross-identity facial performance transfer application. We make the code available on our project page.

9 April 2024 in ACM Transactions on Graphics
Verified
Journal Article
Open access
KTKartik Teotia
Kartik Teotia
MRMallikarjun B R
Mallikarjun B R
XPXingang Pan
Xingang Pan
HKHyeongwoo Kim
Hyeongwoo Kim
PGPablo Garrido
Pablo Garrido
CTChristian Theobalt
Christian Theobalt
Multi-view volumetric rendering techniques have recently shown great potential in modeling and synthesizing high-quality head avatars. A common approach to capture full head dynamic performances is to track the underlying geometry using a mesh-based template or 3D cube-based graphics primitives. While these model-based approaches achieve promising results, they often fail to learn complex geometric details such as the mouth interior, hair, and topological changes over time. This article presents a novel approach to building highly photorealistic digital head avatars. Our method learns a canonical space via an implicit function parameterized by a neural network. It leverages multiresolution hash encoding in the learned feature space, allowing for high quality, faster training, and high-resolution rendering. At test time, our method is driven by a monocular RGB video. Here, an image encoder extracts face-specific features that also condition the learnable canonical space. This encourages deformation-dependent texture variations during training. We also propose a novel optical flow-based loss that ensures correspondences in the learned canonical space, thus encouraging artifact-free and temporally consistent renderings. We show results on challenging facial expressions and show free-viewpoint renderings at interactive real-time rates for a resolution of 480 x 270. Our method outperforms related approaches both visually and numerically. We will release our multiple-identity dataset to encourage further research.

13 July 2024
Verified
Conference Paper
Open access
PRPramod Rao
Pramod Rao
GFGereon Fox
Gereon Fox
AMAbhimitra Meka
Abhimitra Meka
MRMallikarjun B R
Mallikarjun B R
FZFangneng Zhan
Fangneng Zhan
CTChristian Theobalt
Christian Theobalt
Achieving photorealistic 3D view synthesis and relighting of human portraits is pivotal for advancing AR/VR applications. Existing methodologies in portrait relighting demonstrate substantial limitations in terms of generalization and 3D consistency, coupled with inaccuracies in physically realistic lighting and identity preservation. Furthermore, personalization from a single view is difficult to achieve and often requires multiview images during the testing phase or involves slow optimization processes. This paper introduces Lite2Relight , a novel technique that can predict 3D consistent head poses of portraits while performing physically plausible light editing at interactive speed. Our method uniquely extends the generative capabilities and efficient volumetric representation of EG3D, leveraging a lightstage dataset to implicitly disentangle face reflectance and perform relighting under target HDRI environment maps. By utilizing a pre-trained geometry-aware encoder and a feature alignment module, we map input images into a relightable 3D space, enhancing them with a strong face geometry and reflectance prior. Through extensive quantitative and qualitative evaluations, we show that our method outperforms the state-of-the-art methods in terms of efficacy, photorealism, and practical application. This includes producing 3D-consistent results of the full head, including hair, eyes, and expressions. Lite2Relight paves the way for large-scale adoption of photorealistic portrait editing in various domains, offering a robust, interactive solution to a previously constrained problem.

14 December 2025
Verified
Conference Paper
Open access
PRPramod Rao
Pramod Rao
AMAbhimitra Meka
Abhimitra Meka
XZXilong Zhou
Xilong Zhou
GFGereon Fox
Gereon Fox
MRMallikarjun B R
Mallikarjun B R
CTChristian Theobalt
Christian Theobalt

31 October 2023 in International Journal of Computer Vision
Verified
Journal Article
Open access
PDF file
PRPramod Rao
Pramod Rao
BMB. R. Mallikarjun
B. R. Mallikarjun
GFGereon Fox
Gereon Fox
TWTim Weyrich
Tim Weyrich
BBBernd Bickel
Bernd Bickel
MEMohamed Elgharib
Mohamed Elgharib
Portrait viewpoint and illumination editing is an important problem with several applications in VR/AR, movies, and photography. Comprehensive knowledge of geometry and illumination is critical for obtaining photorealistic results. Current methods are unable to explicitly model in 3D while handling both viewpoint and illumination editing from a single image. In this paper, we propose VoRF, a novel approach that can take even a single portrait image as input and relight human heads under novel illuminations that can be viewed from arbitrary viewpoints. VoRF represents a human head as a continuous volumetric field and learns a prior model of human heads using a coordinate-based MLP with individual latent spaces for identity and illumination. The prior model is learned in an auto-decoder manner over a diverse class of head shapes and appearances, allowing VoRF to generalize to novel test identities from a single input image. Additionally, VoRF has a reflectance MLP that uses the intermediate features of the prior model for rendering One-Light-at-A-Time (OLAT) images under novel views. We synthesize novel illuminations by combining these OLAT images with target environment maps. Qualitative and quantitative evaluations demonstrate the effectiveness of VoRF for relighting and novel view synthesis, even when applied to unseen subjects under uncontrolled illumination. This work is an extension of Rao et al. (VoRF: Volumetric Relightable Faces 2022). We provide extensive evaluation and ablative studies of our model and also provide an application, where any face can be relighted using textual input.

2 June 2021 in arXiv
Verified
Preprint
Open access
PDF file
GFGereon Fox
Gereon Fox
WLWentao Liu
Wentao Liu
HKHyeongwoo Kim
Hyeongwoo Kim
HSHans-Peter Seidel
Hans-Peter Seidel
MEMohamed Elgharib
Mohamed Elgharib
CTChristian Theobalt
Christian Theobalt
There are concerns that new approaches to the synthesis of high quality face videos may be misused to manipulate videos with malicious intent. The research community therefore developed methods for the detection of modified footage and assembled benchmark datasets for this task. In this paper, we examine how the performance of forgery detectors depends on the presence of artefacts that the human eye can see. We introduce a new benchmark dataset for face video forgery detection, of unprecedented quality. It allows us to demonstrate that existing detection techniques have difficulties detecting fakes that reliably fool the human eye. We thus introduce a new family of detectors that examine combinations of spatial and temporal features and outperform existing approaches both in terms of detection accuracy and generalization.