18 December 2018 in IEEE Transactions on Visualization and Computer Graphics
In this paper, we propose the first sketching system for interactively personalized and photorealistic face caricaturing. Input an image of a human face, the users can create caricature photos by manipulating its facial feature curves. Our system firstly performs exaggeration on the recovered 3D face model, which is conducted by assigning the laplacian of each vertex a scaling factor according to the edited sketches. The mapping between 2D sketches and the vertex-wise scaling field is constructed by a novel deep learning architecture. Our approach allows outputting different exaggerations when applying the same sketching on different input figures in term of their different geometric characteristics, which makes the generated results "personalized". With the obtained 3D caricature model, two images are generated, one obtained by applying 2D warping guided by the underlying 3D mesh deformation and the other obtained by re-rendering the deformed 3D textured model. These two images are then seamlessly integrated to produce our final output. Due to the severe stretching of meshes, the rendered texture is of blurry appearances. A deep learning approach is exploited to infer the missing details for enhancing these blurry regions. Moreover, a relighting operation is invented to further improve the photorealism of the result. These further make our results "photorealistic". The qualitative experiment results validated the efficiency of our sketching system.
13 October 2020 in IEEE Transactions on Visualization and Computer Graphics
In the game and film industries, modeling 3D heads plays a very important role in designing characters. Although human head modeling has been researched for a long time, few works have focused on animal-like heads, which are of more diverse shapes and richer geometric details. In this work, we present SAniHead, an interactive system for creating animal-like heads with a mesh representation from dual-view sketches. Our core technical contribution is a view-surface collaborative mesh generative network. Initially, a Graph Convolutional Neural Network(GCNN) is trained to learn the deformation of a template mesh to fit the shape of sketches, giving rise to a coarse model. It is then projected into vertex maps where image-to-image translation networks are performed for detail inference. After back-projecting the inferred details onto the meshed surface, a new GCNN is trained for further detail refinement. The modules of view-based detail inference and surface-based detail refinement are conducted in an alternating cascaded fashion, collaboratively improving the model. A refinement sketching interface is also implemented to support direct mesh manipulation. Experimental results show the superiority of our approach and the usability of our interactive system. Our work also contributes a 3D animal head dataset with corresponding line drawings.
19 July 2023 in IEEE Transactions on Visualization and Computer Graphics
Modeling 3D avatars benefits various application scenarios such as AR/VR, gaming, and filming. Character faces contribute significant diversity and vividity as a vital component of avatars. However, building 3D character face models usually requires a heavy workload with commercial tools, even for experienced artists. Various existing sketch-based tools fail to support amateurs in modeling diverse facial shapes and rich geometric details. In this article, we present SketchMetaFace - a sketching system targeting amateur users to model high-fidelity 3D faces in minutes. We carefully design both the user interface and the underlying algorithm. First, curvature-aware strokes are adopted to better support the controllability of carving facial details. Second, considering the key problem of mapping a 2D sketch map to a 3D model, we develop a novel learning-based method termed “Implicit and Depth Guided Mesh Modeling” (IDGMM). It fuses the advantages of mesh, implicit, and depth representations to achieve high-quality results with high efficiency. In addition, to further support usability, we present a coarse-to-fine 2D sketching interface design and a data-driven stroke suggestion tool. User studies demonstrate the superiority of our system over existing modeling tools in terms of the ease to use and visual quality of results. Experimental analyses also show that IDGMM reaches a better trade-off between accuracy and efficiency.
10 August 2023 in Computer Graphics Forum
16 December 2020 in Computer Graphics Forum
4 April 2026 in SSRN Electronic Journal
Traditional computer-aided design (CAD) is highly specialized and time-consuming, posing a significant barrier for users without 3D modeling expertise. Even with the recent surge of AI-based methods, CAD modeling remains challenging due to three fundamental issues: 1) the lack of simple yet controllable interaction modalities; 2) the absence of 3D representations that are both amenable to learning paradigms and faithful to CAD characteristics; and 3) the scarcity of large-scale datasets consisting of inconsistent modeling sequences. To address these challenges, we propose SelfSketch2CAD, a framework that leverages geometric priors for self-supervised CAD modeling sequence learning from single-view freehand sketches. Our method leverages predefined differentiable CAD modeling operators, combined with an implicit signed distance function (SDF) representation, to automatically infer CAD modeling sequences from 2D sketches, and composes primitive-level SDFs via the union operation into a holistic SDF field, enabling supervision solely from target shapes without requiring ground-truth modeling sequences. To mitigate the inherent ambiguity of sparse line drawings, we further introduce a multifaceted sketch perception module, which uses a U-Net to predict depth and normal maps from the input sketch, applies a pre-trained DINO encoder to extract features from both the sketch and its inferred geometric cues, and finally fuses these features through a multi-stage cross-attention mechanism to guide subsequent CAD operator prediction. Extensive experiments on the ABC dataset show that our approach consistently outperforms existing methods in both quantitative metrics and qualitative visual fidelity, demonstrating its effectiveness for sketch-based CAD content creation. The source code will be released for research use.