We implement high-quality sketch colorization using two-stage conditional generative adversarial network (GAN) training based on different intermediate features. The intermediate features used in autonomous colorization are the grayscale parsing and interval pixel-level color parsing. The autonomous colorization based on grayscale parsing feature can learn the spacial topology of pixels in the first stage to guide the colorization in the second stage. The autonomous colorization based on pixel-level color parsing feature can learn the color information of few feature points in the first stage to guide the colorization of all pixels in the second stage. Additionally, we use the intermediate feature of sampling points as constraint and achieve the color reconstruction using Laplacian mesh editing as a special second stage. Furthermore, the interactive colorization uses the superpixel color parsing as the intermediate feature. Specifically, we use the simple linear iterative cluster (SLIC) to obtain a palette that maintains the edges in the first stage to guide the colorization in the second stage. As for evaluation metrics, we propose a colorcoded local binary pattern (CCLBP) score based on color distances from the first-order 8 pixels to the central pixel, to measure the degrees of color blurring and mess. We also propose a light-sensitivity (LS) score based on the reversed grayscale map, to measure the degrees of auto painting and overfitting of the color hint. According to the L1 distances between the original and generated color images based on these scores, compared with state-of-the-art methods including one stage approaches such as pix2pix and PaintsChainer and two-stage approaches such as Style2Paints and DeepColor, our model can achieve the highest-quality autonomous colorization. Moreover, compared with pix2pix, PaintsChainer and Style2Paints with color hints, according to the proposed objective evaluation as well as the user visual study, our model can achieve the highest-quality interactive colorization as well.

This paper presents the CASIA-GO entry to the Generation and Evaluation of Non-verbal Behaviour for Embedded Agents (GENEA) Challenge 2023. The system is originally designed for few-shot scenarios such as generating gestures with the style of any in-the-wild target speaker from short speech samples. Given a group of reference speech data including gesture sequences, audio, and text, it first constructs a gesture motion graph that describes the soft gesture units and interframe continuity inside the speech, which is ready to be used for new rhythmic and semantic gesture reenactment by pathfinding when test audio and text are provided. We randomly choose one clip from the training data for one test clip to simulate a few-shot scenario and provide compatible results for subjective evaluations. Despite the 0.25% average utilization of the whole training set for each clip in the test set and the 17.5% total utilization of the training set for the whole test set, the system succeeds in providing valid results and ranks in the top 1/3 in the appropriateness for agent speech evaluation.

Existing sketch‐based image processing methods include sketch recognition, sketch synthesis and sketch‐based image retrieval. For sketch creation, a meaningful task is proposed namely disentangled and controllable sketch creation (DCSC) based on disentangling the structure and color enhancement. Specifically, as the first subtask, sketch structure enhancement (SSE) is used to enhance a non‐professional sketch (NPS) and obtain a professional sketch (PS), which is a process denoted as NPS2PS. A data set named SketchMan is first provided, consisting of NPSs and PSs with various postures in different scenes. SSE is trained as a conditional image‐to‐image translation problem, and there are three models: direct sketch‐to‐sketch (SS), grayscale guided SS and contour guided SS. Multiple IOU metrics are proposed based on Corner Point Map (CPM), Straight Line Map (SLM) and Segmented Area Map (SAM). As the second subtask, sketch color enhancement (SCE) is trained as a two‐stage framework containing a topology enhancement network (TE‐Net) that maps a sketch to the corresponding grayscale domain and a color injection network (CI‐Net) that injects the global color feature to the AdaIN residual blocks to perform adaptive sketch colorization. The TE‐Net and CI‐Net disentangle the topological and color features to perform more controllable and diverse SCE results. Experimental results demonstrate that our proposed methods are effective to address the challenging and meaningful DCSC task compared with other state‐of‐the‐art methods.