17 August 2021 in IEEE Transactions on Image Processing
Automatic sketch colorization is a challenging task in both computer graphics and computer vision since all the color, texture, shading generation have to be created based on the abstract sketch. Besides, it is a subjective task in painting process, which needs illustrators to comprehend drawing priori (DP), such as hue variation, saturation contrast and gray contrast and utilize them in the HSV color space which is closer to human visual cognition system. As such, incorporating supplementary supervision in the HSV color space may be beneficial to sketch colorization. However, previous methods improve the colorization quality only in the RGB color space without considering the HSV color space, often causing results with dull color, inappropriate saturation contrast, and artifacts. To address this issue, we propose a novel sketch colorization method, dual color space guided generative adversarial network (DCSGAN), that considers the complementary information contained in both the RGB and HSV color space. Specifically, we incorporate the HSV color space to construct dual color spaces for supervising our method with a color space transformation (CST) network that learns transformation from the RGB to HSV color space. Then, we propose a DP loss that enables the DCSGAN to generate vivid color images with pixel level supervision. Additionally, a novel dual color space adversarial (DCSA) loss is designed to guide the generator at global level to reduce the artifacts to meet audiences’ aesthetic expectations. Extensive experiments and ablation studies demonstrate the superiority of the proposed method over previous state-of-the-art (SOTA) methods.
31 October 2023 in IEEE Transactions on Image Processing
Automatic sketch colorization is a challenging task that aims to generate a color image from a sketch, primarily due to its inherently ill-posed nature. While many approaches have shown promising results, two significant challenges remain: limited color patterns and a wide range of artifacts such as color bleeding and semantic inconsistencies among relevant regions. These issues stem from the operation of traditional convolutional structures, which capture structural features in a pixel-wise manner, resulting in inadequate utilization of regional information within the sketch. Therefore, we propose the Region-Assisted Sketch Coloring (RASC) method, which introduces an intermediate representation called the ’Region Map’ to explicitly characterize the regional information of the sketch. This Region Map is derived from the input sketch and is effectively formulated by our RASC architecture, enhancing the perception of region-wise features beyond the original pixel-wise features. Specifically, we start by employing the sketch encoder to extract hierarchical feature maps from the input sketches. Subsequently, we introduce a coarse-to-fine decoder comprising a series of Region-based Modulation (RM) blocks. This decoder modulates features that combine the modulation results of its previous block and the sketch features of the corresponding encoder block with our Region Formulation module. Each module explicitly formulates the sketch features in a region-wise manner. This accurately captures both the inner-region local style and inter-region global context dependency, resulting in various color patterns and fewer synthesis artifacts. Our experimental results show that our proposed method surpasses state-of-the-art methods in both synthetic and real sketch datasets.
26 July 2024 in Multimedia Systems
16 June 2021 in IEEE MultiMedia
In our daily life, we prefer to describe the location of a person with semantic words such as “travel,” “cinema” rather than geographic terminology such as GPS coordinates ($32^{\circ }24^{^{\prime }}12^{^{\prime \prime }}$32∘24'12''N, $2^{\circ }90^{^{\prime }}26.5^{^{\prime \prime }}$2∘90'26.5''E). This is because semantic places are easier to understand compared with precise geographic location. Existing methods only take text-image pairs into consideration, neglecting the user attribute, which is essential in semantic place prediction. In this work, we explore the impact of user attribute and propose user attribute included multimodal fusion models for semantic place prediction. Specifically, due to the complementarity of different modalities, multimodal fusion networks can adaptively learn the relationship between different modalities to fully use representational information from each modality. Besides, we propose a multi-image processing module to solve the scenario that a piece of Microblog text is attached to multiple images. Extensive experiments on our collected dataset verified the effectiveness of the proposed approach.
28 September 2024 in Pattern Recognition