23 November 2022 in IEEE Transactions on Multimedia
Existing image-to-image translation approaches can deal with simple scenes or styles effectively, such as summer-to-winter, horses-to-zebra, and photo-to-map. Although a great progress has been made by GAN-based methods recently, the performance of night-to-day (N2D) translation remains unsatisfactory due to imbalanced/poor visibility, and thus leading to translation ambiguity. To improve the quality of N2D translation, we propose an unpaired translation scheme based on a semantic prior generator, namely SPN2D-GAN, in a weakly-supervised manner with consideration of both image and semantic information. Specifically, we design a novel N2D generator, which can adopt the semantic information of images as prior knowledge to generate more reasonable and realistic results. Also, we suggest adjusting the brightness of nighttime images to boost the visibility, so that the generator can better extract content information. Moreover, the proposed SPN2D-GAN translates images by enforcing the distribution of daytime images in both image and semantic domains on final outputs. Besides, the cycle consistency is employed to preserve the fidelity between translations from two directions. Extensive experimental results are provided to reveal the effectiveness of our design, and demonstrate its superior performance over other state-of-the-art N2D translation approaches both quantitatively and qualitatively. Our code is available at https://github.com/brightknight177/SPN2DGAN .
2 December 2022 in IEEE Transactions on Multimedia
Deep neural networks have made significant progress in various tasks under the assumption of the same distribution between training and testing data. However, the obtained domain-specific knowledge often suffers from performance degradation when facing out-of-distribution data. Towards addressing the degradation, a critical requirement of such networks is the generalization capability to unseen domains, which is the goal of domain generalization (DG). This paper attempts to learn generalized knowledge from a single synthetic domain and then apply it to real and unknown scenarios. Specifically, we propose a contour-aware instance normalization module to effectively learn domain-invariant features via a novel weight-updating strategy, which can largely exploit the generalized information from the observed data. In addition, a category-level contrastive learning mechanism is proposed through understanding the semantic discrepancy and relevance among samples to mitigate the interference of domain-specific features on classification. Extensive experiments together with ablation studies on widely-adopted datasets are conducted to demonstrate the effectiveness of our design and show the superiority of our method over other state-of-the-art schemes on the task of urban-scene segmentation. The source code is available at https://github.com/leelxh/DG-LGK .