22 January 2021 in IEEE Transactions on Image Processing
Deep learning-based methods have achieved remarkable success in image restoration and enhancement, but are they still competitive when there is a lack of paired training data? As one such example, this paper explores the low-light image enhancement problem, where in practice it is extremely challenging to simultaneously take a low-light and a normal-light photo of the same visual scene. We propose a highly effective unsupervised generative adversarial network, dubbed EnlightenGAN, that can be trained without low/normal-light image pairs, yet proves to generalize very well on various real-world test images. Instead of supervising the learning using ground truth data, we propose to regularize the unpaired training using the information extracted from the input itself, and benchmark a series of innovations for the low-light image enhancement problem, including a global-local discriminator structure, a self-regularized perceptual loss fusion, and the attention mechanism. Through extensive experiments, our proposed approach outperforms recent methods under a variety of metrics in terms of visual quality and subjective user study. Thanks to the great flexibility brought by unpaired training, EnlightenGAN is demonstrated to be easily adaptable to enhancing real-world images from various domains. Our codes and pre-trained models are available at: https://github.com/VITA-Group/EnlightenGAN.
1 December 2023 in IEEE Transactions on Pattern Analysis and Machine Intelligence
This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS), with high performance, low cost, and in-depth interpretation. NAS has been explosively studied to automate the discovery of top-performer neural networks, but suffers from heavy resource consumption and often incurs search bias due to truncated training or approximations. Recent NAS works Mellor et al. 2021, Chen et al. 2021, Abdelfattah et al. 2021 start to explore indicators that can predict a network's performance without training. However, they either leveraged limited properties of deep networks, or the benefits of their training-free indicators were not applied to more extensive search methods. By rigorous correlation analysis, we present a unified framework to understand and accelerate NAS, by disentangling “ TEG ” characteristics of searched networks – T rainability, E xpressivity, G eneralization – all assessed in a training-free manner. The TEG indicators could be scaled up and integrated with various NAS search methods, including both supernet and single-path NAS approaches. Extensive studies validate the effective and efficient guidance from our TEG-NAS framework, leading to both improved search accuracy and over 56% reduction in search time cost. Moreover, we visualize search trajectories on three landscapes of “TEG” characteristics, observing that a good local minimum is easier to find on NAS-Bench-201 given its simple topology, whereas balancing “TEG” characteristics is much harder on the DARTS space due to its complex landscape geometry.
6 May 2022 in CAAI Transactions on Intelligence Technology
Evaluation of calligraphic copy is the core of Chinese calligraphy appreciation and inheritance. However, previous aesthetic evaluation studies often focussed on photos and paintings, with few attempts on Chinese calligraphy. To solve this problem, a Siamese regression aesthetic fusion method is proposed, named SRAFE, for Chinese calligraphy based on the combination of calligraphy aesthetics and deep learning. First, a dataset termed Evaluated Chinese Calligraphy Copies (E3C) is constructed for aesthetic evaluation. Second, 12 hand‐crafted aesthetic features based on the shape, structure, and stroke of calligraphy are designed. Then, the Siamese regression network (SRN) is designed to extract the deep aesthetic representation of calligraphy. Finally, the SRAFE method is built by fusing the deep aesthetic features with the hand‐crafted aesthetic features. Experimental results show that scores given by SRAFE are similar to the aesthetic evaluation label of E3C, proving the effectiveness of the authors’ method.