Among the various generative adversarial network (GAN)-based image inpainting methods, a coarse-to-fine network with a contextual attention module (CAM) has shown remarkable performance. However, due to two stacked generative networks, the coarse-to-fine network needs numerous computational resources, such as convolution operations and network parameters, which result in low speed. To address this problem, we propose a novel network architecture called parallel extended-decoder path for semantic inpainting (PEPSI) network, which aims at reducing the hardware costs and improving the inpainting performance. PEPSI consists of a single shared encoding network and parallel decoding networks called coarse and inpainting paths. The coarse path produces a preliminary inpainting result to train the encoding network for the prediction of features for the CAM. Simultaneously, the inpainting path generates higher inpainting quality using the refined features reconstructed via the CAM. In addition, we propose Diet-PEPSI that significantly reduces the network parameters while maintaining the performance. In Diet-PEPSI, to capture the global contextual information with low hardware costs, we propose novel rate-adaptive dilated convolutional layers that employ the common weights but produce dynamic features depending on the given dilation rates. Extensive experiments comparing the performance with state-of-the-art image inpainting methods demonstrate that both PEPSI and Diet-PEPSI improve the qualitative scores, i.e., the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), as well as significantly reduce hardware costs, such as computational time and the number of network parameters.
21 August 2017 in IEEE Transactions on Consumer Electronics
In this paper, a novel content-preserving video stitching method is proposed. Video stitching has to deal with moving objects in the overlapped area (OA) which often cause problems of structural misalignment or ghost effect. To this end, the proposed method first finds an optimal seam that does not pass through the moving objects, by using the extended dynamic programing (DP) technique based on the energy minimization. Then the content-aware adaptive blending is performed which effectively reduces the color discontinuity while restricting the ghost effect caused by moving objects in the OA. In addition, to reduce the computational complexity, a partial seam-update (PSU) scheme is proposed, in which the seam is re-calculated only for a part of the seam passing through the moving object. Experimental results demonstrate that the proposed method is superior to the conventional ones in terms of both subjective quality and computational complexity. In addition, this method achieves real-time performance on mobile platform, making it applicable to consumer electronic devices.
1 November 2014 in IEEE Transactions on Consumer Electronics
A visible image-based face recognition system can be seriously degraded in real-life environments by various factors including illumination changes, expression changes, occlusion, and disguise. In this paper, a novel feature descriptor for robust face recognition, Eigen Directional Bit-Plane (EDBP), is introduced to address these issues. It is observed that Local Binary Pattern (LBP) can be decomposed into 8 directional bit-planes (DBP), each of which represents certain directional information of the facial image. Principal Component Analysis (PCA) is then applied to the DBP space to obtain a more compact feature, the EDBP. For face recognition, the proposed EDBP is integrated into conventional state-of-the-art classification methods. Simulation results demonstrate that classifiers with EDBP outperform those with existing feature descriptors under illumination changes, expression changes, occlusion, and disguise.
Lossy image compression methods with deep neural network (DNN) include a quantization process between encoder and decoder networks as an essential part to increase the compression rate. However, the quantization operation impedes the flow of gradient and often disturbs the optimal learning of the encoder, which results in distortion in the reconstructed images. To alleviate this problem, this paper presents a simple yet effective way that enhances the performance of lossy image compression without imposing training overhead or modifying the original network architectures. In the proposed method, we utilize an auxiliary branch called a shortcut which directly connects the encoder and decoder. Since the shortcut does not include the quantization process, it supports the optimal learning of the encoder by flowing the accurate gradient. Furthermore, to assist the decoder which should handle additional feature maps obtained via the shortcut, we also propose a residual refinement unit (RRU) following the quantizer. The experimental results show that the image compression network trained with the proposed method remarkably improves the performance in terms of peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and multi-scale structural similarity (MS-SSIM).
Histogram equalisation (HE) methods using the 2D histogram (2DH) have achieved a great success in contrast enhancement. The 2DH is constructed by using the occurrence of a local pixel pair (LPP) consisting of each pixel and its surrounding pixels. However, the 2DH‐based methods often produce over‐stretching artefacts because the low‐textured regions primarily existing in the image induce a spike at some LPPs in the 2DH. To solve this problem, the 2DH is constructed by employing two properties of the human visual system (HVS) as follows: the HVS has the better brightness discrimination in the dark region according to Weber's law, and the HVS is less sensitive to visual artefacts in the higher‐textured regions. To create a spike‐free 2DH, a weighting function reflecting these two properties of the HVS is designed for the LPP. As compared with the popular 2DH‐based methods, the HE with the proposed 2DH can effectively enhance the image contrast while achieving the best perceptual similarity score between the input and output images.