The video captioning task aims to describe video content using several natural-language sentences. Although one- step encoder-decoder models have achieved promising progress, the generations always involve many errors, which are mainly caused by the large semantic gap between the visual domain and the language domain and by the difficulty in long-sequence generation. The underlying challenge of video captioning, i.e., sequence-to-sequence mapping across different domains, is still not well handled. Inspired by the proofreading procedure of human beings, the generated caption can be gradually polished to improve its quality. In this paper, we propose a deep reinforcement polishing network (DRPN) to refine the caption candidates, which consists of a word-denoising network (WDN) to revise word errors and a grammar-checking network (GCN) to revise grammar errors. On the one hand, the long-term reward in deep reinforcement learning benefits the long-sequence generation, which takes the global quality of caption sentences into account. On the other hand, the caption candidate can be considered a bridge between visual and language domains, where the semantic gap is gradually reduced with better candidates generated by repeated revisions. In experiments, we present adequate evaluations to show that the proposed DRPN achieves comparable and even better performance than the state-of-the- art methods. Furthermore, the DRPN is model-irrelevant and can be integrated into any video captioning models to refine their generated caption sentences.
8 April 2022 in IEEE Transactions on Circuits and Systems for Video Technology
Video captioning is a joint task of computer vision and natural language processing, which aims to describe the video content using several natural language sentences. Nowadays, most methods cast this task as a mapping problem, which learns a mapping from visual features to natural language and generates captions directly from videos. However, the underlying challenge of video captioning, i .e., sequence to sequence mapping across the different domains, is still not well handled. To address these problems, we introduce the polishing mechanism in an attempt to mimic human polishing process and propose a generate-and-polish framework for video captioning. In this paper, we propose a two-step transformer based polishing network (TSTPN) consisting of two sub-modules: the generation-module is to generate the caption candidate and the polishing-module is to gradually refine the generated candidate. Specifically, the candidate provides a global information of the visual contents in a semantically-meaningful order, where it is firstly considered as a semantic intersnubber to bridge the semantic gap between the text and video, with the cross-modal attention mechanism for better cross-modal modeling; and it secondly provides a global planning ability to maintain the semantic consistency and fluency of the whole sentence for better sequence mapping. In experiments, we present adequate evaluations to show that the proposed TSTPN achieves the comparable and even better performance than the state-of-the-art methods on the benchmark datasets.
22 August 2022 in Computers and Electrical Engineering
Face super-resolved (SR) images aid human perception. State-of-the-art face SR methods leverage the spatial location of facial components as prior knowledge. However, it remains a great challenge to generate natural textures. In this paper, we propose a Component Semantic Prior guided Generative Adversarial Network (CSPGAN) to synthesize faces. Specifically, semantic probability maps of facial components are exploited to modulate features in CSPGAN through affine transformation. Furthermore, to compensate the overly-smooth performance of the generative network, a gradient loss is proposed to recover the high-frequency details. Meanwhile, the discriminative network is designed to perform multiple tasks which predict semantic category and distinguish authenticity simultaneously. Extensive experimental results demonstrate that the proposed CSPGAN performs best among compared methods in reconstructing photo-realistic textures.
1 July 2021 in Signal Processing: Image Communication