Visual and semantic saliency are important in image captioning. However, single-phase image captioning benefits little from limited saliency without a saliency predictor. In this paper, a novel saliency-enhanced re-captioning framework via two-phase learning is proposed to enhance the single-phase image captioning. In the framework, visual saliency and semantic saliency are distilled from the first-phase model and fused with the second-phase model for model self-boosting. The visual saliency mechanism can generate a saliency map and a saliency mask for an image without learning a saliency map predictor. The semantic saliency mechanism sheds some lights on the properties of words with part-of-speech Noun in a caption. Besides, another type of saliency, sample saliency is proposed to explicitly compute the saliency degree of each sample, which helps for more robust image captioning. In addition, how to combine the above three types of saliency for further performance boost is also examined. Our framework can treat an image captioning model as a saliency extractor, which may benefit other captioning models and related tasks. The experimental results on both the Flickr30k and MSCOCO datasets show that the saliency-enhanced models can obtain promising performance gains.

How to integrate multimodal information sources for face naming in multimodal news is a hot and yet challenging problem. A novel deep cross-modal face naming scheme is developed in this paper to facilitate more effective people news retrieval for large-scale multimodal news. This scheme integrates deep multimodal analysis, cross-modal correlation learning and multimodal information mining, in which the efficient naming mechanism aims to cluster the deep features of different modalities into a common space to explore their inter-related correlations, and a special Web mining pattern is designed to optimize the name-face matching for rare non-celebrity. Such a cross-modal face naming model can be treated as a problem of bi-media semantic mapping and modeled as an inter-related correlation distribution over deep representations of multimodal news, in which the most important is to create more effective cross-modal name-face correlation and measure to what degree they are correlated. The experiments on a large number of public data from Yahoo! News have obtained very positive results and demonstrated the effectiveness of the proposed model.