Dense video captioning aims to localize and describe multiple events in untrimmed videos, which is a challenging task that draws attention recently in computer vision. Although existing methods have achieved impressive performance, most of them only focus on local information of event segments or very simple event-level context, overlooking the complexity of event-event relationship and the holistic scene. As a result, the coherence of captions within the same video could be damaged. In this paper, we propose a novel event-centric hierarchical representation to alleviate this problem. We enhance the event-level representation by capturing rich relationship between events in terms of both temporal structure and semantic meaning. Then, a caption generator with late fusion is developed to generate surrounding-event-aware and topic-aware sentences, conditioned on the hierarchical representation of visual cues from the scene level, the event level, and the frame level. Furthermore, we propose a duplicate removal method, namely temporal-linguistic non-maximum suppression (TL-NMS) to distinguish redundancy in both localization and captioning stages. Quantitative and qualitative evaluations on the ActivityNet Captions and YouCook2 datasets demonstrate that our method improves the quality of generated captions and achieves state-of-the-art performance on most metrics.

This paper proposes a scalable approach for counting pedestrians crossing a virtual line when the crowd is highly dynamic and possibly extremely dense. The approach mainly consists of two parts: local crowd density estimation and pedestrian counting based on accumulating local densities across the line. To obtain a fine estimation of local crowd densities, we divide the neighborhood at the line into a number of blocks. We enforce spatial consistency between local counts in the blocks and those in the enclosing regions to guarantee consistent estimation of local crowd densities. For scalability to various density levels in crowd density estimation, we propose a two-stage strategy: pre-classification of density levels and subsequent regression with overlapped operational ranges. To count pedestrians crossing the virtual line, we accumulate the crowd densities across the line according to the locally estimated velocities. Extensive experimental results demonstrate the effectiveness of the proposed approach and its scalability to crowdedness.

In generic object detection, detectors are often susceptible to foreground objects and background regions that share similar appearances. In this paper, we propose a novel discriminative region mining (DRM) module for object detection, which enables discriminative region localization and representation for accurate object identification. The DRM module is collaboratively optimized by an extra intramodule classification loss in addition to the usual detection loss, which ensures its adequate discriminative capability. Specifically, two derivatives of the DRM module, namely a local DRM module and a contextual DRM module are proposed to excavate local and contextual discriminative regions, respectively. Furthermore, we extend the local DRM module to capture multiple local discriminative regions with a diversity constraint. To explore informative local features, an image upsampling branch is introduced to generate fine-grained representation for the local DRM module. Extensive experiments on the PASCAL VOC and MS COCO datasets demonstrate the effectiveness of the proposed method. Simple baseline detectors with the built-in DRM can achieve state-of-the-art detection performance. For example, the proposed detector achieves a mean average precision of 81.0% on PASCAL VOC 2007 with an input size of 300 × 300 using a ResNet-18 backbone at 24.2 fps on an Nvidia Titan X GPU.

Recently, memory-based networks have achieved promising performance for video object segmentation (VOS). However, existing methods still suffer from unsatisfactory segmentation accuracy and inferior efficiency. The reasons are mainly twofold: 1) during memory construction, the inflexible memory storage mechanism results in a weak discriminative ability for similar appearances in complex scenarios, leading to video-level temporal redundancy, and 2) during memory reading, matching robustness and memory retrieval accuracy decrease as the number of video frames increases. To address these challenges, we propose an adaptive sparse memory network (ASM) that efficiently and effectively performs VOS by sparsely leveraging previous guidance while attending to key information. Specifically, we design an adaptive sparse memory constructor (ASMC) to adaptively memorize informative past frames according to dynamic temporal changes in video frames. Furthermore, we introduce an attentive local memory reader (ALMR) to quickly retrieve relevant information using a subset of memory, thereby reducing frame-level redundant computation and noise in a simpler and more convenient manner. To prevent key features from being discarded by the subset of memory, we further propose a novel attentive local feature aggregation (ALFA) module, which preserves useful cues by selectively aggregating discriminative spatial dependence from adjacent frames, thereby effectively increasing the receptive field of each memory frame. Extensive experiments demonstrate that our model achieves state-of-the-art performance with real-time speed on six popular VOS benchmarks. Furthermore, our ASM can be applied to existing memory-based methods as generic plugins to achieve significant performance improvements. More importantly, our method exhibits robustness in handling sparse videos with low frame rates.