Dense video captioning is a very challenging task because it requires a high-level understanding of the video story, as well as pinpointing details such as objects and motions for a consistent and fluent description of the video. Many existing solutions divide this problem into two sub-tasks, event detection and captioning, and solve them sequentially (“localize-then-describe” or reverse). Consequently, the final outcome is highly dependent on the performance of the preceding modules. Detecting events or generating sentences directly from vast amounts of video information is prone to loss of elaborateness (accuracy and fluency), and hence propagates as a cascade effect to subsequent modules. Parallel decoding approaches have been proposed to overcome these limitations, which leads to another challenge: the bottleneck at the branch point. In this paper, we decompose this sequential approach by proposing a parallel pathway dense video captioning framework (PPVC) that localizes and describes events simultaneously without any bottlenecks. We introduce a representation organization network at the branching point of the parallel pathway to organize the encoded video feature by considering the entire storyline. Then, an event localizer focuses to localize events without any event proposal generation network, a sentence generator describes events while considering the fluency and coherency of sentences. A gating network controls the flow of organized representations to filter out unnecessary information. Our method has several advantages over existing work: (i) the final output does not depend on the output of the preceding modules, (ii) it improves existing parallel decoding methods by relieving the bottleneck of information. We evaluate the performance of PPVC on large-scale benchmark datasets, the ActivityNet Captions, and YouCook2. PPVC not only outperforms existing algorithms on the majority of metrics but also improves on both datasets by 5.4% and 4.9% compared to the state-of-the-art parallel decoding method.
Dense video captioning aims to localize and describe events for storytelling in untrimmed videos. It is a conceptually very challenging task that requires concise, relevant, and coherent captioning based on high-quality event localization. Unlike simple temporal action localization tasks without overlapping events, dense video captioning requires detecting multiple/overlapping regions in order to branch out the video story. Most existing methods generate numerous candidate event proposals and then eliminate duplicate ones using a event proposal selection algorithm (e.g., non-maximum suppression) or generate event proposals directly through box prediction and binary classification mechanisms, similar to object detection tasks. Despite these efforts, the aforementioned approaches tend to fail to localize overlapping events into different stories, hindering high-quality captioning. In this paper, we propose SBS, a dense video captioning framework with a gradual approach that addresses the challenge of localizing overlapping events and eventually constructs high-quality captioning. SBS accurately estimates the number of explicit events for each video snippet and then detects the boundaries context/activities, which are the details for generating the event proposals. Based on both the number of events and boundaries, SBS generates the event proposals. SBS encodes the context of the event sequence and finally generates sentences describing the event proposals. Our framework is fairly effective in localizing multiple/overlapping events, thus experimental results showthe state-of-the-art performance compared to the existing methods.
On-line video streaming service such as Youtube and Netflix is emerging as killer application that constitutes most IP traffic. Users want high-quality video streaming service, however network bandwidth can not keep up user’s demand. In HTTP-based video streaming technology, the video provider divides the video into multiple chunks, encodes it at various bitrate, and stores it in the media server. The client requests video chunk using an adaptive bitrate (ABR) algorithm considering the network conditions. While there are many works to maximize the user satisfaction, state-of-the-art ABR algorithms are still vulnerable to rebuffering due to inaccurate estimates and decisions. To address this problem, we first analyze real-world traces and find several design inferences for an efficient ABR algorithm. Then, we introduce SATE, an ABR algorithm that provides a stable and agile adaptation even in bandwidth constraint and dramatically changing network conditions.We evaluate the performance of SATE in various network settings and demonstrate its efficacy in practice. SATE provides the stable adaptation and dramatically reduced rebuffering while delivering a similar average bitrate compared to the state-of-the-art ABR algorithms.
HTTP-based video streaming technology is widely used in today’s video delivery services. The streaming solution uses the adaptive bitrate (ABR) algorithm for better video quality and user experience. Despite many efforts to improve the quality of experience (QoE), it is very challenging for ABR algorithms to guarantee high QoE to all users in various environments. The video streaming circumstances in the real world have become even more complicated by the proliferation of mobile devices, high-quality content, and heterogeneous configurations of video players. Many ABR algorithms aim to find monotonous strategies that generally perform well without focusing on the complexity of the environments, which can degrade performance. In this paper, we propose ABRaider that guarantees high QoE to all users in a variety of environments in the real world while being generalized with multiple strategies and specialized in each user’s environment. In ABRaider , we propose multi-phase RL consisting of offline and online phases. In the offline phase, ABRaider integrates the strengths of the ABR algorithms and develops policies suitable for various environments. In the online phase, ABRaider focuses on specializing in the environments of individual users by leveraging the computational power of the clients. Experiment results show that ABRaider outperforms existing solutions in various environments, achieving 19.9% and 42.2% QoE improvement in VoD and live streaming, respectively.
In recent times, there has been a notable increase in efforts to simultaneously comprehend vision and language, driven by the availability of video-related datasets and advancements in language models within the domain of natural language processing. Dense video captioning poses a significant challenge in understanding untrimmed video and generating several event-based sentences to describe the video. Numerous endeavors have been undertaken to enhance the efficacy of the dense video captioning task by the utilization of various approaches, such as bottom-up, top-down, parallel pipeline, pretraining, etc. In contrast, the weakly supervised dense video captioning method presents a highly promising strategy for generating dense video captions solely based on captions, without relying on any knowledge of ground-truth events, which distinguishes it from widely employed approaches. Nevertheless, this approach has a drawback that inadequate captions might hurt both event localization and captioning. This paper introduces PWS-DVC, a novel approach aimed at enhancing the performance of weakly supervised dense video captioning. PWS-DVC’s event captioning module is initially trained on video-clip datasets, which are extensively accessible video datasets by leveraging the absence of ground-truth data during training. Subsequently, it undergoes fine-tuning specifically for dense video captioning. In order to demonstrate the efficacy of PWS-DVC, we conduct comparative experiments with state-of-the-art methods using the ActivityNet Captions dataset. The findings indicate that PWS-DVC exhibits improved performance in comparison to current approaches in weakly supervised dense video captioning.