8 April 2022 in IEEE Transactions on Multimedia
Temporal action detection is a challenging task in video understanding, which is usually divided into two stages: proposal generation and classification. Learning proposal features is a crucial step for both stages. However, most methods ignore temporal information of proposals and consider background and action frames in proposals equally, leading to poor proposal features. In this paper, we propose a novel Temporal Attention-Pyramid Pooling (TAPP) method to learn proposal features of arbitrary length action proposals. The TAPP method exploits the attention mechanism to focus on the discriminative part of proposals, suppressing background influence on proposal features. It constructs a temporal pyramid structure to convert arbitrary length proposal feature sequences to multiple fixed-length sequences while retaining the temporal information. In the TAPP method, we design a multi-scale temporal function and apply it to the temporal pyramid to generate final proposal features. Based on the TAPP method, we construct a temporal action proposal generation model and an action proposal classification model, and then we perform extensive experiments on two mainstream temporal action detection datasets for the temporal action proposal and temporal action detection tasks to verify our models. On the THUMOS'14 dataset, our models based on the TAPP significantly outperform the previous state-of-the-art methods for both tasks.
13 December 2024 in ACM Transactions on Intelligent Systems and Technology
Temporal action detection, a critical task in video activity understanding, is typically divided into two stages: proposal generation and classification. However, most existing methods overlook the importance of information transfer among proposals during classification, often treating each proposal in isolation, which hampers accurate label prediction. In this article, we propose a novel method for inferring semantic relationships both within and between action proposals, guiding the fusion of action proposal features accordingly. Building on this approach, we introduce the Proposal Semantic Relationship Graph Network (PSRGN), an end-to-end model that leverages intra-proposal semantic relationship graphs to extract cross-scale temporal context and an inter-proposal semantic relationship graph to incorporate complementary neighboring information, significantly improving proposal feature quality and overall detection performance. This is the first method to apply graph structure learning in temporal action detection, adaptively constructing the inter-proposal semantic graph. Extensive experiments on two datasets demonstrate the effectiveness of our approach, achieving state-of-the-art (SOTA). Code and results are available at http://github.com/Riiick2011/PSRGN .
17 October 2022 in Multimedia Tools and Applications
17 August 2023 in Applied Intelligence
16 April 2024 in IEEE Access
The spatio-temporal action detection task requires the output of the temporal and spatial positions as well as the action category of the target action instances in the form of action tubes. However, the current definition of video-level metrics in spatio-temporal action detection tasks is not sufficiently clear and unified to fully describe the ability of network models to perform spatio-temporal detection. Furthermore, existing tube linking methods are not only heavily dependent on the quality of the detection stage but also lack reliable linking criteria, resulting in poor tube linking performance. To address these issues, this study proposes a hierarchical linking method based on multiple clues. This method first dynamically utilizes various correlation clues at two levels, including appearance features, spatial overlap, motion prediction, category scores, tube length, and tube confidence status, to reduce the negative impact of unreliable information on the correlation. Then, it employs inter-class correlation to handle the mutual influence between different categories, followed by joint probability data association to address the mutual influence between correlated objects, ultimately achieving robust and accurate online linking of action tubes. The method is experimentally compared with other correlation methods on the untrimmed UCF24 and MultiSports datasets, demonstrating state-of-the-art tube link performance. We also conducted ablation experiments to explore the impact of the different modules and stages in the proposed tube-linking method.