7 November 2025 in Pattern Recognition
5 March 2025 in IEEE Signal Processing Letters
Action recognition in dynamic video remains challenging, particularly when distinguishing between visually similar actions. While existing methods often rely on holistic representations, they overlook the fine-grained details that are significant for accurate classification. We propose a novel Fine-grained Teacher-student CLIP (FT-CLIP) that integrates body part analysis with holistic action recognition through a teacher-student architecture, bridging the gap between fine-grained action parsing and overall action understanding. The teacher model processes individual body parts alongside specialized description to generate part-specific features, which are then aggregated and distilled into the student model. Through knowledge distillation with learnable prompts, the student model effectively learns to capture subtle action distinctions while maintaining efficient inference. FT-CLIP achieves a more nuanced understanding of complex actions by progressing from detailed body part analysis to comprehensive action recognition. Experiments on Kinetics-TPS under a fully-supervised setting and on HMDB51 and UCF101 under a zero-shot setting demonstrate the effectiveness of our method.
2 September 2025 in IEEE Transactions on Multimedia
As a video understanding task, activity parsing aims at encompassing actions into multiple levels of activity components, including activity, sub-activity and atomic action, enabling understanding of complex video scenes within multimedia systems. Existing methods form activity parsing as a multi-task learning problem to predict multi-granular activity labels simultaneously, which ignores modeling the hierarchical structure and the fine-grained transitions of activity components at different levels. In this paper, we propose a Hierarchical Adaptive Reasoning Graph (HARG) to model the hierarchical structure (i.e., object level $\rightarrow$ atomic action level $\rightarrow$ activity level) dynamically and precisely. To achieve that, an object reasoning graph (ORG) and an atomic action reasoning graph (ARG) are designed to reason fine-grained information transitions between multiple actors at different levels. In addition, an adaptive segmentation module (ASM) is investigated for bridging the gap among different levels, permitting step-by-step reasoning from the object level to the atomic action level. Experimental results show our method outperforms state-of-the-art methods on two activity parsing datasets, achieving hierarchical modeling and fine-grained reasoning for activity understanding. The code is available on GitHub: https://github.com/whuoyj/HARG.
1 April 2026 in IEEE Transactions on Circuits and Systems for Video Technology
Visual Relationship Forecasting (VRF) in video aims to anticipate relations among objects without observing future visual content. The task relies on capturing and modeling the semantic coherence in object interactions, as it underpins the evolution of events and scenes in videos. However, existing VRF datasets provide limited support for learning such coherence. Their noisy annotations fail to reflect distinct action or scene changes, and weak correlations exist between different actions and relational transitions in subject-object pairs. Furthermore, existing methods struggle to distinguish similar relationships and overfit to unchanging relationships in consecutive frames rather than reflecting the semantic coherence of object interactions. To address these challenges, we present SemCoBench, a benchmark that emphasizes semantic coherence for visual relationship forecasting in video. Based on action labels and short-term subject-object pairs, SemCoBench decomposes relationship categories and dynamics by cleaning and reorganizing video datasets to ensure predicting semantic coherence in object interactions. In addition, we propose the Semantic Coherent Transformer (SemCoFormer), which consists of a Relationship Augmented Module (RAM) and a Coherent Reasoning Module (CRM). The RAM bidirectionally enhances cross-modal features to better distinguish similar relationships, while the CRM performs sparse encoding of cross-frame relation transitions to model dynamic relational changes. The experimental results on SemCoBench demonstrate that modeling the semantic coherence is a key step toward reasonable, fine-grained, and diverse visual relationship forecasting, contributing to a more comprehensive understanding of video scenes. Our project page: https://lyao-61.github.io/ SemCo/.