Jiahao Zhu

Jiahao Zhu

Auto-generated profile, based on publicly available information

Jiahao Zhu

Hubei Key Laboratory of Distributed System Security, Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan, China

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
17 November 2023 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
DLDaizong Liu
Daizong Liu
JZJiahao Zhu
Jiahao Zhu
XFXiang Fang
Xiang Fang
ZXZeyu Xiong
Zeyu Xiong
Huan WangHuan Wang
Huan Wang
PZPan Zhou
Pan Zhou
Temporal sentence grounding (TSG) aims to locate a semantically related segment of an untrimmed video guided by a sentence query. Since the untrimmed videos are too long, almost all existing TSG works first sparsely down-sample each video into a shorter video of a fixed length and then conduct multimodal interactions with the query sentence for reasoning. However, this video down-sampling process may introduce a challenging issue that confuses the latter grounding process: Due to the video down-sampling, some query-related frames may be filtered out; this process may remove the specific boundary frames of the target segment and take the adjacent irrelevant frames as new boundaries, easily leading to cross-modal misalignment and introducing both boundary-bias and reasoning-bias. Therefore, it is important to keep the grounding consistency (both temporal annotations and boundary predictions) between the original and the sampled videos. To this end, in this paper, we propose a novel Conditional Video Diffusion Network (CVDN) for TSG to learn extra visual semantics to enrich and refine the biased new boundaries, which enables soft-label boundary prediction for fine-grained frame-query reasoning. Specifically, we first construct a conditional video diffusion model which is separately trained to recover the consecutive semantics of the filtered frames between the adjacent sampled frames. Through the designed stochastic interval sampling strategies in the training process, this diffusion model is able to generate absent coherent semantics between the sparsely sampled frames and in turn enrich and refine them, benefiting the integral activity understanding for TSG. In this manner, the incorrect new boundaries will be refined to be closely correlated to the original boundary frames and contain sufficient query-related information, which is crucial for accurate segment prediction. Extensive experiments on three challenging datasets demonstrate the effectiveness of CVDN.

2 September 2024 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
ZXZeyu Xiong
Zeyu Xiong
DLDaizong Liu
Daizong Liu
XFXiang Fang
Xiang Fang
XQXiaoye Qu
Xiaoye Qu
JDJianfeng Dong
Jianfeng Dong
PZPan Zhou
Pan Zhou
Video sentence grounding (VSG) is the task of identifying the segment of an untrimmed video that semantically corresponds to a given natural language query. While many existing methods extract frame-grained features using pre-trained 2D or 3D convolution networks, often fail to capture subtle differences between ambiguous adjacent frames. Although some recent approaches incorporate object-grained features using Faster R-CNN to capture more fine-grained details, they are still primarily based on feature enhancement and lack spatio-temporal modeling to explore the semantics of the core persons/objects. To solve the problem of modeling the core target's behavior, in this paper, we propose a new perspective for addressing the VSG task by tracking pivotal objects and activities to learn more fine-grained spatio-temporal features. Specifically, we introduce the Video Sentence Tracker with Memory Network and Masked Attention (VSTMM), which comprises a cross-modal targets generator for producing multi-modal templates and search space, a memory-based tracker for dynamically tracking multi-modal targets using a memory network to record targets' behaviors, a masked attention localizer which learns local shared features between frames and eliminates interference from long-term dependencies, resulting in improved accuracy when localizing the moment. To evaluate the performance of our VSTMM, we conducted extensive experiments and comparisons with state-of-the-art methods on three challenging benchmarks, including Charades-STA, ActivityNet Captions, and TACoS. Without bells and whistles, our VSTMM achieves leading performance with a considerable real-time speed.

2 January 2023 in arXiv
Verified
Preprint
Open access
PDF file
JZJiahao Zhu
Jiahao Zhu
DLDaizong Liu
Daizong Liu
PZPan Zhou
Pan Zhou
XDXing Di
Xing Di
YCYu Cheng
Yu Cheng
ZXZeyu Xiong
Zeyu Xiong
Temporal sentence grounding (TSG) aims to identify the temporal boundary of a specific segment from an untrimmed video by a sentence query. All existing works first utilize a sparse sampling strategy to extract a fixed number of video frames and then conduct multi-modal interactions with query sentence for reasoning. However, we argue that these methods have overlooked two indispensable issues: 1) Boundary-bias: The annotated target segment generally refers to two specific frames as corresponding start and end timestamps. The video downsampling process may lose these two frames and take the adjacent irrelevant frames as new boundaries. 2) Reasoning-bias: Such incorrect new boundary frames also lead to the reasoning bias during frame-query interaction, reducing the generalization ability of model. To alleviate above limitations, in this paper, we propose a novel Siamese Sampling and Reasoning Network (SSRN) for TSG, which introduces a siamese sampling mechanism to generate additional contextual frames to enrich and refine the new boundaries. Specifically, a reasoning strategy is developed to learn the inter-relationship among these frames and generate soft labels on boundaries for more accurate frame-query reasoning. Such mechanism is also able to supplement the absent consecutive visual semantics to the sampled sparse frames for fine-grained activity understanding. Extensive experiments demonstrate the effectiveness of SSRN on three challenging datasets.

21 February 2023 in arXiv
Verified
Preprint
Open access
PDF file
ZXZeyu Xiong
Zeyu Xiong
DLDaizong Liu
Daizong Liu
PZPan Zhou
Pan Zhou
JZJiahao Zhu
Jiahao Zhu
Temporal sentence grounding (TSG) aims to localize the temporal segment which is semantically aligned with a natural language query in an untrimmed video.Most existing methods extract frame-grained features or object-grained features by 3D ConvNet or detection network under a conventional TSG framework, failing to capture the subtle differences between frames or to model the spatio-temporal behavior of core persons/objects. In this paper, we introduce a new perspective to address the TSG task by tracking pivotal objects and activities to learn more fine-grained spatio-temporal behaviors. Specifically, we propose a novel Temporal Sentence Tracking Network (TSTNet), which contains (A) a Cross-modal Targets Generator to generate multi-modal templates and search space, filtering objects and activities, and (B) a Temporal Sentence Tracker to track multi-modal targets for modeling the targets' behavior and to predict query-related segment. Extensive experiments and comparisons with state-of-the-arts are conducted on challenging benchmarks: Charades-STA and TACoS. And our TSTNet achieves the leading performance with a considerable real-time speed.