3 June 2021 in IEEE Transactions on Circuits and Systems for Video Technology
In this work, we introduce a novel task – Human-centric Spatio-Temporal Video Grounding (HC-STVG). Unlike the existing referring expression tasks in images or videos, by focusing on humans, HC-STVG aims to localize a spatio-temporal tube of the target person from an untrimmed video based on a given textural description. This task is useful, especially for healthcare and security related applications, where the surveillance videos can be extremely long but only a specific person during a specific period is concerned. HC-STVG is a video grounding task that requires both spatial (where) and temporal (when) localization. Unfortunately, the existing grounding methods cannot handle this task well. We tackle this task by proposing an effective baseline method named Spatio-Temporal Grounding with Visual Transformers (STGVT), which utilizes Visual Transformers to extract cross-modal representations for video-sentence matching and temporal localization. To facilitate this task, we also contribute an HC-STVG datasetThe new dataset is available at https://github.com/tzhhhh123/HC-STVG. consisting of 5,660 video-sentence pairs on complex multi-person scenes. Specifically, each video lasts for 20 seconds, pairing with a natural query sentence with an average of 17.25 words. Extensive experiments are conducted on this dataset, demonstrating that the newly-proposed method outperforms the existing baseline methods.
21 June 2023 in IEEE Transactions on Circuits and Systems for Video Technology
Natural language tracking aims to localize the target object referred to by a language description using a sequence of bounding boxes in video frames. Compared with traditional visual single object tracking initialized with only a bounding box (BBox), this task introduces high-level semantic information to reduce the ambiguity of BBox and enhance the ability to retrieve the target in a global manner. Thus, it can yield more accurate and robust tracking results. Previous methods usually adopt off-the-shelf grounding and tracking branches to tackle this task, where feature representations are learned in isolation without benefiting each other. The two branches can associate with each other to discover crucial clues since the language description and the template image provide information from different sources. Therefore, we propose a unified transformer method for natural language tracking named TransNLT, which utilizes isomorphic Transformer structures for grounding and tracking branches where collaborative learning is enabled to construct comprehensive features for the target. In addition, we propose a Selective Feature Gathering (SFG) module which can integrate cross-modal global information of the tracking target from the visual template and language description. Through effective interaction of visual and language information, we can achieve better results than tracking with only a single modality. Extensive experiments on three popular natural language tracking benchmarks show our proposed TransNLT outperforms previous state-of-the-art methods.
30 January 2024 in IEEE Transactions on Multimedia
iamese and Transformer trackers have demon strated exceptional performance in visual object tracking. These methods utilize initial and potentially online templates to locate the target in subsequent frames. Despite their success, these trackers are vulnerable to changes in the target's appearance due to slow template updates and interference from similar objects, resulting from the absence of scene information. To address these issues, we introduce a reference region within our tracker. The reference region is updated rapidly, providing short-term scene information. By associating the initial template, reference region, and current search region, we enhance the tracker's ability to adapt to changes in target appearance and discriminate between the target and other objects. Additionally, we propose a novel Reference-Enhance (RE) module, which aggregates contextually relevant information from the reference region to enhance the template feature. Extensive experiments show our method achieves state-of-the-art performance on six popular visual object tracking benchmarks while running at over 40 FPS.