25 November 2019 in IEEE Transactions on Instrumentation and Measurement
Real-time vision based system of fault detection (RVBS-FD) for freight trains aims to complete routine maintenance tasks efficiently for ensuring railway transportation security. However, most existing systems are designed to detect only one specific type of faults or even one fault, which fail to deal with multi-fault detection. Recently, the rapid development of deep learning techniques enables systems to provide a robust solution for the RVBS-FD of freight trains. But general convolutional neural networks (CNN) cannot fully meet the actual requirements in terms of the real-time, accuracy, and resource constrains for the RVBS-FD of freight trains. To solve these problems, we propose a CNN-based detector called Light FTI-FDet for the RVBS-FD of freight train. First, we use the multi-region proposal networks which extract a set of prior bounding boxes to achieve initial fault proposal generation. Then, a powerful multi-level region-of-interest pooling is presented for proposal classification and accurate detection. We finally design a reliable model reduction scheme to pursue fast speed with high detection accuracy in a simple manner. The experimental results on five typical fault benchmarks indicate that our Light FTI-FDet achieves higher accuracy and fast speed with about 17% model size of the well-known Faster R-CNN detector, substantially outperforming state-of-the-art methods.
29 April 2024 in International Journal of Computer Vision
The task of few-shot action recognition aims to recognize novel action classes using only a small number of labeled training samples. How to better describe the action in each video and how to compare the similarity between videos are two of the most critical factors in this task. Directly describing the video globally or by its individual frames cannot well represent the spatiotemporal dependencies within an action. On the other hand, naively matching the global representations of two videos is also not optimal since action can happen at different locations in a video with different speeds. In this work, we propose a novel approach that describes each video using multiple types of prototypes and then computes the video similarity with a particular matching strategy for each type of prototypes. To better model the spatiotemporal dependency, we describe the video by generating prototypes that model the multi-level spatiotemporal relations via transformers. There are a total of three types of prototypes. The first type of prototypes are trained to describe specific aspects of the action in the video e.g., the start of the action, regardless of its timestamp. These prototypes are directly matched one-to-one between two videos to compare their similarity. The second type of prototypes are the timestamp-centered prototypes that are trained to focus on specific timestamps of the video. To deal with the temporal variation of actions in a video, we apply bipartite matching to allow the matching of prototypes of different timestamps. The third type of prototypes are generated from the timestamp-centered prototypes, which regularize their temporal consistency while serving as an auxiliary summarization of the whole video. Experiments demonstrate that our proposed method achieves state-of-the-art results on multiple benchmarks.
18 April 2025 in IEEE Transactions on Circuits and Systems for Video Technology
The task of weakly supervised temporal sentence grounding (WSTSG) aims to detect temporal intervals corresponding to a language description from untrimmed videos with only video-level video-language correspondence. For an anchor sample, most existing approaches generate negative samples either from other videos or within the same video for contrastive learning. However, some training samples are highly similar to the anchor sample, directly regarding them as negative samples leads to difficulties for optimization and ignores the correlations between these similar samples and the anchor sample. To address this, we propose Positive Sample Mining (PSM), a novel framework that mines positive samples from the training set to provide more discriminative supervision. Specifically, for a given anchor sample, we partition the remaining training set into semantically similar and dissimilar subsets based on the similarity of their text queries. To effectively leverage these correlations, we introduce a PSM-guided contrastive loss to ensure that the anchor proposal is closer to similar samples and further from dissimilar ones. Additionally, we design a PSM-guided rank loss to ensure that similar samples are closer to the anchor proposal than to the negative intra-video proposal, aiming to distinguish the anchor proposal and the negative intra-video proposal. Experiments on the WSTSG and grounded VideoQA tasks demonstrate the effectiveness and superiority of our method.
8 August 2025 in International Journal of Computer Vision