5 April 2021 in IEEE Access
Background subtraction (BGS) is a fundamental video processing task which is a key component of many applications. Deep learning-based supervised algorithms achieve very good performance in BGS, however, most of these algorithms are optimized for either a specific video or a group of videos, and their performance decreases dramatically when applied to unseen videos. Recently, several papers addressed this problem and proposed video-agnostic supervised BGS algorithms. However, nearly all of the data augmentations used in these algorithms are limited to the spatial domain and do not account for temporal variations that naturally occur in video data. In this work, we introduce spatio-temporal data augmentations and apply them to one of the leading video-agnostic BGS algorithms, BSUV-Net. We also introduce a new cross-validation training and evaluation strategy for the CDNet-2014 dataset that makes it possible to fairly and easily compare the performance of various video-agnostic supervised BGS algorithms. Our new model trained using the proposed data augmentations, named BSUV-Net 2.0, significantly outperforms state-of-the-art algorithms evaluated on unseen videos of CDNet-2014. We also evaluate the cross-dataset generalization capacity of BSUV-Net 2.0 by training it solely on CDNet-2014 videos and evaluating its performance on LASIESTA dataset. Overall, BSUV-Net 2.0 provides a ~5% improvement in the F-score over state-of-the-art methods on unseen videos of CDNet-2014 and LASIESTA datasets. Furthermore, we develop a real-time variant of our model, that we call Fast BSUV-Net 2.0, whose performance is close to the state of the art.
15 December 2021 in IEEE Access
Temporal variations, such as sudden motion, acceleration and occlusions, occur frequently in real-world videos and force video-modeling networks to account for them. However. often they are not beneficial for recognizing actions at coarse granularity and thus may impede spatio-temporal learning. Prior solutions to this problem usually introduce multiple network branches to process input frames at different sampling rates or employ special components to explore inter-frame relations, which are computationally expensive. In this paper we propose a simple and flexible Dynamic Equilibrium Module (DEM) for video modeling through adaptive Eulerian motion manipulation. The proposed module can be directly inserted into 3D and (2+1)D backbone networks to effectively reduce the impact of temporal variations on video modeling and learn spatio-temporal representations with higher robustness. We demonstrate performance gains due to the use of DEM in R3D and R(2+1)D models on Kinetics-400, UCF-101, and HMDB-51 datasets.