31 August 2023 in IEEE Signal Processing Letters
Memory network has been extensively used to record prototypical normal patterns to prevent overgeneralization of the network to reconstruct anomalies for video anomaly detection. However, existing memory-based methods only record the lossy representation of normal item prototypes, without recording the rich relationships between them. In this work, we propose an Associative Memory with Spatio-Temporal Enhancement (AMSTE) which introduces the global context information constraint of motion to enhance the appearance features and learn the normal item prototypes and their relationship. Specifically, we utilize two encoders to extract spatio-temporal features with the Spatio-Temporal Enhancement Module (STEM) to enhance appearance features with global motion constraints. Then, the prototypical patterns of normal data and their relationships are recorded in the item memory and relational memory, respectively. Finally, we retrieve features from the memory pools and reconstruct the video frame through the decoder. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our approach. The code will be released at https://github.com/HuYongting/AMSTE .
9 May 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Existing unsupervised video anomaly detection methods based on prediction typically employ a memory module to limit the generalization ability of the network so that normal frames can be accurately reconstructed or predicted while abnormal frames cannot. These memory-based methods usually utilize memory to record the fusion prototypes of appearance and motion. However, the motion part of the fusion prototypes is obtained from implicit motion representation, which is incomplete and constrain the ability for abnormal detection. To tackle the above issue, we proposed a Two-Stage framework with Memory via video Decomposition and bidirectional Consistency (TSMDC), which employs explicit motion data to learn comprehensive motion prototypes and use video decomposition and bidirectional consistency to learn fine granularity and advanced prototypes. We first decompose the video clip into three components: motion, scene, and object. In the first stage, the features of the motion are extracted to obtain a comprehensive motion representation, and its prototypes are stored in the motion memory. For further use of the bidirectional information of video, we present a cascaded frame prediction network that is utilized to learn and record the advanced spatio-temporal prototypes in the second stage. Specifically, the fine-granularity features of the scene and object are extracted and fused to predict the future frame. Then the initial frame of the video clip is predicted based on bidirectional consistency and motion prototypes enhancement. And the advanced spatio-temporal prototypes of video are recorded in this process. Anomalies are evaluated using the combination anomaly score of the predicted future and initial frame.Extensive experimental results on three public datasets indicate the effectiveness of the proposed method. Code will be available at https://github.com/yangugu/TSMDC.
4 July 2026 in Medical Image Analysis
17 July 2026 in Biomedical Signal Processing and Control