Long Short-Term Dynamic Prototype Alignment Learning for Video Anomaly Detection

1 Introduction

With the ever-growing volumes of surveillance cameras, it is highly demanded how to effectively recognize the abnormal events that may seriously threaten public security in surveillance videos, such as explosion, fighting, crimes, and traffic accidents. It is impractical to promptly detect the abnormal events by watching all surveillance videos, because vast amounts of videos are captured every second. Therefore, we need to develop intelligent video surveillance system to automatically detect abnormal events, where video anomaly detection (VAD) [Huang et al., 2022d; Huang et al., 2022c; Huang et al., 2021] is the core technology.

∗Corresponding author.

Generally, current methods can be grouped into weaklysupervised and unsupervised methods according to the manners of model training. Weakly-supervised VAD [Sultani et al., 2018; Lv et al., 2021b; Huang et al., 2022a; Wu et al., 2023b; Zhang et al., 2022] requires both normal and anomalous data to train model. Sultani et al. [Sultani et al., 2018] established a weakly-supervised dataset UCF-Crime, which activates this direction. Although remarkable gain has been achieved in prior works [Lv et al., 2021b], weakly-supervised VAD still suffers from two drawbacks: 1) it can only recognize abnormal events included in the training set and cannot detect unseen anomalies; and 2) it still needs to collect amounts of abnormal samples, although time-intensive video temporal annotation is not required. Whereas, unsupervised methods generally formulate VAD as an outlier detection problem. The anomaly detector is trained with only normal data for learning the normal patterns. During the test phase, the learned model discriminates behaviors that outside of the learned normal patterns as abnormal events. Early works [Li et al., 2013] mainly focus on manually designing appropriate features to represent videos. However, such approaches are difficult to transfer among different scenarios. With the recent advances in deep learning, deep neural networks (DNNs) based schemes have become the mainstream solutions to VAD. Generally, DNNs-based approaches follow two frameworks: frame reconstruction and prediction. Frame reconstruction-based VAD learns to reconstruct the normal events and detects abnormal events by the larger reconstruction errors. Prediction-based VAD takes prior frames as the input of model to predict current frame and detects frames with poor prediction as anomalies.

Despite the remarkable performance gain, DNNs-based VAD still suffers from three issues: 1) Previous DNNsbased approaches allow to reconstruct or predict anomalies well. As the reconstruction goal is the original input, reconstruction-based models usually reduce the loss by simply memorizing the pixel-level details of inputs, which results in the abnormal frames can be also well reconstructed. Although prediction-based VAD can avoid this problem to some extent, it cannot guarantee larger prediction errors for abnormal behaviors because of the powerful generalization ability of DNNs. 2) Previous prediction-based methods lack sufficient abilities to exploit the long-term temporal contexts of video events. Specifically, existing prediction-based methods only predict a short-distance frame (e.g., the next frame) with tiny differences from the input sequence, which cannot explicitly exploits the temporal contexts. In other words, vanilla frame prediction with small information gap cannot explicitly exploit the temporal contexts. 3) The diversity of normal patterns has been overlooked. For instance, it is normal that no one or large amounts of pedestrians pass by the avenue, but these two cases have completely different dynamic patterns.

In this work, we present a Prototype-guided Dynamics Matching Network (PDM-Net) for VAD to enhance the discriminant and robustness of anomaly detector. Specifically, PDM-Net adopts a novel prototype-guided and dynamicaware video prediction framework for VAD. To tackle the issue 1), we adopt a prototype module which learns representative motion prototypes from normal videos. During the test phase, the motion features of input sequence are reconstructed with normal prototypes, and then reconstructed features are used to predict the target frames. As a result, the model can lessen the generalization ability of model towards anomalies, because abnormal target frames are also predicted with the normal prototypes. To address the issue 2), we propose a Prototype-guided and Dynamic-aware Long-Distance Frame Prediction (PDLP) paradigm for VAD. Our model first stores the normal long-term motion prototypes learned from normal long sequences into prototype module, and then it uses motion features extracted from short sequences to recall the stored normal long-term motion prototypes. Finally, the recalled normal long-term motion prototypes are exploited to help compensate the missing motion information between the short input sequence and long-distance target frame. To mitigate the issue 3), we adopt a feature discrimination module to preserve the diversity of normal patterns.

The main contributions of this work are listed as follows:

1. We develop a novel prototype-guided and dynamicaware long-distance frame prediction framework for VAD, which can jointly help the anomaly detector explicitly exploit the long-term temporal contexts and lessen the generalization ability of model on anomalies.
2. A long-term dynamic prototypical network is designed to learn the long-term dynamic prototypes of normal patterns, which facilitates the long-distance prediction of normal frames and suppresses those of abnormal frames.
3. Experimental results on four public datasets demonstrate the superiority of our method.

2 RelatedWork

2.1 Video Anomaly Detection

Unsupervised video anomaly detection (VAD) [Huang et al., 2022b; Wu et al., 2023a; Huang et al., 2024; Huang et al., 2020] is typically formulated as an out-of-detection task in previous works, where a normal model is trained on only normal videos to recognize events outside of the learned model as anomalies. Some early works [Li et al., 2013] adopt object detection and tracking methods to extract highlevel features. However, these conventional approaches require the prior knowledge to design appropriate features. Recently, DNNs-based VAD methods have demonstrated superior performance over hand-crafted features-based methods. Many approaches adopt deep Auto-Encoder (AE) to learn the normal patterns and quantify the extent of abnormalities using reconstruction errors [Hasan et al., 2016]. As the reconstruction target is the input frame, these deep AE-based models overlook temporal information of videos and typically reduce their loss by memorizing the pixel details of input frame. Subsequently, several variants of AE are developed to capture the temporal patterns of normal videos. For example, Luo et al. [Luo et al., 2021] introduced recurrent neural networks into AE. Liu et al. [Liu et al., 2018] utilized adversarial training to help AE perform frame prediction task. To lessen the generalization ability of DNNs, Gong et al. and Park et al. [Gong et al., 2019; Park et al., 2020] introduced a memory module into deep AE. Lv et al. [Lv et al., 2021a] dynamically learned the prototypes of normal patterns via an attention mechanism for quantifying the normalities of each pixels. These memory-guided methods directly learn normal patterns from the short-distance frame prediction, i.e., utilizing several previous frames to predict current frame. Thus, they lack sufficient abilities to explicitly exploit the long-term temporal contexts of video events. To explicitly exploit the temporal contests, our proposed prototype-guided dynamic matching network (PDMNet) learns how to match the learned normal long-term prototypes with the short-term inputs.

2.2 Prototypical Networks

Prototypical learning has proven its effectiveness in various pattern recognition tasks. The earliest prototypical learning can be traced back to the k-nearest neighbor which represents data with k nearest neighbors [Wang et al., 2022]. Generally, conventional prototypical learning is mainly built on handcrafted features. Recently, prototypical networks combining prototypical learning and DNN have demonstrated their superiority in various tasks [Wang et al., 2021d; Jiang et al., 2022; Wang et al., 2021c; Liu et al., 2023b; Wang et al., 2023; Wang et al., 2021b; Wang et al., 2021a]. Prototypical networks can directly extract deep features by neural networks and incorporate prototypical learning to optimize the neural networks. For instance, Liu et al. [Liu et al., 2023a] presented a region prototypical network to improve the performance of weak supervised image segmentation, which learns region prototypes to locate the inactivated objects more accurately. Lee et al. [Lee et al., 2021] proposed a memory alignment method for improving the performance of video prediction, which learns the prototypes across training samples and uses the learned prototypes to facilitate video prediction at test time. Instead of directly learning prototypes from the inputs, PDM-Net learns long-termmotion prototypes from normal video clips and then recalls them using the short inputs for predicting the long-distance frames.

3 Methodology

3.1 ProblemFormulation

Prediction-based VAD typically leverages frame prediction to train a normal model. Then anomalies are detected by poor prediction based on the following assumption: the learned normal model can predict normal future frames well while poorly predict abnormal frames. Assume Vt-N+1:t = {Xi}ti=t-N+1 indicates a video sequence containing N consecutive frames, and Xt is the t-th frame. Given previous N frames Vt-N+1:t, the goal is to train a predictor P that minimizes the difference between predicted frame ̂Xt+1 = P(Vt-N+1:t) and actual frame Xt+1.

To exploit the temporal information, we develop a novel prototype-guided and dynamic-aware long-distance prediction (PDLP) paradigm, as shown in Figure 1. Specifically, PDLP utilizes the short input video sequence V(t-N+1):(t-(N-n)+1) to predict its long-distance target frame Xt+1. Theoretically, the predictor P needs to bridge the larger information gap by explicitly exploiting the longterm temporal contexts of videos, so as to accurately predict the target frame Xt+1. Actually, only simply increasing the distance between input sequence and target frame cannot enable our model to obtain better performance (detailed in Ablation Study). Thus, we introduce the prototype module to tackle this problem. The dynamic features FSD of short inputs are used as queries to match the normal long-term dynamic prototypes learned from the normal long sequences. Then, the recalled prototype features Fpro are aggregated as FAgg k to predict the target frame Xt+1

where V is the short sequence V(t-N+1):(t-(N-n)+1), and F indicates the prototypical module. Finally, the prediction error and the matching error are used to calculated the anomaly score for detecting anomalies.

3.2 PDLP-Enabled Video Anomaly Detection

Figure 1 shows the architecture of our prototype-guided dynamics matching network (PDM-Net) at the inference phase.

The short input sequence goes through two branches to predict the long-distance target frame Xt+1. One (lower branch of Figure 2) is long-term dynamic prototypes matching (LDPM), which uses the dynamic features of short sequence to match the long-term dynamic prototypes of normal patterns stored in the prototype module. The other one (upper branch of Figure 2) is the proposed PDLP module, which utilizes the spatio-temporal features to predict the target frame with the assistant of matched long-term dynamic prototypical features.

As for the branch of LDPM, the residual sequence R is taken as the input of dynamic encoder ESDM. The dynamic feature FSD = ESDM(R) is extracted for matching the long-term dynamic prototypes Γ from the prototype module. Specifically, a matching vector M is calculated according to the prototype matching strategy. FSD is represented as the weighted combination of prototypes Γ, i.e., Fpro = Γ ⊗M. In this way, Fpro of the short input can be considered to contain the long-term temporal dynamic contexts. Then, Fpro is embedded into the upper branch PDLP to predict the longdistance target frame Xt+1.

With regards to the branch of PDLP, each frame of the short sequence is independently fed into the spatial encoder ESP for extracting the appearance features FSP k = ESP (Xk). Further, FSP k is fed into a spatio-temporal aggregation module composed of a stack of convolutional LSTMs (ConvLSTM) [Shi et al., 2015] in time step orders to capture the temporal relations. Each cell in ConvLSTM outputs a cell memory Ck and a hidden state Hk. Then, we design a feature aggregation module to aggregate the spatio-temporal features and prototypical feature, as shown in Figure 1. Specifically, Fpro and Ck are concatenated and fed into the multi-layer perception to generate the channel-wise attention Apro k = MLP(Fpro©Ck). © indicates concatenate operator. Further, the channel-wise refined prototypical feature ̂Fpro k =

Apro k ⊗Fpro and the hidden featureHk from the ConvLSTMs are concatenated to embed long-term dynamic contexts to the aggregated feature FAgg k = ̂Fpro k ©Hk. Since FAgg k can provide the prior of long-term dynamic contexts to the short input sequence, it is fed into the frame decoder D to generate the predicted frame ̂Xt+1 = D(̂Fpro k ©Hk). Finally, the prediction error between the original frame Xt+1 and predicted frame ̂Xt+1 as well as the matching error between dynamic feature FSD and its prototype Γ∗ are used to calculated the anomaly score (detailed in Section 3.5). At the training phase, we employ a prediction loss to constrain the PDLP, which is defined as

3.3 Dynamic Prototype Matching Learning

Inspired by [Lee et al., 2021], we introduce a long short-term dynamic prototype matching scheme to accurately match the long-term dynamic with the short sequence. Figure 2 illustrates the overall training procedure of our prototype module. The prototype module is trained alternately with two steps: 1) learning long-term dynamic prototypes from normal long sequence; and 2) recalling the corresponding normal longterm dynamic prototypes with the short sequence. Notably, the prototype module is updated only at the first step.

Long-term Dynamic Prototypes Learning

As for the first step, the normal long sequence Vlong N with N consecutive frames is taken as the input. Then, the corresponding residual sequence Rlong N-1 is fed into the long-term dynamic context encoder ELDC to extract the long-term dynamics f LDC . f LDC = {f LDC k }Kk=1 (K = h×w) is divided to exploit local dynamic contexts. Notably, the local dynamic features f LDC k ∈ Rc are regarded as the query items.

The prototype module Γ = {Γ i}Bi=1 contains B prototype items Γ i with c channels. A matching matrix M = {mk}Kk=1

is created to address the location of prototype module, and each vector mk is used to match a query feature f LDC k with all prototype items. Each element value mk,i of mk can be used as the matching probability, which is calculated as

where d(·, ·) is the cosine similarity. The prototype module outputs the local prototypical features f pro k = ∑Bi=1 mk,i · Γ i. The prototypical feature Fpro = {f pro k }Kk=1 is derived as an ensemble of K local feature f pro k . Finally, Fpro is integrated into PDLP to help the short sequence Vshort n predict the long-term target frame Xt+1, as described in Section 3.2.

As for the parameters updating of prototype module, we update the prototype item Γi with all query features whose nearest prototype item is Γi. Let Zi indicate the set of the corresponding query features for Γi. Similar to Eq. 3), the attention weight is calculated by

and renormalized with the query features in Zi as ̂ωk,i = ωk,i maxj∈Zi ωj,i . The parameter updating is represented as

where N(·) denotes ℓ2 norm.

Prototypes Matching Learning

At the second step, the short sequence Vshort n with n (n < N) consecutive frames is taken as the input of model. The goal is to train our model to learn how to recall the long-term dynamic contexts stored in the prototype module using the short inputs. Similar to the first step, the residual sequence Rshort n-1 is used to obtain the corresponding dynamics. The short-term dynamic feature f SD is extracted by a short-term dynamic matching encoder ESDM. The local dynamic feature f SD k of f SDM = {f SD k }Kk=1 (K = h×w) is adopted as the query item. The prototype matching progress is same as the first step. Then the recalled prototypical feature is embedded into LPLP for video prediction. At this step, the parameters of prototype module are fixed. Except for the prototype module, all network parameters are optimized.

3.4 Feature Discrimination

The feature discrimination contains two components, i.e., prototype dispersion and feature compactness. They are respectively embedded into the first and second training step for preserving the diversity of prototypes and reducing the intraclass variances of normal patterns. Specifically, the prototype dispersion enables the prototype module to learn representative prototypes of normal patterns by enforcing prototypes far away from each other at the first training step. Inspired by [Lai et al., 2021], we evaluate the dispersibility between prototypes from the perspective of interclass scatter matrix by

where ki denotes the number of query features that Γ i is their nearest prototype, and ¯ Γ indicates the mean vector of all prototypes in the prototype module. The trace Tr(D) of interclass scatter matrix can be used to measure the dispersibility of prototypes. Thus, we encourage the prototype module to learn various long-term dynamic prototypes by maximizing Tr(D), and the prototype dispersion loss can be represented as Ldis = -Tr(D). Notably, Ldis only optimizes the prototype module at the first training step.

Moreover, the feature compactness enforces the query features close to their nearest prototypes in the latent space for reducing the intraclass variations. In this way, query features with the same dynamic patterns can be accurately matched to the same prototype. Here, we adopt a feature compactness loss [Park et al., 2020] which measures the average distance between query feature and their prototypes

where Γ ∗k is the most-relevant prototype of f SD k .

3.5 Training and Inference

Training. As mentioned above, the optimization of PDMNet contains two steps. At the first step, the network weights of long-term dynamic context encoder ELDC (θ), prototype model F (Γ) and PDLP networks P (φ) are optimized by L1st = Lpre + λ(Ldis + Lcom). The prototype model Γ is updated according to Eq. 5. At the second step, the weights of short-term dynamic matching encoder ESDM (γ) and PDLP networks P (φ) are optimized by L2nd = Lpre + λLcom. In this step, the network weights of prototype module are fixed. The optimization of model is described as Algorithm 1.

Algorithm 1: Optimization of our PDM-Net

Inference. As shown in Figure 1, our PDM-Net only takes short sequence as the input at the testing phase. The LDPM branch extracts the short-term dynamic features to match the long-term dynamic prototypes from the prototype module for assisting the PDLP branch to generate the predicted frame ̂Xt+1. To quantify the abnormal extent of a video frame, the prediction error between the predicted frame ̂Xt+1 and actual frame Xt+1 is used to calculate the anomaly score. Besides, we also evaluate the matching error between the query feature f SD k and its prototype Γ ∗k . Here, we apply the peak signal to noise ratio to measure the prediction error, namely Spre. The lower Spre indicates poorer prediction and Xt+1 tends to be abnormal. We calculate the matching error as

The query feature f SD k is similar to the corresponding normal prototype Γ ∗k indicates Xt+1 is normal. Following the previous methods, we adopt the same normalization function [Liu et al., 2018] to normalize Spre and Smat to [0, 1]. The final anomaly score is calculated as

where β is the balance weight, andM(·) is the normalization function over the whole video frames.

4 Experiments

4.1 Experimental Setup

We conduct experiments on four unsupervised VAD benchmarks, UCSD Ped1 [Li et al., 2013], UCSD Ped2 [Li et al., 2013], CUHK Avenue [Lu et al., 2013], and ShanghaiTech [Luo et al., 2021] to evaluate performance. Under the unsupervised setting, the training set only consists of normal videos. Following prior works [Liu et al., 2018], Area Under ROC (AUC) is adopted as the evaluation metric. We resize each input frames to intensity of [0, 1] and resolution of 256 × 256. Adam with the initial learning rate of 0.0002 is adopted to optimize our PDM-Net. The number B of prototypes in the prototype module is set as 100. The lengths of long and short sequences are set as to 13 and 9, respectively. λ and β are set as 0.1 and 0.6.

4.2 Comparison with Other Methods

We compare our PDM-Net with state-of-the-art VAD approaches. The detailed results are listed in Tabel 1. On ShanghaiTech, PDM-Net is superior to all the methods involved in the comparison with an average AUC of 74.2%, which is higher 0.5% points than that of AMMCN. Our PDM-Net outperforms the vanilla prediction frameworkbased methods FP and PCM, which validates the effectiveness of our PDLP paradigm. On UCSD Ped1, PDM-Net can obtain a new state-of-the-art performance with an average AUC of 85.2%, which is higher 0.9% than the previous best result of 84.3% AUC reported by F2PN. On UCSD Ped2, PDM-Net is superior to all the baselines involved in the comparison with an AUC of 97.7%, which is higher 0.4% than that of previous state-of-the-art method ITAEGM. Compared with previous memory-guided methods MemAE and MNAD, PDM-Net can not only exploit the long-term temporal contexts of normal videos, but also lessen the generalization ability of model to anomalies. The improvement indicates the superiority of our prototype module with dynamics matching learning. On Avenue, PDM-Net obtains a competitive performance with the averaged AUC of 88.1%.

4.3 Ablation Study

We study the contribution of each component in our PDMNet, and the experimental results are shown in Table 2. A basic model (module A) adopts vanilla prediction framework, which is trained with the prediction loss Lpre (Eq. 2). It obtains 82.5% AUC on UCSD Ped1, 94.1% AUC on UCSD Ped2, and 84.8% AUC on Avenue. Then, we adopt the longdistance frame prediction (module B), which only simply increases the distance between the input sequences and target frames. The performance of module B is degraded by 0.4% to 0.5% on all datasets compared with Module A. Without the assistance of long-term dynamic context prototypes in the prototype module, the model lacks sufficient ability to exploit the long-term temporal contexts for bridging the large information gap between input and target, resulting in poor prediction on both normal and abnormal frames.

Impact of Prototype Module

We embed the prototype module into the model A and B to construct the module C and D, respectively. Clearly, embedding our prototype module enables module C to obtain 1.3% to 1.7% improvements over the basic module A on all datasets. Comparing module B and module D, we can observe that embedding prototype module enables module D to obtain the performance improvements of 2.5%, 3.2% and 2.7% of AUC scores on three datasets. The significant performance improvements demonstrate the validity of our prototype module.

Impact of PDLP Framework

We adopt our proposed prototype-guided and dynamic-aware long-distance frame prediction (PDLP) framework (module D), i.e, embedding the prototype module into the longdistance frame prediction-based module B. Compared with the vanilla frame prediction framework (Module A), PDLP framework-based module D can achieve the performance gains of 2.0%, 2.8%, and 2.2% of AUC scores on three datasets, which demonstrates the superiority of our PDLP framework. Compared with the module C which is embedded the same prototype module, our PDLP framework can improves the performance by 0.7% on UCSD Ped1, by 1.1% on UCSD Ped2 and by 0.9% on Avenue, which indicates that the performance gain of our PDLP does not simply come from the superposition of modules, but from the more effective coupling between modules.

Impact of Feature Discrimination

Then, we embed the feature discrimination module into PDLP framework-based module D to construct our overall model (module E). The feature discrimination enables our model to obtain improvements of 0.7%, 0.8% and 1.1% on three datasets, respectively. The performance improvement demonstrates the effectiveness of feature discrimination.

Visualization of Long-Distance Prediction

We show the actual and residual frames to check the reliability of our model for VAD. In Figure 3, the second and fourth rows are residual frames between actual and predicted frames. Red boxes represent the abnormal regions. It is obvious that the prediction of anomalies by our PDM-Net is significantly worse than that of normal regions, which indicates that our method can achieve the goal of lessening the generalization ability of learned model to anomalies.

Visualization of Anomalous Event Detection

Figure 4 shows some instances of anomaly score curves from our PDM-Net. It is clear that our PDM-Net is able to correctly response to normal and abnormal events in time. Specifically, the curve rises sharply when an anomalous event suddenly appears and continuously remains at a quite high level when the anomalous event is in progress. When the objects resulting in the anomalies disappears, the curves drop to a relatively low level.

5 Conclusion

In this paper, we propose a novel prototype-guided dynamics matching framework that can jointly exploit the long-term temporal contexts and preserve the diversity of normal patterns, while lessening the generalization ability of model to anomalies. Specifically, a prototype-guided and dynamicaware long-distance frame prediction (PDLP) framework is adopted to exploit the temporal contexts of normal events. Moreover, a prototype module with dynamic matching learning is designed to provide the short normal inputs with long-term dynamic prototypes of normal events, which helps the model to bridge the large information gap for achieving PDLP. Extensive experimental results on four datasets demonstrate the superiority of our method.

Acknowledgments

This work was supported by National Natural Science Foundation of China (62301621) and Shenzhen Science and Technology Program (20231121172359002).