Safeguarding Sustainable Cities: Unsupervised Video Anomaly Detection through Diffusion-based Latent Pattern Learning

1 Introduction

The development of sustainable cities requires high-quality community management, which is strongly supported by surveillance technology. Through surveillance systems, community managers can monitor activities in their neighborhoods and effectively ensure that normal activities are taking place. To reduce the cost of human supervision, Video Anomaly Detection (VAD), which is designed to automatically detect abnormal activities [Wu et al., 2021], has gained increasing attention.

In the real world, anomalous events are rare and unconstrained, which makes collecting and labeling enough anomalous events nearly impossible. Accordingly, most previous methods detect anomalies in a weakly supervised setting that learns with video-level labels [Sultani et al., 2018; Tian et al., 2021], or One-Class Classification (OCC) supervised setting that only learns from normal data [Wang et al., 2022; Chen et al., 2022]. The OCC supervised VAD refers to model the distribution of normal pattern with normal data, labeling the events that deviate from the model as anomalous events. However, labeling videos or labeling training sets as purely normal events is still labor-intensive and costly, especially when dealing with massive amounts of surveillance video in the real world. To alleviate this problem, a natural idea is to perform Unsupervised VAD (UVAD) which aims to detect anomalies directly from completely unlabeled videos as shown in Figure 1. Several recent works [Giorno et al., 2016; Pang et al., 2020; Yu et al., 2022; Zaheer et al., 2022] have explored this approach. Figure 1 illustrates the different supervised settings in VAD.

* Equal Contribution.

† Corresponding Author.

Despite the progress made, two prominent limitations constrain further improvements in the performance of existing UVADmethods. (1) Representative methods [Yu et al., 2022; Zaheer et al., 2022] generate pseudo-labels by reconstruction or simple classification, and the model learns the distribution of normal patterns under the supervision of pseudo-labels. However, the noise in the pseudo-labels (incorrect pseudolabels) limits the effectiveness of model. (2) Existing UVAD methods attempt to overfit the training set for learning normal patterns, ignoring the fact that unsupervised anomaly detection is essentially an open-set problem. In the unsupervised setting, the training set cannot include all normal patterns, thus only fitting the training set will make the model treat the unseen normal events as abnormal events.

Inspired by recent studies [de Morais et al., 2019; Chen et al., 2022; Flaborea et al., 2023] of the distribution of features of abnormal skeletal anomalies, we explore the distribution of latent patterns of abnormal and normal events. As shown in Figure 2, we find distributional difference in the latent patterns. In the case of normality, different patterns of the same event are varied but correlated, i.e., they are biased towards the ground truth of what actually happens. In the case of abnormality, different patterns of the same event are diverse but not correlated. The distinctions outlined above offer the potential to identify diverse normality and abnormality within the real world. Nevertheless, exploiting this discrepancy requires the generation of multiple diverse patterns for the same occurrence. Introducing diffusion models [SohlDickstein et al., 2015; Ho et al., 2020] into UVAD might fulfill this requirement. Diffusion models are characterized by minor modifications and corrections to the generated samples at each step, thus having the ability to learn multiple latent patterns or forms of the same action or event.

To this end, we propose a latent pattern learning framework for UVAD based on diffusion models, called DiffVAD, which comprehensively consider the diversity of normal and abnormal patterns. Given a video sequence that is either normal or abnormal, the video frames are corrupted to random noise, after which multiple patterns of the corrupted frames are reconstructed based on the clean input frame in the past. DiffVAD distinguishes between normal and abnormal by comparing the distribution of different patterns, with different normal patterns for the same event tending to be more correlated. To enhance this correlation, we condition the reconstruction of damaged frames on past motion features. Furthermore, we refine the pseudo-labels based on knowledge aggregation from neighbours sample to reduce the effect of noise. Since noise in pseudo-labels is unavoidable, we evaluate the reliability of the refined pseudo-labels to penalize unreliable pseudo-labels to guide learning through reliable pseudo-labels.

The main contributions of the article are as follows:

• A diffusion-based latent pattern UVAD learning framework is proposed for sustainable cities. We exploits the motion conditioned diffusion model to generate multiple patterns of the same event to account for the openness of anomaly detection.
• A pseudo-label refinement strategy based on knowledge aggregation is proposed to alleviate the effect of noise. We weight the loss based on the reliability of pseudolabels to train the model with reliable pseudo-labels.
• Extensive experiments on six benchmark datasets show that our method achieves state-of-the-art performance and even outperforms the OCC methods.

2 RelatedWork

2.1 Video Anomaly Detection

VAD as One-Class Classification. Most previous work considers VAD as one class classification task due to the difficulty of collecting and labeling abnormal samples. The detection model learns the feature distribution of the normal pattern only from the normal samples during training, and deviation points that do not conform to the model representation are labeled as anomalies during detection. Reconstructionbased methods [Lv et al., 2021] and prediction-based methods [Liu et al., 2018a; Wang et al., 2022] are the two mainstream methods. These two methods typically use autoencoders (AEs) [Hasan et al., 2016], memory-augmented AEs [Gong et al., 2019; Lv et al., 2021] to reconstruct predict frames so that frames with large reconstruction or prediction errors are recognized as anomalies.

Weakly Supervised VAD. Unlike OCC methods that require frame-level annotations, some work [Sultani et al., 2018; Feng et al., 2021; Tian et al., 2021; Wu et al., 2021; Zhang et al., 2023] has attempted to train models with videolevel annotations. Video-level labels are provided in such a way that normal labeled videos are completely normal, while abnormal labeled videos contain both normal and abnormal content. Mainstream weakly supervised methods [Feng et al., 2021; Lv et al., 2023] detect video-level anomalies through multi-instance learning.

Unsupervised VAD. To address the need for detection of large amounts of unlabeled surveillance video in the real world, limited recent work [Giorno et al., 2016; Pang et al., 2020; Yu et al., 2022; Zaheer et al., 2022] has explored UVAD. For instance, Del et al. [Giorno et al., 2016] pioneer the exploration of UVAD by detecting dramatic changes in the anomalies. Pang et al. [Pang et al., 2020] obtain preliminary results by using pre-trained DNNs then refine the results by performing two classes of ordered regression in a self-training manner. Yu et al. [Yu et al., 2022] propose the normality advantage and extend the reconstruction from OCC supervised VAD to UVAD. Zaheer et al. [Zaheer et al., 2022] proposes a cooperative learning framework to generate pseudo-labels for self-training of detection models.

Although the above methods can detect anomalies from unlabeled data through self-training and pseudo-labeling, the noise in the pseudo-labels limits the improvement of performance. Meanwhile, these methods fit the distribution of the training set without considering the unseen normal patterns. In contrast to the above methods, we refine the pseudo-labels and estimate their uncertainty to eliminate noise. Furthermore, diffusion model [Ho et al., 2020] is introduced to account for the diversity of normal and abnormal patterns.

2.2 DiffusionModel

Recently, diffusion models [Ho et al., 2020] have achieved superior performance in many generative tasks, outperforming GANs [Kaneko and Harada, 2020] and AEs [Kim, 2022]. Diffusion models have also found many applications in computer vision tasks such as image restoration [Song et al., 2021], image super-resolution [Metzger et al., 2023], and image editing [Yang et al., 2023a]. For VAD, Yan et al. [Yan et al., 2023] introduces diffusion models to improve anomaly detection. They utilizes the robustness of the diffusion model to noise to predict and reconstruct video frames in feature space without resorting to other auxiliary networks. However, this OCC work is only a single-pattern reconstruction of events and is not applicable to UVAD with openness. The latest work [Flaborea et al., 2023] utilizes diffusion models to generate unbounded skeletal motion to detect anomalies, but its applicability is limited to human-related anomalies.

Inspired by generating unbounded data [Flaborea et al., 2023], for unlabeled videos, our work covers multiple patterns of normal and anomalies by latent pattern learning of diffusion models. Differently, our approach learns unbounded patterns rather than being limited to skeletal motion. In particular, to enable the model to accurately learn abnormal and normal from unlabeled data, we propose a pseudo-label refinement strategy to alleviate the effect of noise.

3 Methodology

3.1 Overall Framework

Figure 3 depicts the overall architecture of proposed framework. The diffusion-based latent pattern learning framework for UVAD consists of a diffusion-based generator (performing forward diffusion process and reverse diffusion process) and pseudo-label refinement module. In addition, we introduce motion conditions in the generator network to help the network generate more relevant multiple patterns for the input frames. Specifically, we condition on the motion features of the history frame sequence that preceded the input frame sequence. Pseudo-labels for input frames can be generated by observing the distribution of generated patterns (counting the distances of potential patterns from the ground truth); discrete distributions are more likely to be anomalous. The whole framework is trained under the supervision of pseudo-labels.

3.2 Generation of Latent Patterns

We define a diffusion-based generative network that learns to reconstruct multiple latent patterns for corrupted future frame sequences under the condition of clean past frame sequences.

Let I = {i1, . . . , iN} be a sequence belonging to N time-continuous frames. We divide X into two parts: a sequence of past history frames I1:k and a future sequence Ik+1:N with k ∈ {1, . . . , N}.

In the forward process q, we corrupt the sequence of future video frames by adding random noise. We sample a random noise map εk+1:N from the distribution N(0, I) and add it to Ik+1:N to randomly destroy its pixels.

The extent of added noise depends on the variance scheduler βt ∈ (0, 1) and the diffusion time step t. As a result, q at each diffusion time step t makes it=T n indistinguishable from a noisy frame with random sampling.

The reverse process pθ performs denoising of the noisy frames and estimates the noise map εk+1:N through the U-net structure εθ. The architecture of the main diffusion model is the neural network represented by the purple blocks in Figure 3, tasked with estimating the noise in the sequence of input frames and consequently generating multiple latent patterns. Our diffusion network of latent patterns gradually shrinks and then expands (reconstructs) the spatial dimension of the input frame sequence. To take into account the temporal dimension of the input sequence, we construct a U-Net with the spatio-temporally separable GCN (STS-GCN) layer proposed in [Sofianos et al., 2021]. In detail, the U-Net receives the input Ik+1:N and the motion time conditioning signal h+τθ(t), which provides the network with the diffusion time step and encoded features of I1:k. Furthermore, to align the dimension of this conditioning signal with the dimension of the network layer, the former is fed to the embedding layer, which projects it to the correct vector space. This embedded conditioning signal is then fed to each STS-GCN layer.

Formally, to obtain an approximation of εk+1:N , we define the approximation objective conditioned on the diffusion time step t and the feature embedding h of I1:k. We define the approximation objective as [Flaborea et al., 2023]:

We smooth Lappr as follows:

Given a random noise sample z and consider it as the start for generating latent patterns I P=m k+1;N of the input frame sequence. P = m represents mth pattern with m ∈ {1, . . . ,M}. We perform the generation by [Flaborea et al., 2023]:

We generate M distinct latent patterns I 1 , . . . , I M . For each I m, we compute the reconstruction error by means of the smoothing loss Lsmooth (|I - I m|) used in training.

Conditions of Generation. The conditioning strategy is a key element of the diffusion model we designed, as it directly affects the quality of the output. Depending on the features of the history k frames, the latent patterns of the output can be more relevant to the ground truth, helping to distinguish between normal and anomalous.

The right part in Figure 3 illustrates our conditioning strategy, where we use 3D convolution on I1:k to extract features embedding h. 3D convolution to extract features as conditioning signals is simple and effective. Since the diffusion models benefits from being conditional on the time step t, we add the embedding τθ(t) to the embedding h and feed the generated motion time signal to each layer of our network εθ. τθ(t) is implemented with a MLP.

3.3 Generation of Pseudo-labels

In our proposed UVAD framework, the pseudo-labels generated from this round of training are used for the next round of training. The pseudo-labels are generated on the basis of observing the distribution of the generated latent patterns.

M different latent patterns I 1 , . . . , I M is generated by εθ, we compute tthe mean of the distances of the generated latent patterns relative to the ground truth by smoothing the loss:

We use Dth as the thresholds for generator εθ, ̂ yI as the pseudo-lable of I:{

Inspired by [Zaheer et al., 2022], we choose to generate thresholds by a data-driven approach. For the output of generator, µD and σD is the mean and standard deviation of distance D for each batch. The Dth is as follow:

This approach of using a fixed percentage of the maximum distance of the generated latent patterns relative to the ground truth as the threshold is simple and effective. In addition we have tried other implementations, such as continuously adjusting the thresholds with training. These methods have the same effect as the approach of Eq. (9), which shows that our method is robust to the selection of thresholds.

3.4 Refinement Strategy of Pseudo-Labels

The refinement of pseudo-labels is done by aggregating the knowledge of the nearest neighbor samples. We assume that features of semantically similar images should be neighboring in the feature space. This assumption is satisfied by extracting similar features from the samples using contrastive learning. The strategy is shown on the left side of Figure 3.

Formally, given a video frame in and a feature extractor fθ, we obtain a feature vector z = fθ(in) from the video frame in. The feature z is then used to search for neighbors of the sample in in the feature space. Thus, the pseudo-label of in is refined by aggregating knowledge from the selected neighbors. For this purpose, the anomaly probabilities ˆ pi outputs of the selected neighbors are averaged to perform a soft voting:

where I is the index set of the selected neighbors, and ˆ pi is the average score of vector zi. To obtain fine pseudo-labels, we use the argmax operation on ˆ pi:

The above refinement process entails searching a representation of the feature space in which neighboring samples are searched. This is allowed by the bank, which stores the features obtained from the samples and the original anomaly probabilities. Neighbors are then selected by calculating the cosine distance between the features of in and the features stored in the bank. According to [He et al., 2020], in order to keep the information stored in the bank more stable, we use the slowly varying momentum model g′i (·) = h′i (f ′ i (·)) to update the feature z and the anomaly probabilities p′ .

The refined pseudo-label ̂ yi is used as a self-supervised signal for loss calculation during training.Since refinement is an iterative process, the refined pseudo-labels obtained through knowledge aggregation of neighbors still contain some noise. To address this problem, we reweight the training loss by estimating the uncertainty of the pseudo-label refinement.

Following [Litrico et al., 2023], we exploit an entropybased uncertainty estimation method that exploits the consistency between neighboring predictions. The basic idea is that if the network predicts the same kind of neighboring samples, we can consider the derived pseudo-labels to be reliable (low uncertainty). The average score vector ˆ pi, obtained by averaging the probabilistic outputs of the neighbors, has low entropy in the former case while high entropy in the latter. Therefore, we reweight the training loss by computing the weight w. The weight w is more important for pseudolabels obtained from ˆ pi with low entropy and less important for pseudo-labels obtained from ˆ pi with high entropy.

Formally, given the frame in, we obtain the average score vector ˆ pi from the soft-voting strategy. note that ˆ pi is obtained by averaging over a probability distribution, thus it is still a probability distribution. We compute the entropy of ˆ pi as:

Based on the normalized entropy value H(ˆ pi), we obtain the weight w of the sample in as:

3.5 Training and Inference

Training. Based on the pseudo-labels, we generate latent patterns for input data and use Lsmooth weighted according to uncertainty in order to encourage to generate as more relevant patterns as possible for pseudo-labeled normal data and as more inrelevant patterns as possible for pseudo-labeled abnoraml data. After updating the parameters of εθ, we generate new pseudo-labels for this batch of data. Before starting a new round of training, the pseudo-labels need to be refined and uncertainty estimated.

Inference. For the input sequence of unlabeled frames, we generate M latent patterns for each frame as trained and count their distance D from the ground truth. We consider the mean as aggregation statistics for the distance D between the generated latent patterns and the ground truth. Latent patterns for normal events are more correlated, and the distribution of latent patterns for anomalous events is more discrete.

4 Experiments

4.1 Implementation Details

Datasets

Six real-world benchmark datasets are used to evaluate our approach, including the ShanghaiTech dataset [Luo et al., 2017], CUHK Avenue dataset [Lu et al., 2013], UCSD dataset [Mahadevan et al., 2010], Subway dataset [Adam et al., 2008], UMN dataset [Mehran et al., 2009], and UCFCrime dataset [Sultani et al., 2018]. These datasets correspond to different scenarios, including scenarios such as schools, pedestrian streets, subway entrances and exits, and city streets. In particular, the UCF-Crime dataset contains some illegal activities that harm sustainable cities.

Note that the training set in these datasets contains only normal events, with abnormal activity occurring only in the test set. To perform UVAD, we employ two types of UVAD settings based on previous UVAD work: (i) Partial mode [Giorno et al., 2016; Liu et al., 2018b; Yu et al., 2022] : only the original test set of the dataset is used for learning, while the original training set is discarded. (ii)Mergedmode [Pang et al., 2020; Zaheer et al., 2022]: the original training and test sets are merged into one unlabeled set for learning. For both modes, labels are strictly not used in learning. Performance evaluation is performed only on the original test set to allow comparison with existing VAD methods.

Evaluation Metrics

Following the existing method [Pang et al., 2020; Lin et al., 2022; Zaheer et al., 2022], the area under the ROC curve (AUC) is used for evaluation and comparison. The AUC is calculated based on the frame-level annotations of the test videos in each dataset.

Training Details

We resize each video frame to 256 × 256 and normalize it to the range [- 1,1]. We set the numbers of generated latent patterns M to 10. Both the generator network and the condition extractor network are trained with AdamW-based stochastic gradient descent method. The learning rate, and weight decay were set to 0.0002, and 0.0001, respectively. Training epochs are set to 60; 60; 60; 10; 25; 30; 15 on Ped1, Ped2, Avenue, ShanghaiTech, Subway, UMN and UCF-Crime, respectively. We use pre-trained Resnext101 [Xie et al., 2017] as the extractor for motion features in experiments.

4.2 Results

Quantitative Results

We compare our method with state-of-the-art OCC supervised VAD methods and UVAD methods on six benchmark datasets. For a fair comparison, we provide three versions of DiffVAD under different training settings. DiffVADOCC represents the OCC supervised version trained with normal data only. DiffVAD- and DiffVAD+ represent the unsupervised versions in partial mode and merged mode, respectively. The results are summarized in Table 1. Generally, our proposed method is significantly better than all previous UVAD methods, and even higher than most OCC supervised methods. This demonstrates the proposed method is perfectly suited to the need for anomaly detection in real-world scenarios. In the unsupervised setting, DiffVAD- outperforms DiffVAD+ on all datasets. This is because partial mode contains less data, i.e., fewer unseen events, than merged mode.

Qualitative Results

The examples in Figure 4 show anomaly curves of the testing video from the ShanghaiTech dataset compared among MemAE [Gong et al., 2019], PMem [Park et al., 2020], GCL [Zaheer et al., 2022] and DiffVAD. MemAE and PMem are both OCC supervised VAD methods based on memory modules, while GCL is an unsupervised VAD method. All three methods perform single-pattern reconstruction of the input data. The anomaly curve shows the anomaly scores of all frames in the video in turn, allowing a more intuitive comparison of the performance of different methods. It can be seen that DiffVAD performs significantly better than the other methods. The anomaly scores of DiffVAD are lower and more stable on normal segments. On abnormal segments, the anomaly score curves of the other three methods have obvious errors whilst DiffVAD has longer abnormal durations and more accurate anomaly scores. The curves shown in the Figure 4 are consistent with the results in Table 1.

4.3 Ablation Study and Discussion

As shown in Table 2, to validate each component in the proposed DiffVAD effectiveness in detail, the ablation study is conducted on the ShanghaiTech dataset. Not using any components represents a binary classification of abnormal and normal. It can be seen that both training the model through the classification task and performing single pattern reconstruction for training perform poorly. A significant improvement in AUC scores was obtained by designing multiple latent pattern generation and diffusion models to form a latent pattern learning framework.

Latent pattern learning improves 16.7% and 5.7% compared to classification and single pattern reconstruction, respectively. The effectiveness of the pseudo-labels refinement is demonstrated by the 3.8% increase in score after introducing it during training. This is explained by the fact that more reliable pseudo-labels help the model to learn the feature representations more correctly and also help to produce the correct training loss weights. The results in Table 2 illustrate that the components of our design fit each other and justify our design's rationality.

In addition to the ablation experiments with diffusion models as the generator, we also experiment with VAE [Kim, 2022] and GANs [Kaneko and Harada, 2020] as generators in the same setup. As shown in Table 2, both VAEs and GANs performed less than 80% as generators, while the diffusion model achieved a score of 81.9%. The superior performance of the diffusion-based generator demonstrates the suitability of the diffusion model for the UVAD task. It is notable that the latent pattern learning and pseudo-labels refinement also improve the score by about 14% and 3% when other models as the generator.

Evaluation of Latent Patterns

Unlike previous work [Gong et al., 2019; Park et al., 2020; Liu et al., 2021] that performs single pattern reconstruction, our proposed DiffVAD generates multiple latent patterns of the same event. In the case of normality, different patterns of the same event are varied but correlated, i.e., they are biased towards the ground truth of what actually happens. In the case of abnormality, different patterns of the same event are diverse but not correlated. Figure 5 shows the normal frames, the abnormal frames in the same scene and the visualization of normalcy maps of the multiple latent patterns on UCSD Ped2, Avenue and ShanghaiTech. We can see that (1) the normalcy maps of the multiple latent patterns is more similar to the normal frame than abnormal frames, indicating that the latent patterns of normal events is more biased toward the ground truth of what really happens. (2) There is no anomalous events in the normalcy map, such as vehicles, jumping and running. Anomalous features in anomalous frames are not reconstructed due to the smaller similarity. Therefore, even in the same scene, the latent pattern of normal events would not be similar to the anomalous one.

Analysis of Pseudo-Labels Refinement Design

The pseudo-label refinement strategy we devise not only produces finer pseudo-labels, but also weights the training loss according to its uncertainty to avoid noise in the early stages of training. The results in Table 2 demonstrate the benefits of pseudo-labels refinement designed in this way. The proposed design functions similarly to the teacher-student model. Differently, we do not need the teacher model only to aggregate the information of the nearest neighbors in the feature space. Meanwhile, it is also the refinement design based on aggregated nearest neighbors that allows us to train with reliable pseudo-labels without the need for auxiliary networks or deeper models, keeping the lightweight feature. DiffVAD meets the need for fast detection of real-world scenarios.

5 Conclusion

UVAD technology is an important safeguard for sustainable cities. Existing UVAD methods are limited by the openness problem to learn sufficient patterns, we address this challenge by introducing diffusion models into the video anomaly detection task. Specifically, we propose a latent patternbased learning framework that learns unseen event representations in training by generating multiple latent patterns of the same event, with fast inference speeed. The design of pseudo-labels refinement and uncertainty estimation minimize the noise problem. The comprehensive experiments on six benchmarks and exhaustive ablation studies validate the effectiveness of proposed framework.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grants ( 62201072, 62101064, 62171057, U23B2001, 62001054, 62071067), the Ministry of Education and China Mobile Joint Fund (MCM20200202, MCM20180101), and the BUPT Excellent Ph.D. Students Foundation (Grant CX20241007).