Neural Contextual Anomaly Detection for Time Series

1 Introduction

Anomaly Detection (AD) for real-valued time series data has many practical applications, such as monitoring machinery for faults, finding irregular behavior in IoT sensor data, improving the availability of computer applications and (cloud) infrastructure, and monitoring patients vital signs, among many others. Since Shewhart [1931] pioneering work on statistical process control, statistical techniques for monitoring and detecting abnormal behavior have been refined, and deployed in countless highly impactful applications.

Recently, deep learning techniques have been successfully applied to various anomaly detection problems [Ruff et al., 2021]. For time series, these methods have demonstrated great performance for large-scale monitoring problems [Ren et al., 2019; Gao et al., 2020; Ayed et al., 2020]..

Classically, anomaly detection on time series is cast as an unsupervised learning problem, where the training data contains both normal and anomalous instances, but without knowing which is which. However, in many practical applications, a fully unsupervised approach can leave valuable information unutilized, as it is often possible to obtain (small amounts of) labeled anomalous instances, or to characterize the relevant anomalies in some general way. Ideally, an effective method for anomaly detection employs a semi-supervised approach, allowing to utilize information about known anomalous patterns or out-of-distribution observations, if any of these are available.

∗These authors contributed equally.

In this work 1 , we introduce Neural Contextual Anomaly Detection (NCAD), a framework for anomaly detection on time series that can scale seamlessly from the unsupervised to supervised setting, allowing to incorporate additional information, both through labeled examples and through known anomalous patterns. Our approach is based on breaking each time series into overlapping, fixed-size windows. Each window is further divided into two parts: a context window and a suspect window (see fig. 1), which are mapped into neural representations (embedding) using Temporal Convolutional Networks (TCNs). Our aim is to detect anomalies in the suspect window. Anomalies are identified in the space of learned latent representations, building on the intuition that anomalies create a substantial perturbation on the representation of the time series, resulting in the embeddings becoming distant.

Time series anomalies are inherently contextual. We account for this in our methodology by extending the Hypersphere Classifier (HSC) [Ruff et al., 2020a] loss, originally developed for computer vision, to a contextual hypersphere loss, which dynamically adapts the hypersphere's center to the context's representation. Further, we use data augmentation techniques to facilitate learning of the boundary between the normal and anomalous classes. We employ a variant of Outlier Exposure (OE) [Hendrycks et al., 2019] to create contextual anomalies, combined with simple injected anomalies.

In summary, we make the following contributions:

1. Propose a simple yet effective framework for time series anomaly detection that achieves state-of-the-art performance across well-known benchmark datasets, covering univariate and multivariate time series, and across the unsupervised, semi-supervised, and fully-supervised settings (open-source code of NCAD is available 2 );

1 Extended version: https://doi.org/10.48550/arXiv.2107.07702

2 https://github.com/awslabs/gluon-ts/tree/master/src/gluonts/ nursery/ncad

1. Build on related work on deep anomaly detection using the hypersphere classifier [Ruff et al., 2020a] and expand it to introduce contextual hypersphere detection.
2. Adapt the Outlier Exposure [Hendrycks et al., 2019] and Mixup [Zhang et al., 2018] methods to the particular case of anomaly detection for time series.

2 RelatedWork and Background

Anomaly Detection is an important problem with many applications and has consequently been widely studied. We refer the reader to one of the recent reviews in the topic for a general overview of methods [Ruff et al., 2021].

We are interested in anomaly detection for time series. This is a problem typically framed in an unsupervised way. A traditional approach is to use a predictive model, estimating the distribution (or confidence bands) of future values conditioned on historical observations, and mark observations as anomalous if they are considered unlikely under the model. [Shipmon et al., 2017] use deep (recurrent) neural networks to parametrize a Gaussian distribution and use the tail probability to detect outliers. [Aubet et al., 2021] propose to use probabilistic masking to prevent anomalies in the training data to affect the model. [Siffer et al., 2017] propose SPOT and DSPOT, which use extreme value theory to model the tail of the distributions. [Ehrlich et al., 2021] expanded their framework using a robust deep forecaster.

Effective ideas for deep anomaly detection that deviate from the predictive approach have been successfully imported to the time series domain from other fields. For reconstruction based methods, e.g. with Variational AutoEncoders (VAEs) [Xu et al., 2018; Park et al., 2018; Su et al., 2019], or density based methods, e.g. with Generative Adversarial Networks (GANs) [Schlegl et al., 2017; Li et al., 2019].

Compression-based approaches have become very popular in image anomaly detection. The working principle is similar to the one-class classification used in the support vector data description method [Tax and Duin, 2004]: instances are mapped to latent representations which are pulled together during training, forming a sphere in the latent space; instances that are distant from the center are considered anomalous.

[Ruff et al., 2018; Ruff et al., 2020b] build on this idea to learn a neural mapping ϕ(·) : RD → RE , such that the representations of nominal points concentrate around a (fixed) center c, while anomalous points are mapped away from that center. In the unsupervised case, DeepSVDD [Ruff et al., 2018] achieves this by minimizing the Euclidean distance ∑i ||ϕ(wi) - c||2 , subject to a suitable regularization of the mapping and assuming that anomalies are rare. THOC [Shen et al., 2020] applies this principle to the context of time series, by extending the model to consider a multiple spheres.

[Ruff et al., 2020a] propose the Hypersphere Classifier (HSC), improving on DeepSVDD by training the network using the standard Binary Cross-Entropy (BCE) loss, this way extending the approach to the (semi-)supervised setting. With this method, they can rely on labeled examples to regularize the training and do not have to resort to limiting the network. In particular, the HSC loss is given by setting the pseudo-probability of an anomalous instance (y = 1) as p = 1- ℓ(ϕ(wi)), i.e.

where ℓ : RE → [0, 1] maps the representation to a probabilistic prediction. Choosing ℓ(z) = exp(-||z||2 ), leads to a spherical decision boundary in representation space.

Current work on semi-supervised anomaly detection indicates that including even only few labeled anomalies can already yield remarkable performance improvements on complex data. Notably, [Hendrycks et al., 2019] improve detection by incorporating large amounts of out-of-distribution examples from auxiliary datasets during training. For time series data, however, artificial anomalies and related data augmentation techniques have not been studied extensively. [Smolyakov et al., 2019] used artificial anomalies to select thresholds in ensembles of anomaly detection models. Most closely related to our approach, SR-CNN [Ren et al., 2019] trains a supervised CNN on top of an unsupervised anomaly detection model (SR), by using labels from injected single point outliers.

Fully supervised methods are not as widely studied because labeling all the anomalies is too expensive and unreliable in many applications. An exception is the work by [Liu et al., 2015] who propose a system to continuously collect anomaly labels and to iteratively re-train and deploy a supervised random forest model. The U-NET-DEWA approach of [Gao et al., 2020] relies on training a supervised model, relying on data augmentations that preserve the anomaly labels to increase the training set size.

3 Neural Contextual Anomaly Detection

We combine a window-based anomaly detection approach with a flexible training paradigm and effective heuristics for data augmentation to produce a state-of-the-art system for anomaly detection.

We consider the following general time series anomaly detection problem: We are given a collection of N discretetime time series X = {x(i) 1:Ti }i=1,...,N where for time series i and time step t = 1, . . . , Ti we have an observation vector x(i) t ∈ RD . We further assume that we are given a corresponding, set of partial anomaly labels Y = {y(i) 1:Ti }i=1,...,N with y(i) t ∈ {0, 1, ?}, indicating whether the observation x(i) t is normal (0), anomalous (1), or unlabeled (?).

The goal is to predict anomaly labels ˆ y1:T , with yt ∈ {0, 1} given a time series x1:T . Instead of predicting the binary labels directly, we predict a positive anomaly score for each time step, which can subsequently be thresholded to obtain anomaly labels satisfying a desired precision/recall trade-off.

3.1 Windows-Based Representations

Similar to other work on time series AD (e.g. [Ren et al., 2019]), we convert the time series problem to a vector problem by splitting each time series into a sequence of fixed-size windows of length L. A window is simply a segment of sequential values from a single time series. Let WL(X) denote the set of all possible windows of length L, allowing overlapping, that can be generated from the data X .

On a given window w ∈ WL(X), we identify a partition of two subsequent segments, w = (w(c) , w(s) ), with lengths C and S (typically C ≫ S), which we call the context window and the suspect window, respectively (see fig. 1 for an illustration).

Our goal is to detect anomalies in the suspect window relative to the local context provided by the context window. We keep a single label y for the windows w, such that y = 1 if any timestep in its suspect segment contains an anomaly, and y = 0 otherwise. In this way, the datasets for training and evaluation are formed by pairs of windows and their labels: D = {(wi, yi)}w∈WL(X )

For our model, we use a neural network encoder ϕ : RT×D → RE to map D-dimensional time series of length T (possibly variable) into fixed-size vector representations. We use a CNN with exponentially dilated causal convolutions [van den Oord et al., 2016], and in particular, the TCN architecture [Franceschi et al., 2019] with adaptive maxpooling along the time dimension.

Let us denote as z = ϕ(w) and z(c) = ϕ(w(c) ) the vector representations of a window w ∈ WL(X) and its context segment w(c) . Intuitively, we want to train ϕ(·) in such a way that representations z and z(c) are pulled together if no anomaly is present in the suspect window, and pushed apart otherwise.

3.2 Contextual Hypersphere Loss

We propose a contrastive loss function, which can be interpreted as a contextual version of the Hypersphere Classifier loss (eq. (1)).

Consider a distance-like function dist(·, ·) : RE ×RE → R+ measuring similarity between two vector representations; and consider a scoring function ℓ : R → [0, 1] mapping distances to pseudo-probabilistic scores in the unit interval.

Our contextual hyperphere loss is defined as:

Note that this is the HSC loss, with the difference that the center of the hypersphere is chosen dynamically for each instance as the representation of the context.

In our experiments, we use the Euclidean distance dist(x, z) = ∥x- z∥2 and a radial basis function ℓ(z) := exp(-z2 ), to create a spherical decision boundary, resulting in the loss function

3.3 Data Augmentation

We introduce a collection of data augmentation methods that inject synthetic anomalies. These data augmentation methods explicitly do not attempt to characterise the full data distribution of anomalies, which would be infeasible. Rather, we combine effective generic heuristics that work well for detecting common types of out-of-distribution examples.

Contextual Outlier Exposure (COE) Motivated by the success of Outlier Exposure for out-of-distribution detection [Hendrycks et al., 2019], we propose a simple taskagnostic method to create contextual out-of-distribution examples. Given a time series window w = (w(c) ,w(s) ), we induce anomalies into the suspect segment, w(s) , by replacing a chunk of its values with values taken from another time series. The replaced values in w(s) will most likely break the temporal relation with their neighboring context, therefore creating an out of distribution example. In our implementation, we apply COE at training time by selecting random examples in a minibatch and permuting a random length of their suspect windows. In multivariate time series, as anomalies do not have to happen in all dimensions, we randomly select a subset of the dimensions in which the windows are swapped.

Anomaly Injection We propose to inject Point Outliers (po) in the time series using a simple method: at a set of randomly selected time points we add (or subtract) a spike to the time series. The spike is proportional to the inter-quartile range of the points surrounding the spike location. Like for COE, in multivariate time series we simply select a random subset of dimensions on which we add the spike. These simple point outliers serve the same purpose as COE: create clear labeled abnormal points to help the learning of the hypersphere. In addition to these, in some practical applications, it is possible to identify general characteristics of anomalies that should be detected. Some widely-known anomalous patterns include: sudden changes in the location or scale of the series (change-points); interruption of seasonality, etc. We have used this approach in our practical application and the domain knowledge allowed to improve the detection performance. As they require and domain knowledge it would be unfair to compare our method when incorporating these; therefore, in the results table we only use the point outliers described above.

WindowMixup If we do not have access to training labels and know little about the relevant anomalies, we mainly rely on COE and po, which may result in significantly missmatch between injected and true anomalies. To improve generalization of our model in this case, we propose to create linear combinations of training examples and their labels inspired by the MIXUP procedure [Zhang et al., 2018]. The MIXUP data augmentation technique was proposed in the context of computer vision and creates more variety in training examples, but more importantly, the soft labels result in smoother decision functions that generalize better. MIXUP is suited for time series applications: convex combinations of time series most often result in realistic and plausible new time series. We show that MIXUP can improve generalization of our model even in cases with a large mismatch between injected and true anomalies.

We include illustrations of these techniques in the extended version of this article 1 .

3.4 NCAD Training

We combine the elements described above and train the encoder by minimizing the contextual hypersphere loss using Stochastic Gradient Descent (SGD).

We start with a raw dataset of time series and their labels (X, Y) = {(x(i) 1:Ti , y(i) 1:Ti )}i=1,...,N . A first augmentation 3 is applied to the series before spliting into windows, yielding (X ′ , Y ′ ) = anomalize((X, Y)). We then define the training dataset by splitting the original and augmented data into windows Daug = {(wi, yi)}wi∈WL(X∪X ′ ) (see section 3.1).

Mini-batches B = {(wi, yi)}i=1,...,batch size are formed by taking random subsets of windows from the training dataset:

3 We inject po during the first augmentation.

B ⊂ Daug . We apply a second stage of data augmentation 4 here by introducing synthetic anomalies on the batches, yielding B′ = anomalize(B).

The score function minimized during training sums the contextual loss function L from eq. (2) over examples in the original and augmented batches:

where θ parametrizes the neural encoder ϕθ(·).

Rolling predictions Once the model is trained, we can apply it to the raw time series in a rolling fashion, possibly defining a stride parameter. Each time step may be part of multiple windows at test time (if the stride is smaller than the suspect window length), obtaining multiple anomaly scores, which can be aggregated by averaging. Such rolling scheme have the advantage of generating anomaly scores with lower latency.

4 Experiments

In this section, we compare the performance of our approach with alternative methods on public benchmark datasets, and exploring the model behavior under different data settings, demonstrating its effectiveness from the unsupervised to the supervised setting. Further details on the experiments are included in the extended version of the article 1 .

4.1 Benchmark Datasets

We benchmark our method to others on five datasets, spanning the univariate, multi-variate, supervised and unsupervised settings.

For the multivariate setting, we use: Soil Moisture Active Passive satellite (SMAP) and Mars Science Laboratory rover (MSL), two datasets published by NASA [Hundman et al., 2018], with 55 and 27 series respectively (lengths of the time series vary from 300 to 8500 observations); and Server Machine Dataset (SMD), a 5 weeks long dataset with 28 38-dimensional time series each collected from a different machine in large internet companies [Su et al., 2019]. These three datasets each have a pre-defined train/test split, where anomalies in the test set are labeled, while the training set contains unlabeled anomalies.

For the univariate setting, we use: YAHOO, a dataset by [Yahoo! Labs, 2015] consisting of 367 real and synthetic time series. And KPI, univariate dataset released in the AIOPS data competition by [Tsinghua Netman Lab, 2018]. It consists of KPI curves from different internet companies in 1 minute interval. 5 For both, following [Ren et al., 2019], we use the last 50% of the time points of each of the time series as test set and split the rest in 30% training and 20% validation set.

4 We use COE and MIXUP during the batch augmentation.

5 While we share many of the concerns expressed by [Wu and Keogh, 2020] about the lack of quality benchmark datasets for time series anomaly detection, we use these commonly-used benchmark datasets here for lack of better alternatives and to enable direct comparison of our approach to competing methods.

4.2 Evaluation Setup

Measuring the performance of time series anomaly detection methods in a universal way is challenging, as different applications often require different trade-offs between sensitivity, specificity, and temporal localization. To account for this, various measures that improve upon simple pointwise classification metrics have been proposed, e.g. the flexible segment-based score proposed by [Tatbul et al., 2018] or the score used in the Numenta anomaly benchmark [Lavin and Ahmad, 2015]. To make our results directly comparable, we follow the procedure proposed by [Xu et al., 2018] (and subsequently used in other work [Su et al., 2019; Ren et al., 2019; Shen et al., 2020]), which offers a practical compromise: point-wise scores are used, but the predicted labels are expanded to mark an entire true anomalous segment as detected correctly if at least one time point was detected by the model. 6 Shortcomings of this evaluation protocol have been uncovered by [Kim et al., 2022]: if the dataset has many long segments labeled as anomalous (as e.g. in the SWaT and SMAP datasets), then the correction allows high F1 scores to be achieved even based on weak (or even random) anomaly scores. However, as they also point out, this is less of a problem when anomalies are very short as in KPI and YAHOO. To emphasize this point we include the F1 scores for the baseline that they proposed, which performs significantly worse on the data sets with short anomaly segments. While in practice a detection threshold would have to be chosen on a validation set or through user input, we align our experimental protocol with this body of prior work and report optimistic F1 scores computed by choosing the best threshold on the test set. For each dataset, a single best threshold is chosen and used on all the time series of the test set. 7

6 We use the implementation by [Su et al., 2019].

Hyperparameters were chosen using the validation set for YAHOO and KPI, and a standard setting is inferred for the other datasets (see extended article for details 1 ). We use short suspect windows because we are interested in the harder problem of streaming anomaly detection where it is important to detect anomalous points early. Further, we run the model 10 times on each of the benchmark datasets and report mean and standard deviation. We provide scripts to reproduce the results on the benchmark datasets shown below.

Comparison partners Beyond several deep state of the art methods, we compare to a set of classical shallow anomaly detection methods on fixed time series windows: OC-SVM [Sch¨olkopf et al., 1999], Isolation Forest [Liu et al., 2008], and LOF [Breunig et al., 2000]. In addition, we compare to a non-deep probabilistic method using Gaussian processes [Bock et al., 2022]. We also include the baseline 1 of [Kim et al., 2022], for which the anomaly score of each point is sampled from a uniform distribution.

4.3 Benchmark Results

Table 1 shows the performance of our NCAD approach compared to the state-of-the-art methods proposed for the different datasets and settings, as well as a set of non-time series baselines.

The univariate datasets contain labels for anomalies both on the training and the test set, we evaluate our method on them both in the supervised setting ((sup.)) and the unsupervised setting ((un.)). Our approach significantly outperforms competing approaches on YAHOO, performs similarly to the best unsupervised approach on KPI, and better than the best supervised approach.

7 Note that this is done for all methods, keeping the relative performance of different methods comparable, even if the absolute scores are higher than what can be expected in a practical setting.

It is important to note that while other methods are either designed for the supervised or unsupervised setting, our method can be used seamlessly in both settings. For NCAD, in the supervised setting the training labels are used in the loss, whereas in the unsupervised setting only the labels of the simple synthetic anomalies are used for training.

For KPI (sup.) we only compare to a single baseline because the other methods are not developed for the supervised context. The YAHOO-supervised experiments are included in table 2, our approach outperforms the state-of-the-art [Gao et al., 2020] significantly with 79% point-wise F1 score versus 69.3% F1 score for their approach.

On the multivariate datasets, all benchmark methods are designed for unsupervised anomaly detection, and none of the dataset contains training labels. Our method outperforms the state of the art by a reasonable margin on MSL. On SMAP our average score is slightly lower than the best score. On SMD, OmniAnomaly and USAD outperform our approach. We note that these methods train one model for each of the 28 time series, while we train a single global model, which is more representative of a real world scenario.

4.4 Ablation Studies

To better understand the advantage brought by each of the components of our method, we perform an ablation study on the SMAP and MSL datasets, shown in table 3. The first row corresponds to the entire NCAD framework, using contextual hypersphere loss and data augmentation. The rows labeled "- contextual . . . " do not use the contextual hypersphere described in section 3.2, but instead a model trained using the original hypersphere classifier loss on the whole-window representation ϕ(w). First, we see that the contextual hypersphere has a significant impact on performance when no data augmentation is used, increasing performance by 11.81% F1 on SMAP and 43.44% F1 on MSL. By only removing the contextual hypersphere but keeping the data augmentation methods ("- contextual" row), we observe a small performance deterioration with respect to the full model, 1.98% F1 on SMAP and 1.17% F1 on MSL.

We also see that each of the data augmentation techniques improves the performance further.

Further ablation results on the supervised Yahoo dataset can are shown in table 2. Using only the true labels but no data augmentation (NCAD SUPERVISED), our approach significantly outperforms U-NET-RAW, and performs on-par with U-NET-DE, without relying on time series decomposition and using an arguably much simpler architecture.

When we use the po data augmentation, our approach outperforms the full U-NET-DEWA by a large margin, hinting at the possibility that addressing the class imbalance problem by creating artificial anomalies is more effective than using their strategy of loss weighting while keeping the labels intact.

In the supervised setting, injecting the generic COE anomalies (either individually or in combination with po) hurts performance, presumably by steering the model away from the specific kind of anomalies that are labeled as anomalous in this data set. On the other hand, adding MIXUP generally improves performance. The contrastive loss is crucial for good performance, as shown by the rows labeled - CONTRASTIVE, where it is replaced with a standard softmax classifier.

4.5 Scaling FromUnsupervised to Supervised

To investigate how the performance of our approach changes as we scale from unsupervised, to semi-supervised, to fully supervised, we measure the performance of our approach as a function of the amount of ground truth labels on the YAHOO dataset, shown in fig. 2.

Firstly, we observe that the performance increases steadily with the amount of true anomaly labels, as desired. Secondly, by using synthetic anomalies (either po or COE), we can significantly boost the performance in the regime when no or only few labels are available. Finally, by using an injection technique that is well-aligned with the desired type of anomalies (po in this case, as YAHOO contains a large number of single-point outliers), one can significantly improve performance over relying solely on the labeled data, this is explained by the very high class imbalance in anomaly detection. The flipside is, of course, that injecting anomalies that may be significantly different from the desired anomalies (COE in this case) can ultimately hurt when enough labeled examples are available.

4.6 Using Specialized Anomaly InjectionMethods

In many practical applications one may have access to some domain knowledge about the type of anomalies that are to be detected, our method allows to incorporate this easily. While in all our benchmarks we rely on completely generic anomalies for injection (COE and po), a by-product of our methodology is that the model can be guided towards detecting the desired class of anomalies by designing anomaly injection methods that mimic the true anomalies. Designing such methods is often simple compared to finding enough examples of true anomalies as they are rare.

Table 4 demonstrates the effectiveness of this approach: The first dimension of the SMAP dataset contains slow slopes that are labeled as anomalous in the dataset. These are harder to detect for our model when only using COE and po because these cannot create similar behavior. We can design a simple anomaly injection that injects slopes to randomly selected regions and labels them as anomalous. Training NCAD with these slopes gives a model that achieves a much better score.

This approach can be effective in applications where anomalies are subtle and closer to the normal data, and where some prior knowledge is available about the kind of anomalies that are to be detected. However one may not have this prior knowledge or the resources required to create these injections. This is a limitation of this technique which prevents it from being generally applicable. This is the reason why we did not use in for the comparison to the other methods.

4.7 Generalization From Injected Anomalies

Artificial anomalies will always differ from the true anomalies to some extent, be it the ones created by COE, po, or more complex methods. This requires the model to generalize from imperfect training examples to true anomalies. By design, the hypersphere formulation can help to bridge this generalization gap, and we use MIXUP further improve the generalization capabilities of the model.

Figure 3 shows the results of an experiment exploring one aspect of this generalization ability for NCAD. The model is trained with injected single-point outliers, and we measure the detection performance for anomalies of longer width. For this experiment we use a synthetic base data set containing simple sinusoid time series with Gaussian noise. We create multiple datasets from this base dataset adding true anomalies of varying width by convolving spike anomalies with Gaussian filters of different widths. For training, regardless of the shape of the true anomalies, we use po and train models using different MIXUP rates, i.e. fraction of training examples with MIXUP applied. We observe that MIXUP helps the model to generalize in this setting: the higher the MIXUP rate, the better the model generalizes to anomalies that differ from the injected examples, achieving higher F1 scores.

5 Conclusion

We present NCAD, a methodology for anomaly detection in time series that achieves state-of-the-art performance in a broad range of settings, including both the univariate and multivariate cases, as well as across the unsupervised, semisupervised, and supervised anomaly detection regimes. We demonstrate that combining expressive neural representation for time series with data augmentation techniques can outperform traditional approaches such as predictive models or methods based on reconstruction error.