RTdetector: Deep Transformer Networks for Time Series Anomaly Detection Based on Reconstruction Trend

1 Introduction

Multivariate time series anomaly detection technology is extensively employed across various domains, including industrial equipment monitoring [Xie et al., 2024], vehicle diagnostics [Wei et al., 2024], network system surveillance [Lim et al., 2024], and financial risk assessment [Jiang et al., 2024]. Due to the development of sensor technology, various sensors are widely used to record industrial process variables. These sensors generate thousands of interconnected multivariate time series datasets within these systems. It is crucial to accurately diagnose the real-time status of equipment from these extensive sensor data.

∗Corresponding authors

Currently, most anomaly detection methods rely on reconstruction-based approaches. While these methods can effectively identify pattern anomalies, they may overfit to abnormal patterns, making it difficult to detect amplitude anomalies [Schmidl et al., 2022]. To solve this problem, some studies [Kim et al., 2021] prevent incorrect fitting of abnormal data by allowing the model to learn the changing trend of the data. Despite significant progress, these approaches are typically based on idealized datasets for learning model variation trends. However, real-world data are often nonstationary, meaning that the statistical properties and joint distributions of time series data change over time. These methods fail to account for the dynamic nature of data distribution, leading to an inability of the models to effectively learn the correct trend for data reconstruction. Figure 1 shows that the existing methods have incorrectly learned the changing trend of the data, resulting in the reconstruction of anomalies that should not exist at normal times with the wrong changing trend.

There are two key challenges in enabling a model to accurately learn the true reconstruction trends of data. Challenge

1: How to restore the original distributional variations of time series data during reconstruction. In order to allow the model to learn a more stable data distribution, most algorithms [Tuli et al., 2022; Yang et al., 2023] preprocess the time series by stabilizing it. While this approach can enhance the model's predictability, it inevitably leads to the loss of information regarding the original distributional changes, thereby reducing the accuracy of anomaly detection. Challenge 2: The model's inherent architecture does not capture the true variation trends of the time series. Previous methods often rely on the concurrent input of both a window sequence and a complete sequence [Wang et al., 2024] to help the model learn the trend of data variations. Although this approach allows the model to learn the changing trend of the data to a certain extents, the model itself is not designed specifically to recover the reconstruction trend of time series data. As a result, it fails to effectively learn the true changing characteristics of the reconstruction trend of the data.

In response to the above challenges, we propose a reconstruction trend transformer time series anomaly detector (RTdetector). This model captures global features by processing both the window sequence and complete sequence within the Transformer framework and learns the global trend information of intrinsic time series through RT-Attention. The global trend information of the reconstructed data is restored by reconstruction trend enhancement (RTE) of the model's decoder output. Additionally, the difference between the reconstructed data and the real data is amplified through the attention score, thereby enhancing the detection of anomalies that are difficult to distinguish. The contribution of our RTdetector are summarised as follows:

• We propose a global anomaly attention mechanism based on reconstruction trend to capture global information within a window by simultaneously processing the window sequence and the complete sequence. The global feature information of the intrinsic time series is restored through RT-Attention.
• We design a self-conditioning transformer based on RTE, which recovers the global trend features of the input data in the output by aggregating them. Additionally, it uses focus scores to amplify reconstruction errors for better predictability.
• Extensive experiments conducted on four public datasets demonstrate that RTdetector achieves state-of-the-art performance in detecting anomalies in multivariate time series.

2 RelatedWork

In this section, we provide a concise yet comprehensive review of the current landscape in deep models for time series anomaly detection and stationarization for time series detection.

2.1 DeepModels for Time Series Anomaly Detection

In recent years, the use of carefully designed deep learning structures to achieve high-precision multivariate time series detection has garnered significant attention among researchers. A method combining Spectral Residual and CNN was proposed to detect timing anomalies in service systems [Ren et al., 2019]. In the real world, it is difficult for multivariate time series data to have all label information, so unsupervised anomaly diagnosis methods have been widely studied [Deng and Hooi, 2021; Li et al., 2022]. The multivariate spectrum signal frequency consistency is employed for unsupervised anomaly detection [Abdulaal et al., 2021]. For time series anomaly detection, an unsupervised approach utilizing LSTM networks is proposed, wherein anomaly identification is achieved through architectural optimization of LSTM models in conjunction with support vector machine algorithms [Ergen and Kozat, 2019]. USAD [Audibert et al., 2020] employs an autoencoder based on adversarial training, ensuring efficient model training. Among unsupervised learning methods, reconstruction-based anomaly detection techniques have been widely studied due to their effectiveness in solving highdimensional and nonlinear data problems. However, they often tend to overfit to abnormal patterns, which can lead to an inability to accurately diagnose anomalies.

2.2 Anomaly Detection in Time Series Based on Reconstruction

Although reconstruction-based anomaly detection methods can effectively identify pattern anomalies, they are likely to overfit abnormal data, reconstructing the same abnormal data, which makes it difficult to detect amplitude anomalies. To solve these problems, some studies have tried to amplify the difference between abnormal data and reconstructed data to enable the model to better diagnose amplitude anomalies [Schlegl et al., 2017]. To further address the issue of overfitting to abnormal data, numerous studies have aimed to prevent the reconstruction of erroneous fitting results by enabling models to learn the underlying trends of the original data. RevIN [Kim et al., 2021] recovers the statistics of time series data through reversible instance normalization. DCdetector uses a single-scale architecture to extract local features and global correlations to effectively capture the temporal information of long-term series [Yang et al., 2023]. D3R supplements the global information of data through decomposition and reconstruction [Wang et al., 2024]. While these methods have mitigated the problem of overfitting to abnormal data by capturing global trend information to some extent, most models themselves do not learn the reconstruction trend of the data. Instead, they rely on preprocessing the data outside the model, leading to insufficient capability in capturing the reconstruction trend. Consequently, the anomaly detection performance remains unsatisfactory.

3 Methodology

For multivariate time series data with timestamps of length T :

where each data point xt ∈ Rd is collected at timestamp t from different sensors. Here, d is the data dimension, representing the number of sensors. The anomaly detection problem can be defined as: given training data X , for an unknown data ˆ X, where the data length is ˆ T as the test sequence with the same modality as the training data. We need to predict Y = {y1, . . . , y ˆ T }, where yt ∈ {0, 1} represents whether the point is anomalous (1 represents an anomaly and 0 represents a normal point).

3.1 Overall Architecture

Transformer have been widely used in time series data anomaly detection [Yan et al., 2024]. However, as mentioned above, it is often difficult to capture time series relationships [Zeng et al., 2023] when using Transformer for anomaly detection, and it often lacks the capability to extract global reconstruction information. Therefore, we designed the architecture shown in Figure 2.

We use the complete sequence c to capture the trend of global time series changes, and the window sequence w to capture the relationship between adjacent time series. The masked multi head attention is used to mask the data of subsequent timestamps to prevent the decoder trainer from obtaining data of future timestamps. We designed two decoders to adjust the training model by focusing on the Focus score, which quantifies the difference between the generated and original data, thereby enhancing sensitivity to abnormal intervals. The RTE module is used to restore the information lost by stationarization, which will be discussed in detail in Section 3.2. We designed the RT-Attention module to address the inability of traditional attention mechanisms to capture data reconstruction trends, which will be discussed in detail in Section 3.3. Finally, we use the outputs of the two decoders concurrently to determine the presence of an anomaly.

3.2 Self-conditioning Transformer Based on RTE

In order to make the data distribution of the model more stable and reduce the deviation trend of the data, we stabilize the input time series data X = (x1, x2, . . . , xT ) to obtain the stationarized data X ′ = [x′1 , x′2 , . . . , x′T ]. The stationarization formula is shown as follow:

where µx, σx ∈ Rd×1 and ⊙ is the element-wise product. Although data stabilization can achieve better prediction results by reducing the non-stationarity of the sequence, it often lead to the model's inability to accurately capture the original data reconstruction trends. Therefore, we employ the RTE module to recover the lost information. After the model completes the prediction, the input prediction result is O′ = [o′1 , o′2 , . . . , o′ˆ T ]. We use µx and σx to supplement the lost information of o′i and obtain the final prediction result O = {o1, o2, . . . , o ˆ T }. The RTE formula is as follows:

By transforming the model input through stationarization and the output through RTE, the underlying model processes stabilized inputs to yield enhanced predictive outcomes. Subsequently, these predictive results are combined with the RTE module to restore the inherent variability of the data. This approach ensures that the model incorporates trend of change of the data during anomaly detection, thereby enhancing the accuracy of detection.

We adopted the concept of TranAD [Tuli et al., 2022] and utilized the self-conditioning Transformer for model training. Initially, we employed Input Window W ∈ RK×d with a Focus score F = [0]K×d for training where K is the local contextual window of length. To ensure the generated results closely matched the original input, we applied the L2-norm in training both Decoder 1 and Decoder 2:

Where O′1 and O′2 are the decoder results, followed by a RTE operation. In this stage, we allow the two decoders to generate values as close to the window data as possible to ensure the stability of training. Then, we introduce the concept of an adversarial network and utilize the reconstruction loss of L1 as the focus score. After adversarial training, we obtain the final output ˆO2. Decoder 2 attempts to distinguish the generated data from the input window value as much as possible, while Decoder 1 strives to make the generated value as close to the input window as possible to confuse Decoder 2. Through this stage, the attention weight is adjusted to provide higher neural network activation for the subsequence, thereby advancing the short-term time trend. The training objective is:

Combining the two stages, the loss can be determined as follows:

where η is the training parameter. Finally, our Selfconditioning Transformer is shown in Algorithm 1. This model uses both global and local features of the data to complete model training and achieve better prediction accuracy.

3.3 Reconstruction Trend Attention

In order to capture both global and local trend features of the data, we input both complete sequence and window sequence into the Transformer. The window sequence captures the short-term time series trend and the complete sequence captures the overall data trend. To allow the Transformer to learn the trend of changes of the data at the bottom layer, we employ a novel reconstruction trend attention mechanism. This mechanism captures specific time dependencies from the original data sequence through the bottom-level Attention module. The vanilla Attention mechanism is defined as follows [Vaswani et al., 2017]:

Algorithm 1 The RTdetector training algorithm

Require: Encoder E, Decoders D1 and D2

Parameter: Iteration limit N

• 1: Initialize weights E, D1, D2
• 2: n← 0
• 3: do
• 4: for t = 1 to T

5: O′1 ,O′2 ←D1(E(Wt,⃗0)),D2(E(Wt,⃗0))

7:

8:

10:

11:

ˆO2 = σx ⊙ ( ˆO′2 + µx)

L2 = η‖O2 - Wt‖2 - (1 - η)‖ ˆO2 - Wt‖2

Update weights of E, D1, D2 using L1, L2

• 12: n ← n + 1

13: while (n < N)

where Q (query), K (key), V (value) ∈ RT×dk and T is queries length. Each query Q = [q1, q2, ..., qT ] can be calculated as qi = f(xi) where f is embedding layer and input series X = (x1, x2, . . . , xT ). After stationarization, the model received X ′ = [x′1 , x′2 , . . . , x′T ], and X ′ can get by equation 2. Thus we can get the query Q′ = [q′1 , ..., q′T ] by the equation:

where µQ = 1 T ∑Ti=1 qi ∈ Rdk×1 . and the Q′ can written as (Q- 1µ⊤Q)/σx where 1 ∈ RT×1 is an all-ones vector. The K′ can same calculated as (K - 1µ⊤K )/σx. Thus the QKT can calculated as:

ˆO′2 ← D2(E(Wt, ‖O1 - Wt‖2))

and the attention can calculated as:

where S is the softmax calculation, We find the (QµK)1⊤ and 1(µ⊤QµK)1⊤ are both ∈ RS×S , the same as σ2 xQ′K′⊤ and 1(µ⊤QK⊤). Thus we can obtain equation 12 from equation 11.

where c is a matrix with the same columns and the same number of rows and columns as x.

In order for the model to capture the date change trend of Q′ , K′ , we use two MLP to obtain τ = σ2x , ∆ = KµQ respectively. The RT-Attention can be denoted:

where A is the attention calculation. RT-Attention effectively learns the sequence stationary information while capturing the change trend of the original sequence through the change factor.

3.4 Anomaly Detection Inference

Our anomaly detection inference process is shown in Algorithm 2.The anomaly score is defined as follows:

where the ˆW is the unseen window data.We considered both Decoder outputs for obtaining anomaly scores. At the time of anomaly detection, when the anomaly score of any dimension time series is larger than a threshold, we assume

Algorithm 2 The RTdetector testing algorithm

Require: Trained Encoder E, Decoders D1 and D2, Test Dataset ˆW

• 1: for t = 1 to ˆ T
• 2: O′1 ,O′2 ←D1(E( ˆWt,⃗0)),D2(E( ˆWt,⃗0))
• 3: O1,O2 = σx ⊙ (O′1 +µx), σx ⊙ (O′2 +µx)
• 4: ˆO′2 ← D1(E( ˆWt, ‖O1 - W‖2)), D2(E( ˆWt, ‖O1 - W‖2))
• 5: ˆO2 = σx ⊙ ( ˆO′2 +µx)
• 6: s = 12 ‖O1 - ˆW‖2 + 12 ‖ ˆO2 - ˆW‖2
• 7: yi ← (si ≥ POT(si)) ? 1 : 0
• 8: y = ∨i yi

that the entire multivariate time series is anomalous. To ensure a fair comparison, we employ the Peak Over Threshold (POT) method to dynamically and automatically determine the threshold [Siffer et al., 2017].

4 Experimental

4.1 Experimental Settings

Datasets

We evaluate RTdetector extensively on four real-world datasets: (1) UCR [Dau et al., 2019] is a dataset containing a variety of time series, and we only use data obtained from natural sources (InternalBleeding and ECG datasets), (2) MITBIH Supraventricular Arrhythmia Database (MBA) [Moody and Mark, 2001; Goldberger et al., 2000] is a database for studying arrhythmias provided by the Massachusetts Institute of Technology in the United States. It contains multiparameter cardiopulmonary data and ECG signal diagnostic information recorded by electrocardiologists. (3) Server Machine Dataset (SMD) [Su et al., 2019] is a collection of resource utilization calls of 28 machines within 5 weeks. We use the non-stationary time series for training and testing which named machine-1-1, 2-1, 3-2 and 3-7. (4) Soil Moisture Active Passive (SMAP) [Hundman et al., 2018] is a data set that contains telemetry information data collected by NASA using the SMAP satellite, and the telemetry anomaly data is annotated by experts.

Baselines

We extensively compare our model with 11 baselines, including the reconstruction-based models: OmniAnomaly [Su et al., 2019], MSCRED [Zhang et al., 2019], MAD-GAN [Li et al., 2019]; the density-estimation methods: DAGMM [Zong et al., 2018]; the autoregression-based models: LSTM-NDT [Hundman et al., 2018] USAD [Audibert et al., 2020],CAEM [Zhang et al., 2021], GDN [Deng and Hooi, 2021], TranAD [Tuli et al., 2022]; the classic methods: MERLIN [Nakamura et al., 2020].

Evaluation Criteria

We use recall, precision, F1-score and the area under the receiver operating characteristic curve (ROC/AUC) to evaluate the detection performance of the model [Huet et al., 2022].

We use common anomaly detection criteria for a fair comparison, if any individual time series within the multivariate data is diagnosed as an anomaly, the entire multivariate time series is classified as anomalous [Su et al., 2019; Tuli et al., 2022].

Implementation Details

Following the approach of TranAD, we use non-overlapping sliding windows to obtain sub-windows [Tuli et al., 2022]. For all datasets, the sliding window size is fixed at 10. If the anomaly score at a time point exceeds a certain threshold, the entire window is considered anomalous. We use the Adam optimizer [Kingma and Ba, 2014] to train our model with an initial learning rate of 0.01, a step size of 0.5, 64 hidden units in the encoder layers, and a dropout rate of 0.1 in the encoders. All RTdetector experiments are implemented in PyTorch [Paszke et al., 2019] on an NVIDIA GeForce RTX 2080 Ti GPU.

4.2 Performance Evaluation

We first evaluated our RTdetector on four real-world multivariate datasets using eleven competitive baselines, as shown in Table 1. Our proposed RTdetector outperforms state-ofthe-art baseline methods on most datasets. Specifically, it achieves superior detection performance across all datasets except SMAP, while still demonstrating excellent results on SMAP. We believe this is due to the obvious cyclical nature of the SMAP dataset, where the amplitude of normal data does not vary significantly, thus diminishing the advantage of the RTdetector.

Notably, our method significantly outperformed others on the SMD dataset. This dataset is characterized by high non-stationarity, making it challenging to capture the inherent trends. This highlights the difficulty previous methods faced in effectively capturing accurate reconstruction trends in datasets with complex trend variations. In our model, we address this by using a self-conditioning Transformer based on RTE to restore the inherent trends of the reconstructed data. Additionally, our RT-Attention effectively captures the true data trends from the model's lower layers, thereby improving the application of reconstruction trends in anomaly detection.

4.3 Ablation Experiments

To validate the effectiveness and necessity of our design, we conducted ablation studies on the RTE and RT-Attention modules of the RTdetector. Specifically, we removed the RTE module and replaced the RT-Attention module with vanilla Attention, as shown in Table 2. We observed that the RTE module compensates for information lost due to stationarization by supplementing it after the model's decoder output, making the reconstructed data more closely align with the true data trends. Removing this module led to significant decreases in precision, F1-score, and AUC, demonstrating that this module effectively enhances the model's detection accuracy. From the ablation experiments on the RT-Attention module, we found that using the RT-Attention module instead of vanilla attention, resulted in significant improvements in precision, F1-score, and AUC, though recall slightly decreased. We believe this is because the vanilla Attention module tends to overfit abnormal patterns, leading to more false positives. In contrast, RT-Attention enables the model to learn the correct reconstruction trends at the lower layers, thereby improving anomaly detection results.

4.4 Overhead Analysis

To demonstrate the computational efficiency of our method, we further provide the average training time of all models on each dataset. As illustrated in Table 3, our RTdetector achieves a reduction in training time of over 75% compared to most baseline methods.

4.6 Hyperparameter Analysis

We conducted a sensitivity analysis on the window size parameter. As shown in Figure 4, our results indicate that when the window size exceeds 10, the F1 score decrease significantly. We believe that this is because an excessively large window size will cause the anomaly to be hidden in the long local data and difficult to accurately diagnose. It is difficult for the model to accurately predict its changing trend from the local information of a long sequence.

5 Conclusions

To address the issue of overfitting anomalies in reconstruction-based time series anomaly detection, we proposes a novel anomaly detection algorithm named RTdetector. In RTdetector, a self-conditioning transformer based on RTE is designed to enhance model predictability by amplifying the difference between reconstruction results and original data. RTE ensures that reconstruction results adhere to the normal data trends. Additionally, a global anomaly attention mechanism based on reconstruction trends is designed to enable the model to learn the true reconstruction trends of the data, further ensuring consistency between the final reconstruction trends and the original data. Extensive experiments shown that RTdetector outperforms existing state-of-the-art algorithms on four benchmark datasets.

Acknowledgments

This paper is supported by the National Natural Science Foundation of China (Grant no.62302527), the Young Scientists Fund of the National Natural Science Foundation of China (Grant No. 62302529), the Hunan Provincial Natural Science Foundation of China (2023JJ40770).

4.5 Visual Analysis

We have conducted a visual analysis of the RTdetector we proposed and compared it with traditional methods. As shown in Figure 3, our findings indicate that the RTdetector method does not aim to generate data that closely resembles real data like traditional methods. Instead, it only generates a reconstruction trend of the data and determines the occurrence of a fault based on whether the real data aligns with this reconstruction trend. By focusing on the data reconstruction trend, our approach can effectively reduce false positives associated with traditional methods. Specifically, it avoids reconstructing anomalies that should not exist due to sudden changes in the data.