SMARTformer: Semi-Autoregressive Transformer with Efficient Integrated Window Attention for Long Time Series Forecasting

1 Introduction

Multivariate Time Series Forecasting, as an interdisciplinary research prevalent in scientific and engineering problems, has witnessed great advances in recent years, with a notable trend of predicting accurate series from short term [Li et al., 2018; Liu et al., 2018; Salinas et al., 2020; Bai et al., 2020; Deng et al., 2021] to long term [Zhou et al., 2021; Li et al., 2023a; Li et al., 2023b]. Recently, Transformer-based models [Vaswani et al., 2017] have demonstrated great potentials on long time series forecasting (LTSF) for capturing longrange correlations.

Despite these potentials, existing transformer-based models do not adequately consider the characteristics of time series data and are still suffering from ineffectiveness and inefficiency in capturing those local dependencies among complicated temporal patterns, impairing the ability to accurately model long time series. For instance, as shown in Figure 1a, when predicting a long time sequence with a size of 720 timesteps for the real-world Exchange-rate dataset [Lai et al., 2018], the non-stationary transformer [Liu et al., 2022] with a typical non-autoregressive decoder, leads to performance collapse in a single prediction step.

Moreover, positional embedding is also directly related to local dependencies. Due to the different scales in time series, directly adding positional embedding to the value embedding, as previous works [Zhou et al., 2022a; Wu et al., 2021], can result in entanglements of various periods, thus leading to confusion for future prediction. Therefore, we develop TIE to better capture the local dependencies.

We further investigate the vanilla attention mechanism's limitations by examining the attention maps among different timesteps computed on the Exchange rate dataset (Figure 2b). The attention map is highly sparse, which aligns with recent research [Zhou et al., 2021]. They found that the attention feature map follows a long-tail distribution, where a few dot-product pairs contribute to attentions while most can be ignored. This limited attention mechanism hinders transformer models from learning better temporal correlations among timesteps, leading to significant computation costs that restrict the practical deployment of such models in real-world applications.

In summary, by investigating the architectures of Transformers for time series interaction, there are several major bottlenecks, i.e., the input embeddings that can incorporate more decoupled prior information, the attention mechanism that can efficiently and effectively model the interaction among different temporal patterns, and the decoder that can stably forecast with consistent long-range dependencies.

We propose the SMARTformer, which stands for SeMiAutoRegressive Transformer with Efficient Integrated Window Attention, to address issues in general LTSF tasks. The SMARTformer architecture is illustrated in Figure 3. To better decouple positional embedding from value embedding, we design a simple yet effective Time-Independent Embedding (TIE) method, which avoids entanglements of various periods and enhances the inductive bias of significant periodic variations. This improvement is shown on the Electricity Dataset in Figure 2a. To capture abundant latent correlations, we design the Integrated Window Attention (IWA), which separates an attention layer into two branches. One branch conducts local self-attention in non-overlapping multi-scale windows, while the other conducts global attention across windows. These designs achieve valuable temporal patterns (as shown in Table 7) and break the bottleneck of computational efficiency (as demonstrated in Figure 5). Furthermore, to enhance the decoder's power for outputting consistent longrange dependencies, we design a semi-autoregressive (SAR) decoder. The promising performance of SAR is demonstrated in Figure 1b, which enables the non-stationary Transformer to better fit the ground truth sequence. Extensive experiments on five public and commonly used multivariate time series datasets from different domains demonstrate the outstanding performance of the proposed SMARTformer.

2 RelatedWork

We review transformer models for LTSF according to the attention mechanism and decoder design.

Efficient Attentions in LTSF. To tackle LTSF tasks, designing effective self-attention mechanisms is crucial. According to attentions, Transformer variants can be roughly categorized into two types. The first type is Temporal Sparse Attention, which sparsifies attention with predefined patterns [Li et al., 2019; Kitaev et al., 2020; Liu et al., 2021; Cirstea et al., 2022]. Through reducing complexity, they are trapped by prefined structures, failing to accurately capture the correlations. The second type is Frequency Domain Attention, which fuses decomposition blocks with Fast Fourier Transform or other frequency analysis method to discover series-wise connections [Wu et al., 2021; Zhou et al., 2022a; Chen et al., 2022]. However, converting data to frequency domain may inevitably lose fine temporal variations, thus leading to sub-optimal solutions. Based on the data distribution of the attention matrices (as illustrated in Figure 2a), the proposed Integrated Window Attention is designed to model local and global interaction at the same time, which is significantly different from existing methods.

Non-Autoregressive Decoding. Autoregressive (AR) decoding is widely used in NLP seq2seq models [Sutskever et al., 2014] and Transformer-based pretraining models [Lewis et al., 2019; Yang et al., 2019]. Meanwhile, AR decoding dominates in short time series forecasting, [Qin et al., 2017; Salinas et al., 2020; Lai et al., 2018; Li et al., 2019]. However, for LTSF, AR decoding achieves unsatisfactory performance due to error accumulation, as mathematically proven in [Sun and Boning, 2022]. Thus, Informer initially adopted Non-autoregressive (NAR) decoding [Gu et al., 2017] to avoid error accumulation and improve efficiency, which was also used in subsequent works (Autoformer [Wu et al., 2021], FEDformer [Zhou et al., 2022b], and Scaleformer [Shabani et al., 2022]). Although these methods achieved encouraging results by injecting multi-scale or decomposition prior, they still neglected the disadvantages of NAR decoding itself. Our SAR decoder acts orthogonally to their contributions and can be easily adapted to them and other time series transformers to consistently enhance their performance.

3 Method

Given a D-variates time series W = [w1,w2, . . . ,wT ] ∈ RT×D with T timesteps, the LTSF problem aims to predict ˜W = [wT+1,wT+2, . . . ,wT+L] ∈ RL×D with L future timestep values. To tackle the LTSF tasks, we design the architecture of the proposed SMARTformer, as shown in Figure 3. In the following, we introduce the major components, i.e., the Time-Independent Embedding, the Integrated Window Attention, and the hierarchical Semi-autoregressive Decoder.

3.1 Time-Independent Embedding

In transformer-based time series forecasting, data embedding significantly impacts performance because attentions are insensitive to data order. To enhance the awareness of temporal variations, local positional and temporal information is often added [Zeng et al., 2022]. Previous studies use fixed positional encoding or date-specific embedding to maintain ordering information. However, time series data often contain waveforms with different periods. Adding positional encoding composed of periodic waveforms may destroy original values by causing phase cancellation. We propose that positional encoding should be decoupled from values in time series embedding to reduce the distortion of original data.

To this end, we design a Time-Independent (TI) embedding method. Raw historical data are used as the encoder input W1:T = {wi,d | 1 ≤ i ≤ T, 1 ≤ d ≤ D}, where wi,d denotes the i-th timestep in dimension d. For each timestep, we embed them through two parts: theValue Embeddingwith a simple 1-D convolutional layer to embed each timestep of raw data to a Cv dimension vector, and the Time-Independent Positional Embedding denoted by Fi, calculated as the concatanation of three types of temporal features, i.e.,

Here E(m/h) ,E(wk) ,E(mth) denote three learnable projection matrices for positional embeddings of minute/hour, weekday and month, respectively. Fi ∈ R Cp denotes the vector after positional embedding. Thus, the positional information is shared among timesteps of the same order on a daily/weekly/monthly basis to enhance the understanding of global temporal changes.

For each token wi, we concatenate and normalize these two embeddings (following [Ba et al., 2016]) as

where Conv (wi) ∈ RCv denotes the latent variable after Value Embedding and Ei ∈ R C (C = Cv + Cp) denotes the input embedding. In this way, we leverage the positional information without destroying data semantics, acquiring global awareness of ultra-long temporal changes.

3.2 IntegratedWindowAttention

First, the input feature X ∈ R T×C is linearly projected to K heads, and our proposed attention splits them into two groups - one with S heads (for Intra-window attention) and the other with K-S heads (for Inter-window attention).

Intra-window Attention. For attention in windows, X is evenly partitioned into non-overlapping windows of equal length w along the time dimension. Here, we assume w is divisible by the whole length T . w can be adjusted to balance the learning capacity and computation complexity. Therefore, intra-window attention can be defined as

where Xi ∈ R TM ×C and M = T/w (i = 1, 2, . . . ,M). W Q k ∈ R C×dk , W K k ∈ R C×dk , and W V k ∈ R C×dk represent the projection matrices of queries, keys, and values for the k-th head respectively where dk is set as C/K. Besides, the window length w is an important parameter because it achieves strong modeling capability while limiting the computation cost, where we adjust by applying small lengths for shallow layers and larger lengths for deep layers respectively to capture multi-scale characteristics.

Inter-window Attention. For remaining K -S heads, we perform Inter-Attention, where we shift the original sequence X into Xs with os timesteps, and split it through the same window size w as the former branch to acquire ˆX as

The shifted sequence ˆX is projected as Queries and X is projected as Keys and Values. Then, we acquire the k-th head attention scores Y i k for the i-th window as

where e · w < os < (e+1) · w, 0 < e < M. os is an offset to establish powerful connections across different windows and enhance interactions among tokens at the edge of windows. Then, we integrate two branches to acquire the final output as

By doing so, the computation complexity of the Integrated Window Attention in two branches is both O(w × L) = O(L), scaling linearly with the input length L, reducing greatly to a lower complexity than the standard MSA and ensuring high throughput on GPUs. Moreover, another benefit of head-splitting is that the learnable parameters Q, K, and V are decomposed into two smaller matrices, which helps to reduce model parameters. And we apply different w window sizes for different layers, small w for early stages and larger w for later. Adjusting the length of w provides the flexibility to enlarge the attention area of each token. Thus, we reconcile the global and local self-attention in a single layer to acquire ample variations in an efficient and effective way.

3.3 Semi-Autoregressive Decoder

We describe our hierarchical SAR decoder to handle robust long sequence prediction, as shown in Figure 3. It stacks the Segment AR layer and the NAR refining layer. These layers share the identical structure of a typical Transformer decoder, except for using the IWA instead.

Segment AR Layer. Despite focusing on modeling a long sequence, local contexts also play a crucial role in knowledge propagation. Therefore, in the lower section of our network, we adopt a Segment AR Decoder Layer, which predicts each subsequence iteratively by applying the same decoder layer. For a long sequence with a length of L to predict in the M-th decoder layer, it generates non-overlapping segments through k steps with a length lk. And the output of the M-th decoder layer ZM de ∈ R L×C is acquired by concatenating all the kstep representations along the time dimension as

where ˆ Y M de(j) denotes the j-th step output.

In the j-th step, we assume Lj for the predicted length. A local representation HM (j) ∈ R Lj×C takes as the input, which comes from the predictions of previous steps and positional information of current step to capture local fine variations. ¯ Y M de(j-1) ∈ R Lj×C denotes a slice from the previous prediction ˆ Y M de(j-1) ∈ R Lj-1×C which we pad or slice to a proper length when j = 1 or lj-1 ̸= lj to align them in the time dimension. The positional embeddings F(j) ∈ R Lj×Cp utilize TIE to enhance the awareness of complicated periodic variations. We concatenate them along the channel dimension and learn through an MLP as

The equation to perform in the Segment AR decoder can be summarized as

where Y N en denotes the final output after all the N layers encoder. As for the predicted length for each step, it can be determined based on the sampling frequency of the dataset. Selecting a length that is an integer multiple of the dataset period (day/ week) can help with model prediction.

NAR Refining Layer. We introduce a NAR Refining Layer to add global representational power beyond the Segment AR Layer. The input of the M+1-th Refining Layer comes from the hidden states of the prior layer Y M de ∈ R L×C . And the equations of the M+1-th decoder layer can be summarized as

We stack these two types of layers interchangeably to capture both global and local contexts efficiently. Such design benefits from both the global horizon of the NAR decoding and the local detail capturing of the AR decoding to enhance the power of the decoder for capturing short and long dependencies steadily.

4 Experiments

4.1 Main Results

We conduct extensive experiments to evaluate the performance of SMARTformer and further perform ablation studies to justify how each component contributes to the final results.

Datasets. We perform empirical studies on five real-world benchmark datasets as follows: (1) Electricity. (2) Weather 2 . (3) Traffic 3 . (4) Exchange[Lai et al., 2018]. (5) ILI 4 . The train/val/test splits for the first three datasets are the same as [Zhou et al., 2021], the last two are split by the ratio of 7:1:2 following [Wu et al., 2021].

Baselines. We select nine strong recent baselines, including: Transformer-based models: Non-stationary Transformers [Liu et al., 2022], FEDformer [Zhou et al., 2022a], Autoformer [Wu et al., 2021], Informer [Zhou et al., 2021]and LogTrans [Li et al., 2019]; MLP-based models: DLinear [Zeng et al., 2022]; RNN-based model LSSL [Gu et al., 2022] and LSTM [Hochreiter and Schmidhuber, 1997].

Forecasting Results. As shown in Table 1, for multivariate time series forecasting, SMARTformer outperforms other deep models impressively in all benchmarks, with 37 top-1 and 40 top-2 cases out of 40 in total. Compared with SOTA works (Non-stationary Transformers, Dlinear and FEDformer), ours yields an overall 10.2%, 10.1% and 17.7% relative MSE reduction. We also list the univariate results of two typical datasets in Table 2. Compared with SOTA works (Non-stationary Transformers and Dlinear), ours yields an overall 13.3% and 51.3% relative MSE reduction. And on the Weather dataset, the improvement can reach more than 70%. It proves the effectiveness of SMARTformer in long-term forecasting.

Implementation details. All the experiments are implemented in PyTorch 1.9.1 [Paszke et al., 2019] and conducted for three runs on a single NVIDIA 3090 GPU. Following previous works, the input sequence length is set to 36 for ILI and 96 for the other four datasets. We also use a data normalization layer RevIN [Kim et al., 2022] as a pre-processing block to further enhance the model's robustness. Each model is trained by ADAM [Kingma and Ba, 2015] using L2 loss and batch size of 32. The model contains 3 encoder layers and 2 decoder layers. And we have an additional set of hyperparameters. In IWA, the shift length os = (M/2+1/2)×w, the window size w ∈ {24, 36, 48}. And in TIE, the dim of value embedding Cv = 3/4 ×D, the dim of Positional Embedding Cp = D/4, D denotes the dim of the model. In the segment AR layer of SAR, the subsequence length is L/4 each time.

2 Weather. https://www.bgc-jena.mpg.de/wetter/.

3 Traffic. http://pems.dot.ca.gov/.

4 ILI. https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html.

Semi-Autoregressive Decoder Generity. We apply our SAR decoder to mainstream Transformers (Informer, Autoformer, and FEDformer) and report the performance promotion of each model (Table 3). Our method consistently improves the forecasting ability of different benchmarks. Overall, it achieves averaged 7.7%, 6.2%, and 6.3% promotion on Informer, Autoformer, and FEDformer, making each of them surpass previous state-of-the-art, which validates that our SAR decoder is an effective framework that can be widely applied to Transformer-based models in long-term forecasting to enhance their performance. It is worth noting that with the predicted length increasing, the performance of the SAR decoder changes quite steadily, implying its robustness in extremely long-term forecasting.

4.2 Ablation Study and Analysis

Ablation Study. We perform an ablation study on the Weather dataset to study the effects of our three main modules: TIE, IWA, and SAR. We consider Transformer as the baseline. As shown in Table 4, we observe that each module is essential for accurate prediction where the accuracy drops more significantly when without SAR or TIE, which justifies our design choices. And we note that when replacing IWA with canonical attention, the accuracy loss is relatively smaller than removing other components. Another benefit of IWA is greatly reducing the computations thus improving efficiency.

Impact of SAR. As shown in Table 6, we evaluate the components of our SAR decoder. Using only the Segment AR Layer performs better than using only the NAR decoder, for providing more dependable local features. And combining both achieves the best results stably, proving the importance of modeling global and local features respectively.

Impact of TIE. In Table 5, we compare TIE with fixed positional embedding (fixed PE) and without positional embedding in SMARTformer. As can be observed, the forecasting performance declines rapidly without positional embedding. And our method outperforms fixed PE, which is well aligned with our design considerations for decoupling PE from value embedding.

Impact of IWA. We study the effects of two branches (Inter-Window and Intra-Window) in IWA. As shown in Table 7, we observe the necessity of both branches. When the predicted length is relatively small, the Intra-Window branch contributes more, and the Inter-Window branch is quite essential, especially in the relatively long predicted length.

Impact of os in IWA. As shown in Table 8, we evaluate the impact of shift os under different e and w situations on the Weather dataset. s = w/2, e = M/2 performs the best, because the design of e and w is essential to enhance interactions among tokens across windows.

Running Time Efficiency Analysis. As shown in Figure 5, we evaluate SMARTformer with 8 strong baselines, Transformer based models: Autoformer, FEDformer, Quatformer [Chen et al., 2022], Scaleformer[Shabani et al., 2022], Non-stationary Transformers (represented as Stationary) and GNN based models: DGCRN [Li et al., 2021], GWNet [Wu et al., 2019] and AGCRN [Bai et al., 2020]. Experiments are conducted on the Weather Dataset with a 96 length input and a 720 predicted length output. Our SMARTformer is the fastest and best accurate among the 9 models.

5 Conclusion

Long time series forecasting is notably associated with the ability to stably capture local dependencies. Thus, we propose SMARTformer, consisting of three effective mechanisms, where Time-Independent Embedding enhances periodic variations, Integrated Window Attention achieves complementary clues in various receptive fields, and SemiAutoregressive Decoder identifies solid local and global characteristics in two stages. Extensive experiments show that our framework achieves superior forecasting performance.

Ethical Statement

There are no ethical issues.

Acknowledgements

This work was partially supported by the National Key Research and Development Program of China (No. 2018AAA0100204), a key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. JCYJ20200109113403826), the Major Key Project of PCL (No. 2022ZD0115301), and an Open Research Project of Zhejiang Lab (NO.2022RC0AB04).