VCformer: Variable Correlation Transformer with Inherent Lagged Correlation for Multivariate Time Series Forecasting

1 Introduction

Multivariate time series (MTS) forecasting is widely used in a range of applications, including energy consumption, weather, traffic, economics, and other fields [Shao et al., 2022; Choi et al., 2022; Wang et al., 2023; Guo et al., 2023; Cast´an-Lascorz et al., 2022]. Unlike univariate time series, MTS involves multiple interrelated time-dependent variables, presenting unique challenges in capturing intricate intervariable dependencies [Han et al., 2023]. Consequently, MTS forecasting has always been a prominent research domain in both industry and academia.

∗Corresponding author

The achievement of Transformer [Vaswani et al., 2017] in natural language processing [Brown et al., 2020] has led to the emergence of numerous Transformer variants for time series prediction tasks.

These models have developed various sophisticated attention mechanisms and enhancements to the Transformer architecture [Zhou et al., 2022a; Wu et al., 2021; Li et al., 2019a; Zhou et al., 2022b], which demonstrate a remarkable modelling ability for temporal dependencies in time series data [Wen et al., 2023].

However, there is an ongoing academic discourse regarding their ability to effectively capture temporal dependencies [Zeng et al., 2023] which typically embed each time step into a mix-channel token and apply attention mechanism on every token. Considering that these methods may overlook the valuable multivariate relationships, which is crucial for MTS forecasting, researchers have begun to focus on ensuring the channel independence and incorporating mutual information to explicitly model multivariate correlations. [Zhang and Yan, 2023; Nie et al., 2023; Yu et al., 2023; Liu et al., 2023a].

Nevertheless, the traditional self-attention mechanism obtain the relationship between two variables via dot-product which can be approximately analogous to attn(v1, v2) shown in Figure 1a. This approach aligns each time step of two variables ignoring the potential existence of different time delays between them, as shown in Figure 1b [John and Ferbinteanu, 2021; Chandereng and Gitter, 2020].

Addressing the limitations of vanilla variable point-wise attention, we introduce the Variable Correlation Transformer (VCformer) to fully exploit lagged correlation inherent in MTS through the Variable Correlation Attention (VCA) module. The VCA module calculates the global strength of correlations between each query and key across different feature. Inspired by stochastic process theory [Chatfield and Xing, 2019; Blight and Chatfield, 1991], it not only computes autocorrelations akin to those in Autoformer [Wu et al., 2021] but also extends this concept to determine lagged crosscorrelations among various variates. The method employs a ROLL operation combined with Hadamard products to approximate these lagged correlations effectively. Furthermore, VCA adaptively aggregates lagged correlation over various lag lengths, thereby determining the comprehensive correlation for each variate. To enhance the model's capability in addressing the non-stationary property in MTS, we design the Koopman Temporal Detector (KTD) module inspired by Koopman theory in dynamics [Koopman, 1931]. Experimentally, VCformer achieves state-of-the-art (SOTA) performance on eight real-world datasets. We also conduct experiments about VCA generality on other previous SOTA Transformer-based models, which demonstrates the powerful capability of modelling channel dependencies of VCA. In general, our contributions lie in the following three aspects:

• We propose a novel model for MTS forecasting, which is called VCformer. It learns both variable correlations and temporal dependencies of MTS.
• We design VCA mechanism to fully exploit lagged correlations among different variates. Additionally, we propose KTD inspired by Koopman theory in dynamics to effectively address non-stationarity in MTS forecasting.
• Experimentally, VCformer achieves top-tier performance on eight real-world datasets. To further evaluate the generality of VCA function, VCA is used in other Transformerbased models and gets better performance.

2 RelatedWork

Advancing beyond contemporaneous Temporal Convolutional Networks [Sen et al., 2019] and RNN-based models [Salinas et al., 2020; Lai et al., 2018], Transformer variants have shown excellent capability in sequence modelling. All the modifications can be divided into two groups according to their focus: solely on modeling temporal dependencies and addressing both temporal and variable dependencies.

For the former, a series of sophisticated attention mechanisms has been developed which can be roughly classified into three categories. The first category is to remove redundant information by introducing sparse bias, thereby reducing the quadratic complexity of vanilla Transformer [Li et al., 2019a; Zhou et al., 2022a]. The second is to transfer the self-attention mechanism from time domain to frequency domain. This shift is facilitated by Fast Fourier Transform or other frequency analysis tools, enabling a more granular extraction for temporal dependencies at sub-series level [Wu et al., 2021; Zhou et al., 2022b]. The third category is related to tackling the distribution shift phenomenon in time series such as De-stationary attention [Liu et al., 2022b]. Beyond these attention-focused innovations, the former also include methods that incorporate multi-resolution analysis of time series via hierarchical architectures [Liu et al., 2022a].

With the primary focus on extracting temporal dependencies, these models designed various exquisite attention mechanisms and fancy architectures. However, a critical vulnerability in these models is the neglect of the rich cross-variable information, which is important for MTS forecasting tasks.

For the later in addressing MTS forecasting, two primary dimensions emerge in multivariate modeling: ChannelIndependent (CI) and Channel-Dependent (CD). CI takes variates of time series independently and adopts the shared backbone. CD predicts future values by taking into account all the channels [Li et al., 2023]. [Nie et al., 2023] introduces patching and CI strategies, significantly enhancing its performance within Transformer-based architectures. Although CI structure is simple, its time-consuming training and inference has catalyzed the development of CD method for modeling multivariate relationships. For the CD method, [Zhang and Yan, 2023] employs temporal and variable attention serially to capture both cross-time and cross-dimension dependencies, while [Yu et al., 2023] applies them in parallel. Moreover, iTransformer [Liu et al., 2023a] revolutionizes the vanilla Transformer by inverting the duties of the traditional attention mechanism and the feed-forward network. They focus on capturing multivariate correlations and learning nonlinear representations respectively.

While these above works acknowledge the significance of modelling multivariate relationships, they adopt the classical self-attention mechanism based on point-wise method, which does not fully exploit the relationship among variable sequences. Despite the existing methods for analysis of lagged cross-correlations in time series [John and Ferbinteanu, 2021; Chandereng and Gitter, 2020; Shen, 2015], these time series Transformers in the literature have not leveraged them among variables, thereby limiting their predictive performance.

3 Method

In MTS forecasting, given historical observations X = {x1, . . . , xT } ∈ RT×N with T time steps and N variates, we predict the future H time steps Y = {xT+1, . . . , xT+H} ∈ RH×N . To tackle this MTS forecasting task, we proposes VCformer which is shown in Figure 2.

3.1 Background

In this section, we first discuss the current limitation of vanilla variable attention in modelling feature-wise dependencies. This then motivates us to propose the variable cross-correlation attention mechanism, which operates across the feature channels for learning cross-correlation among variates.

Next, we review the Koopman theory and treat time series as dynamics. Based on this, we design the KTD module and combine it with the variable cross-correlation attention to learn both channels and time-steps dependencies.

Limitation of Vanilla Variable Attention

In the previous Transformer-based forecasters which adopted attention mechanism for facilitating the temporal dependencies, the self-attention module employs the linear projections to get queries, keys and values Q,K,V ∈ RT×D , where D is the projected dimension. With the queries Q = [q1, q2, . . . , qT ]⊤ and keys K = [k1, k2, . . . , kT ]⊤, the pre-Softmax attention score is the computation with Ai,j = (QK⊤/√D)i,j ∝ q⊤i kj. Nevertheless, feature-wise information, where each of the D features corresponds to an entry of qi ∈ R1×D or kj ∈ R1×D , is absorbed into such innerproduct representation. This thus makes such temporal attention unable to explicitly leverage the feature-wise information. iTransformer [Liu et al., 2023a] considered the limitation of temporal attention and proposed the inverted Transformer to capture cross-variable dependencies that instead computes K⊤Q ∈ RD×D . This simple design is suitable for capturing instantaneous cross-correlation, but it is insufficient for MTS data which is coupled with the intrinsic temporal dependencies. In particular, the variates of MTS data can be correlated with each other, yet with a lag interval. This phenomenon is referred to as lagged cross-correlation in MTS analysis [John and Ferbinteanu, 2021; Chandereng and Gitter, 2020; Shen, 2015]. Additionally, a variate in MTS data can even be correlated with the delayed copy of itself which is termed autocorrelation [Wu et al., 2021]. With yet less-efficient modelling capabilities of cross-correlation, we hereby aim to derive a flexible and efficient correlated attention mechanism that can elevate existing Transformer-based models.

Non-linear Dynamics Tackled by Koopman Theory

Koopman theory [Koopman, 1931; Brunton et al., 2022] shows that a linear dynamical system can be represented as an infinite-dimensional non-linear Koopman operator K, which operates on a space of measurement functions g, such that:

Dynamic Mode Decomposition(DMD) [Schmid and Sesterhenn, 2008] seeks the best fitted matrix K to approximate infinite-dimensional operator K by collecting the observed system states (a.k.a snapshots). However, it is highly nontrivial to find appropriate measurement functions g as well as the Koopman operator K. Therefore, by the universal approximation theorem [Hornik, 1991] of deep networks, many works employ DNNs to learn measurement functions in a data-driven way [Han et al., 2020; Li et al., 2019b; Morton et al., 2019; Li and Jiang, 2021; Lusch et al., 2018].

Koopman theory serves as a connection between finitedimensional nonlinear dynamics and infinite-dimensional linear dynamics, enabling the use of spectral analysis tools for detailed examination. In this paper, we consider time series data X = {x1, . . . , xT } as observations of a series of dynamic system states, where xi ∈ RN is the system state. Therefore, we design the KTD module which leverage Koopman-based approaches to tackle nonlinear dynamics.

3.2 Structure Overview

The proposed VCformer is shown in Figure 2. Following the same Encoder-only structure as iTransformer [Liu et al., 2023a], we adopt the Inverted Embedding : RT ↦→ RD , which regards each univariate time series as the embedded token, instead of embedding multiple variates at the same time as the (temporal) token. By stacking L layers with VCA and KTD modules, the cross-variable relationships and temporal dependencies in time series can be captured. The final prediction is obtained by the Projection : RD ↦→ RH .

3.3 Variable Correlation Attention

Our VCA is comprised of lagged cross-correlation calculation and scores aggregation.

Lagged Cross-correlation Computing

Recall from stochastic process theory [Chatfield and Xing, 2019] that for any real discrete-time process {Xt}, its autocorrelation RX ,X (τ ) can be computed as follows:

Given the queries Q = [q1, q2, . . . , qN] and keys K = [k1, k2, . . . , kN] expressed in feature-wise dimension where qi, kj ∈ RT×1 , we make an approximation for the autocorrelation of variates i:

where ROLL (ki, τ ) denotes the elements of ki shift along the time dimension and ⊙ denotes the Hadamard product. This idea was also harnessed in Autoformer [Wu et al., 2021]. Similarly, we can compute the cross-correlation between variate i and j by:

where τ ∈ [1, T ]. Consequently, we calculate all the variates lagged cross-correlations with different lag lengths in this way.

Scores Aggregation

To obtain the total correlation of variate i and j, we aggregate different lags τ from 1 to T with learnable parameters λ = [λ1, λ2, . . . , λT ] to more accurately calculate the effect of lagged correlation:

Finally, the VCA performs softmax on the learned multivariate correlation map A ∈ RN×N at each row and obtains the output via:

3.4 Koopman Temporal Detector

We employ the KTD to address the non-stationarity in the input series Xvar ∈ RN×D with multivariate correlation information. Remarkably, it is non-trivial to directly capture the non-stationarity in the entire series Xvar, but fortunately we identify that the localized time series exhibits weak stationarity, thereby aligning with the Koopman theory for nonlinear dynamics analysis. Consequently, we divide the input Xvar into DS segments xj of length S:

where each segment can be served as a snapshot for the system. Subsequently, for every xj ∈ RN×S , we leverage MLP-based Encoder : RN×S ↦→RM to project it into a Koopman space embedding zj ∈ RM . According to eDMD [Williams et al., 2015], these embeddings Z = [z1, z2, . . . , zDS ] ∈ R DS ×M are then utilized to calculate the fitted matrix Kvar, facilitating an approximation of the infinite Koopman operator K.

Specifically, given the Koopman embedding Z, we construct two matrices Zback = [z1, z2, . . . , zDS -1 ] ∈ R( DS -1)×M and Zfore = [z2, z3, . . . , zDS ] ∈ R( DS -1)×M , which respectively contain information of adjacent embeddings. After that, the fitted matrix Kvar ∈ RD×D can be calculated as the following equation:

where Z † back is theMoore-Penrose inverse of Zback . Following the deviation of Kvar, we iteratively apply it to predict HS Koopman embeddings as follows:

In this way, a prediction of length H is obtained. Finally, to obtain the output of KTD, we adopt a Decoder : RM ↦→ RN×S , which maps the predicted embeddings back, yielding Yvar as follows:

3.5 Efficient Computation

For each vector pair qi, kj ∈ RT×1 , the time complexity of the lag-correlation (Equation 5) is O (T 2 ). Therefore, calculating COR(qi,K) demands O (NT 2 ) time. It leads the overall complexity of VCA to O (N 2T 2 ) in its current form. To alleviate the computational burden, we apply Fast Fourier Transforms (FFT) based on WienerKhinchin theorem [Wiener, 1930], thus reducing the complexity to O (N 2T logT ). Specifically, for computing the lag-correlation in Equation 4, given discrete time series {Xt} and {Yt}, the RXY(τ) can be calculated via FFT as follows:

where τ ∈ [1, T ]. F and F -1 denotes the FFT and its inverse respectively, and ∗ is the conjugate operation. Specifically, we transform Q and K into frequency domain using FFT. Then element-wise multiplication (a.k.a Hadamard Product) is applied to ith row of the F (Q) and F (K) to compute the LAGGED-COR(qi,K). Extending this process to the entire matrix F (Q) and applying inverse FFTs to these products yield the complete lag-correlations between Q and K. As FFT and inverse FFT each requires O(T logT), the optimized VCA achieves a complexity of O (N 2T log T ).

baselines to ensure the fairness.

4.1 Time Series Forecasting

Table 1 shows the comprehensive experimental results, where the lower MSE/MAE indicates the more accurate result. And we highlight the best in red and bold, while the second in blue and underlined. As we can see, Table 1 illustrate that VCformer consistently achieves top-tier performance across a range of datasets, outperforming other previous SOTA models. It can be attributed to the robust capability of VCA component in extracting correlations among multiple variables. Additionally, it is noteworthy that VCformer achieves the best results on the Exchange dataset which is characterized by high non-stationarity. This success can be owing to the KTD component which augments the power in capturing the nonstationarity from time series. Furthermore, in other datasets like ETT where VCformer does not attain the best benchmark, it still yield competitive results. We also find that the conventional Transformer-based models such as Non-stationary Transformer [Liu et al., 2022b], only achieve the modest performance. It further substantiates the previously discussed limitations of the vanilla attention mechanism in tackling temporal dependencies.

4.2 VCAGenerality

To further explore the effectiveness of VCA, we migrate the VCA module to several well-known Transformer variants:

4 Experiment

Dataset We conduct extensive experiments on eight widelyused real-world datasets [Zhou et al., 2022a], including Electricity Transformer Temperature (ETT) with its four subdatasets (ETTh1, ETTh2, ETTm1, ETTm2), Weather, Electricity, Traffic and Exchange. Following [Zhou et al., 2022a], we adopt the same train/val/test sets with splits ratio 6:2:2. For the ETT datasets, we split the remaining four sub-datasets by the ratio of 7:1:2 following [Wu et al., 2021].

Baselines We carefully select a range of SOTAmethods as baselines to provide a comprehensive comparison with our proposed approach including (1) Transformer-based methods: Stationary [Liu et al., 2022b], Crossformer [Zhang and Yan, 2023], DSformer [Zhang and Yan, 2023], iTransformer [Liu et al., 2023a]; (2) MLP-based methods: DLinear [Zeng et al., 2023], Koopa [Liu et al., 2023b]; (3) TCN-based methods: TimesNet [Wu et al., 2023].

Setups Following [Zhou et al., 2022a], we normalize the train/val/test to zero-mean using the mean and standard deviation from the training set. The Mean Square Error (MSE) and Mean Absolute Error (MAE) are selected as evaluation metrics, consistent with previous works. All of models adopt the same prediction length H = {96, 192, 336, 720}. For the look-back window with length T , we follow the same setting as TimesNet [Wu et al., 2023] which sets T = 96 for all the Non-Stationary Transformer [Liu et al., 2022b], Autoformer [Wu et al., 2021] and Informer [Zhou et al., 2022a]. Since these Transformer-based models have masked decoders in which the partial values of scores (QKT) are replaced with -∞, the FFT in VCA module can not be used to quickly calculate the lag-correlations of queries and keys.

Therefore, we retain the original design of decoder and simply replace self-attention in encoder with VCA. The experimental results are shown in Table 2, where (VCA) represents the replaced model. We can see that the VCA module has significantly improved the performance of traditional Transformer-based models (Non-stationary Transformer, Autoformer and Informer), which yields an overall relative MSE reduction with 13.3%, 14.7% and 36.5%.

4.3 Model Analysis

Ablation Study

In order to comprehensively understand the individual contributions of the key components in VCformer, we conduct ablation experiments covering experiments with both replacing components (Replace) and removing components (w/o), as shown in Table 3. From these results, we can conclude that both VCA and KTD are indispensable for the best performance of VCformer, which utilizes lag-correlation on variate dimension and Koopman detector on temporal dimension. After replacing or removing either one of them, the MSE/MAE

will increase, which validates their effectiveness. Notably, in the datasets with a large number of variates, specifically Traffic and Electricity, the replacement or removal of the VCA module incurs a remarkable increase in MSE/MAE, i.e., an averaged increase of 9% over the increase by KTD. This suggests that VCA plays a more critical role when dealing with a larger number of variates. On the other hand, within the Exchange dataset noted for its volatility, the substitution or removal of the KTD module, which can capture series nonstationarity, results in more noticeable performance drops than that of the VCA module. Conversely, pertaining to the Weather dataset, the experimental results indicate that the omission or replacement of either module doesn't lead to major variances in performance. These results show that the KTD module is good at learning features from non-stationary time series.

Multivariate Correlation Analysis

To enhance the interpretability of learned multivariate correlations by VCA, we provide a visualization case with random selection on series from Exchange. As demonstrated in Figure 3, the upper part presents the variate correlations inherent within the raw series including both input and prediction sequences. These correlations are calculated by Pearson Correlation coefficient as the following equation:

where xi, yi ∈ R run through all time points of the paired variates to be correlated. The lower part portrays the preSoftmax score maps learned by VCA in both the first and the last layers.

From the red box in Figure 3, we can observe that the multivariate correlations learned by shallow layer of VCA are more similar to correlations of the input raw series. And as we explore at a deeper layer of VCA, we find that the learned multivariate correlations are closer to the forecasting sequences. This observation indicates that the focus of learned multivariate correlations shifts progressively from input series to prediction sequences. It also enhances interpretability of VCA which aggregates different lag-correlations to represent these variable relationships.

Model Efficiency Analysis

As shown in the Figure 4, we conduct a comparative study of the VCformer's efficiency with seven baselines. Our assessment considers three aspects: training speed, memory consumption and prediction performance. It can be observed that the time complexity of VCformer is O (N 2T log(T)). However, the coefficient N 2 does not significantly impact the training time when handling datasets with a small number of variates like Exchange. Notably, for the Traffic dataset which contains a large number of variates, the actual computational consumption is not as large as expected. It even outperforms PatchTST and TimesNet, which can be largely attributed to all the required calculations based on matrix operations. And these operations are well parallelized in built-in library.

5 Conclusion

In this paper, we address the limitations of the conventional dot product attention mechanism in extracting multivariate correlations. Then we propose VCformer which contains two effective modules. The VCA module can not only mine the lagged cross-correlation implicit in MTS, but also seamlessly integrate into other Transformer-based models. The KTD module employs MLP modules to derive Koopman embeddings and generates Koopman operator to enhance the capability for capturing non-stationarity in MTS. Extensive experiments shows that VCformer achieves SOTA forecasting performance and its VCA module is general enough to improve performance of various Transformer-based models.

Ethical Statement

There are no ethical issues.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grants U2013201 and 62203308.