Decoupled Invariant Attention Network for Multivariate Time-series Forecasting

1 Introduction

Multivariate time series (MTS) forecasting has been increasingly important in a wide range of real-world applications, such as traffic flow analysis [Yu et al., 2017], weather estimation [Zheng et al., 2015], energy consumption planning [Beccali et al., 2008], etc. A primary challenge in MTS forecasting is to capture the core inter-series (spatial) correlations and intra-series (temporal) dependencies simultaneously [Cao et al., 2020a; Bai et al., 2020]. Despite advancements, accurate forecasting remains elusive due to the inherent complexities, shifted fluctuations, and noise in time series signals, which pose challenges for model learning [Fan et al., 2023].

† Corresponding author.

Existing literatures have tried different strategies to extract the core patterns of time series for accurate forecasting. Some researchers have focused on periodicity modeling to find generalizable periods beneficial for forecasting [Fan et al., 2022; Jiang et al., 2022]; other works priorly define and extract an overall trend of raw series, and jointly learn the trend and the left parts [Wu et al., 2022; Zeng et al., 2023]; besides, [Woo et al., 2022a; Wang et al., 2022] disentangle time series into seasonal and trend and learn their representations respectively. Existing studies typically rely on manually specified core patterns (e.g., periods, trends) of time series based on certain assumptions. Because of focusing solely on these pre-defined patterns and neglecting dependencies on other signals, the performance of multivariate forecasting is significantly compromised.

Inspired by previous works of time series [Lim and Zohren, 2021] and views of domain generalization [Zhou et al., 2022a], we generally categorize time series into two types of patterns for forecasting: (i) invariant patterns, the regular and constant patterns of time series that are easy to illustrate, such as regular seasonal or trend signals; (ii) variant patterns, the volatile and changeable patterns of time series that are hard to capture, such as useful information mixed with noise. Instead of directly specifying the invariant patterns for learning as [Fan et al., 2022; Wu et al., 2022; Woo et al., 2022a], we aim to build a data-driven approach to adaptively distinguish the invariant patterns from the variant patterns in a given raw time series. After that, we can conduct spatial-temporal modeling for multivariate time series from a disentangled learning perspective. However, two challenges have arisen in achieving this goal:

• Challenge I: Adaptive decoupling of invariant and variant patterns presents difficulties. First, these patterns are often intertwined due to the non-stationary nature of time series data, resulting in ambiguous and everchanging boundaries. Moreover, multivariate time series (MTS) encompasses both spatial and temporal di-

mensions, each containing their own invariant and variant patterns. Simultaneously considering these patterns across the Spatiotemporal dimensions further complicates the decoupling process.

• Challenge II: In the temporal dimension, distribution shift are notably pronounced. This heightened variability introduces a substantial perturbation, making it increasingly challenging to discern and differentiate between invariant and variant patterns over time. Furthermore, these shift can adversely impact subsequent forecasting tasks, leading to performance drift.

To address the above challenges, in this paper, we propose Decoupled Invariant Attention Network (DIAN) to adaptively decouple invariant and variant patterns for enhancing MST forecasting.

Specifically, DIAN includes two modules: the Spatial Decoupled Invariant-Variant Learning module and the Temporal Augmented Invariant-Variant Learning module, designed to disentangle patterns in spatial and temporal dimensions, respectively. Drawing inspiration from [Bai et al., 2020], the spatial module employs a spatial invariant-variant attention mechanism to distinguish between spatial invariant and variant patterns. After conducting decoupled invariantvariant convolution, the spatially-learned representations successfully captured both invariant and variant patterns. In the Temporal Augmented Invariant-Variant Learning module, we introduce the Temporal Invariant-Variant Attention to capture correlations between individual timestamps and the overarching temporal representation. To address pronounced temporal distribution shift, we incorporate the Temporal Intervention Mechanism, generating intervened samples for enhanced adaptability to potential shift. In addition, we employ a Joint Optimization strategy to reduce forecasting errors.

Our contributions can be summarized as:

• We introduce a unique perspective on invariant and variant patterns for MTS forecasting.
• We propose DIAN to address the invariant-variant pattern decoupling problem with novel invariant-variant attention designs for the spatial and temporal dimensions, respectively.
• We propose Temporal Intervention Mechanism to against potential distribution shift by reassembling variant patterns among timestamps.
• We conduct extensive experiments on five real-world dataset. The experiment results show consistent superiority in terms of effectiveness and efficiency compared with state-of-the-art methods.

2 ProblemFormulation

We study the problem of multivariate time series forecasting with regard to historical time-evolving multiple variables. Given a time series dataset with N series and T timestamps, let Xt = {x(1) t , · · · , x(i) t , · · · , x(N) t } ∈ RN stands for multivariate values of N series where x(i) t ∈ R1 represents the value of i-th series (variate) at timestamp t. Then, we define the historical observations of length L at timestamp t, Xt-L+1:t = {Xt-L+1, · · · ,Xt} as the lookback window, and the future values of length H, Xt+1:t+H = {Xt+1, · · · ,Xt+H} as the horizon window.

In order to model the relationships among series, we define a dynamic graph structure Gt = (Xt-L+1:t,At), where At ∈ RN×N is the adjacency matrix and Xt-L+1:t ∈ RN×L is the lookback window at timestamp t. Based on the above notations, we can formulate the multivariate time series forecasting task as:

where ̂Yt+1:t+H = {̂ Yt+1, · · · , ̂ Yt+H} are the predicted values corresponding for the horizon window; FΘ is a mapping function with parameters Θ; the dynamic graph structure Gt is learned during training.

3 Methodology

In this section, we illustrate the proposed Decoupled Invariant Attention Network (DIAN) for better modeling and decoupling invariant and variant patterns for enhancing multivariate time series forecasting performance. We first show an overview of our model, and then introduce each component of this model in detail.

3.1 Framework Overview

Figure 1 provides an overview of our framework, known as the Decoupled Invariant Attention Network (DIAN). In Section 3.2, to capture inter-series correlations and dynamic graph structures, we propose Spatial Decoupled InvariantVariant Learning. This approach allows us to derive novel representations that incorporate relationships among different series. Moving on to Section 3.3, we propose Temporal Augmented Invariant-Variant Learning. This technique enables us to obtain representations that capture relationships among timestamps and generate multiple intervened samples, which simulate potential distribution shift. Finally, in Section 3.4, we optimize our model with Joint Optimization.

3.2 Spatial Decoupled Invariant-Variant Learning Assumption 1

Given the dynamic graph structure Gt of time series, there exist spatial invariant patterns QtI that reflect core spatial (interseries) relationships, and spatial variant patterns QtV that encompass a limited amount of valuable information while being intermingled with distribution shift. In this scenario, a function g is assumed to leverage spatial invariant-variant patterns and an adjacency matrix to acquire knowledge about the forthcoming dynamic graph structure at the subsequent timestamp, formally by Gt+1 = g(QtI ,QtV ,At) + ϵ1 and QtI = Xt-L+1:t/QtV .

Spatial Invariant-Variant Attention

Based on the aforementioned assumption, it is understood that the time series can be decomposed into two components: spatial invariant and variant patterns. These patterns capture the essential relationships between the series, and it is necessary to design a mapping function, to extract the inter-series relationships from G and decouple these two patterns, facilitating subsequent predictions.

For this aim, we design a spatial invariant-variant attention network, to enable each series attending to the global inter-series representation for better understanding relationships among series. We capture meaningful spatial dependencies within the global inter-series structure by studying the contribution of each local series, thereby establishing a connection between the local and global contexts. Specifically, we first obtain a spatial overall representation of the current time series using a linear projection layer:

where Rs ∈ R1×T×dc represents the spatial overall representation of the current inter-series relationships with dc dimensions; ProjS represents a linear projection to map series dimension to one; Xt-L+1:t is a lookback window length time series. Then, we need to learn the dependencies between each series and the spatial overall representation. Because the learned representation represents the integrated spatial relationships of the entire time series, a better integration of the inter-series relationships can be achieved, by using the relationship between each individual series (local) and the spatial overall representation (global) as a guidance. For the ith variate (series) X(i) t-L+1:t of the lookback window, we define the attention score as the spatial invariant attention score, which is given by:

where AttnSI (i) represents the spatial invariant attention score of variate i; dS represents the hidden dimension; Wqs and Wks are the embedding layers for query vectors Rs and key vectors X(i) t-L+1:t , respectively. In addition to using invariant attention to learn the core relationships, we also model the unstable inter-series patterns for spatial learning. Following [Zhang et al., 2022], for each i-th variate X(i) t-L+1:t of the lookback window, we define the spatial variant attention score as:

where AttnSV (i) represents the spatial variant attention score of variate i. In this way, the spatial invariant and variant attention scores have a negative correlation, making it easy to distinguish core spatial relations and unstable spatial relations so that we can integrate them reasonably for forecasting.

Decoupled Invariant-Variant Convolution

Previous studies have demonstrated the capability of learning graph structures autonomously, bypassing the need for prior knowledge and thereby circumventing the issue of inaccurate input graphs [Bai et al., 2020]. However, it is crucial to acknowledge that the previous studies also introduce the risk of learning inaccurate graphs. To tackle this challenge, we propose a decoupled invariant-variant convolution network for multivariate time series forecasting. First, we initialize learnable dependency embeddings E ∈ RN×dN with dN as the hidden dimension, and then the adjacency matrix At can be represented as:

In order to enhance the reliability of the convolutional network utilized in graph learning, it is essential to incorporate a dependable relationship between series. Specifically, we incorporate spatial invariant and variant attention to capture the appropriate series relationships within the graph. By utilizing spatial invariant attention scores, we obtain an invariant embedding that represents the stable aspects of inter-series relationships. Similarly, spatial variant attention scores yield a variant embedding that captures the changeable components of these relationships. We formally define the ℓ-th layer of such networks by:

where Hℓ ∈ RN×T×dc represents the hidden representation with dc dimensions of ℓ-th layer; Hℓ+1 I and Hℓ+1 V represent the spatial invariant and variant embeddings respectively. WH I andWH V represents weights in invariant and variant embedding learning; σ is an activation function. Note that when ℓ = 0, we have H0 = Xt-L+1:t. Then, the learned representation incorporating both invariant and invariant patterns can be represented as

where α is a hyperparameter to balance invariant and variant embeddings.

3.3 Temporal Augmented Invariant-Variant Learning

Assumption 2

Given the multivariate window at timestamp t, Xt-L+1:t, there exist temporal invariant patterns P t I that usually reflect core temporal relations, and temporal variant patterns P tV that represent temporal information mixed with distribution shift. Assume there exists a function y that can project temporal invariant and variant patterns into future values, written as: Xt+1:t+H = y(P t I , P tV ) + ϵ2 and P t I =Xt-L+1:t/P tV .

Temporal Invariant-Variant Attention

In order to model temporal dependencies, following Assumption 2, we aim to design a function to effectively decompose the spatially-learned representation of time series, Hℓ+1 into temporal invariant and variant patterns. To achieve this goal, we propose temporal invariant-variant attention mechanism to model temporal dependencies. Specially, we first obtain an overall temporal representation of Hℓ+1 using a linear projection:

where RT ∈ RN×1×dc represents the temporal overall representation of Hℓ+1 with dc dimensions; ProjT represents a linear projection to map time dimension to one; Then, we learn the invariant-variant attention between each timestamp and the temporal overall representation. The temporal overall representation encompasses the temporal information, such as seasonal and trend, within a lookback window. This approach enables us to understand the contribution of each timestamp to the temporal overall representation, and establishes the connection between the local and global aspects. Specifically, for each t-th timestamp, we define the temporal invariant and variant attention score as:

where AttnTI (t) and AttnTV (t) represent the temporal invariant and variant attention score respectively; Hℓ+1 t represents the spatially-learned representation at timestamp t; dM represents the hidden dimension; Wqt and Wkt represent the embedding layers for query vectors RT and key vectors Hℓ+1 t from the time dimension. Then, for each Hℓ+1 t at t-th timestamp, we want to aggregate the patterns for representation:

where AggrI and AggrV are aggregation functions for temporal invariant and variant patterns respectively; ztI and ztV represents the pattern aggregations of temporal invariant and variant patterns. Then, for timestamp t, we acquire pattern summarization ht = ztI + β ∗ ztV fed into subsequent layers, where β is a hyperparameter to balance the temporal invariant and variant patterns.

Temporal InterventionMechanism

Distribution shift in time series pose a significant challenge to the accuracy of time series forecasting, as they often exhibit temporal fluctuations that can seriously impede prediction performance. Additional analysis on distribution shift in time series is provided in Appendix 1 . Therefore, we further introduce a temporal intervention design into the temporal learning. Specially, we first denote the timestamps in a lookback window of length L as {t1, t2, · · · , tL}. Then, we let the intervention process accomplished by randomly replacing temporal variant patterns. Formerly, for i, j ∈ [1,L], we generate an intervened representation:

where sti I and sti V represent the intervened invariant and variant patterns respectively. Then, the intervened invariant and variant patterns are added up as intervened representation as: hti inv = sti I + γsti V , where htinv is the intervened sample at t-th timestamp and γ is a hyperparameter to balance the intervened invariant and variant patterns for forecasting.

1 https://github.com/xhh39/DIAN

3.4 Joint Optimization

In time series forecasting, the classic objective function is set as the Mean Absolute Error,

Idealy, if the essential information is totally extracted and there is no distribution shift mixed with the information, we only use invariant patterns to make prediction as: ˆYt+1:t+H = f(P t I ,RtI ), where f is a fully connected layer predictor. In order to adapt to the distribution shift that often occur in time series forecasting tasks, we introduce both invariant and variant patterns to make prediction as: ˆYt+1:t+H = f(P t I ,RtI , P tV ,RtV ). In section 3.3, we propose the temporal intervention mechanism and obtain intervened samples. In order to simulate various distribution shift that may occur and further adapt to distribution shift, we use the obtained intervened samples to make prediction as: ˆYinv t+1:t+H = f(stI , stV ). Then, new joint optimization with regard to temporal intervention mechanism can be formalized as:

where Lfinal is minimized to exploit invariant and variant patterns while discovering and adapting to the distribution shift ahead of time; λ is a hyperparameter to balance between two objectives.

4 Experiments

We conduct extensive experiments on five real-world time series benchmarks and compare our model with many effective time series forecasting models (including state-of-the-art graph neural network-based models) to validate the performance of our model.

4.1 Experimental Setup

Datasets

We evaluate our proposed method on five real-world datasets and use the min-max normalization to normalize all these datasets. Except for the COVID-19 dataset, we split the datasets into training, validation, and test sets with the ratio of 7:2:1 in a chronological order. For the COVID-19 dataset, the ratio is 6:2:2 because of the limitation of data scale in temporal dimension. More detailed information about the datasets is provided in Appendix 1 .

Evaluation

To compare the performance of different forecasting models, we deploy two widely used evaluation metrics: 1) Root Mean square Error. 2) Mean Absolute Error. More detailed information about the evaluation is provided in Appendix 1 .

Implementation

We use PyTorch to implement our model and baselines. All models were evaluated on a Linux server with one RTX 3090 GPU. We use MAE (Mean Absolute Errors) as the loss function and the Adam Optimizer with a learning rate of 1e-3 with proper early stopping. For the main experiment, we fix the lookback window length as 12 and the horizon window length as 12. More detailed information about the implementation is provided in Appendix 1 .

Baselines

To verify the effectiveness of our model, we compared it with several representative baseline methods on the five datasets. The baseline methods mainly include: (1) classic method VAR [Watson, 1993]; (2) deep learning-based models such as SFM [Zhang et al., 2017], LSTNet [Lai et al., 2018], TCN [Bai et al., 2018], DeepGLO [Sen et al., 2019], and CoST [Woo et al., 2022b]; GNN-based models such as GraphWaveNet [Wu et al., 2019], StemGNN [Cao et al., 2020b], MTGNN [Wu et al., 2020], and AGCRN [Bai et al., 2020]; Transformer-based models including Reformer [Kitaev et al., 2020], Informer [Zhou et al., 2021], Autoformer [Wu et al., 2022] and FEDformer [Zhou et al., 2022b]. In addition, we also compare DIAN with SOTA TAMP-S2GCNets [Chen et al., 2021]. To compare fairly, our experiments for baselines and our model are under the same experimental settings. Due to spatial constraints, more baseline implementation details can be found in Appendix 1 .

4.2 Overall Performance

Table 1 demonstrates the overall performance of thirteen baselines and DIAN. It is obvious that DIAN achieves a new state-of-the-art on all datasets. Compared with the bestperforming across all datasets, DIAN makes an improvement of 11.3% in MAE and 7.9% in RMSE. Notably, we find that on COVID-19 dataset, transformer-based models like Reformer achieve a competitive performance because they excel at capturing temporal dependencies of time series. However, they are not as good as the GNN-based models at capturing spatial dependencies. As a result, on Wiki, Traffic, and ECG datasets, GNN-based models like AGCRN and MTGNN achieve a more promising performance. DIAN not only learns spatial and temporal dependencies simultaneously, but also integrates invariant and variant patterns reasonably, therefore, it outperforms the baseline models.

4.3 Parameters andModel Analysis

Efficiency Analysis

In order to verify the high efficiency of DIAN, we investigate the parameter volumes and training time costs of DIAN and some lightweight baseline models (StemGNN, AGCRN, GraphWaveNet, MTGNN) on Traffic and Wiki datasets. Table 2 shows the comparison of parameter volumes and average training time costs over five rounds of experiments. Obviously, we can find that DIAN always have the lowest volume of parameters among the comparative models. Compared with the baseline models, DIAN achieves a reduction of more than 70.7% and 50.2% in parameter volumes on Traffic and Wiki datasets respectively. In addition, DIAN runs much faster than other baselines by at least 56.3% and 46.9% on Traffic and Wiki datasets respectively. The reason is that DIAN captures relationships between each series/timestamp and the overall representation instead of learning relationships among timestamps/series.

Ablation Study

In this section, we conduct ablation studies to verify the effectiveness of the proposed Spatial Decoupled Invariant-Variant Learning and Temporal Augmented Invariant-Variant Learning in DIAN on Solar and COVID-19 datasets. Specifically, w/o spatial is DIAN without Spatial Decoupled InvariantVariant Learning and w/o temporal represents DIAN without Temporal Augmented Invariant-Variant Learning. We verify the necessity of capturing inter-series (spatial) correlations and intra-series (temporal) dependencies by removing these two parts respectively. The results presented in Table 3 demonstrate that the removal of either spatial or temporal components leads to a decline in model performance, underscoring the effectiveness of each component.

4.4 Visualization

To gain a better understanding of DIAN which can distinguish between invariant patterns and variant patterns in both temporal and spatial dimensions in multivariate time series forecasting. We conduct visualization experiments on the COVID-19 dataset.

Visualization of the Spatial Invariant and Variant Attention Score Learned by DIAN

To demonstrate the capability of Spatial Decoupled InvariantVariant Learning in capturing both stable and potentially volatile spatial relationships, we employed the COVID-19

dataset and visualized the variations of the invariant attention scores and variant attention scores using line graphs within the same lookback window but across different timestamps. Specifically, we computed the differences between each timestamp's attention score and the previous timestamp's attention score (excluding the first timestamp), resulting in (L - 1) difference values. Taking the average of these difference values provided a measure of the intensity of spatial relationship changes within the lookback window for each series. The results are illustrated in Figure 2b, where the blue line represents the variation of the variant attention score, and the orange line represents the variation of the invariant attention score. From the results, it is evident that the variant attention score exhibits more pronounced changes compared to the invariant attention score across all 55 series. Therefore, we can conclude that Spatial Decoupled Invariant-Variant Learning can effectively capture and distinguish stable spatial relationships from potentially unstable ones.

Visualization of Temporal Invariant and Variant Pattern Learned by DIAN

To validate the conformity of temporal invariant and variant patterns with our proposed viewpoint, which states that the variation amplitude of invariant patterns is smaller and more regular, while variant patterns exhibit drastic changes, we utilized the COVID-19 dataset as an example. We visualized the variance of each series within a lookback window using a line graph, as shown in Figure 2a. The blue line represents the variance of variant patterns within a lookback window, while the orange line represents the variance of invariant patterns within the same window. The x-axis denotes the series number. From the results, it is evident that the variance of variant patterns consistently exceeds that of invariant patterns. Thus, we can conclude that through the Temporal Invariant-Variant Attention mechanism, we have successfully decomposed the series into invariant and variant components. The invariant pattern captures the relatively stable part, while the variant pattern captures the part that exhibits significant changes with variations in the environment.

5 RelatedWork

5.1 Time Series Forecasting

Time series forecasting (TSF) has gained significant attention due to its practical significance. In recent years, notable advancements have been made in time series forecasting research. DeepAR [Flunkert et al., 2020] employs an RNN structure for accurate predictions. Nbeats [Oreshkin et al., 2019] achieved remarkable improvements by utilizing residual computation and multiple fully connected layers, producing interpretable outputs. Moreover, transformer-based models have gained popularity in the field of time series forecasting, such as Informer [Zhou et al., 2021] and Autoformer [Wu et al., 2022]. In addition, there have been some recent works that have achieved new heights in the field of time series forecasting [Yi et al., 2024b; Yi et al., 2024a; Hu et al., 2023; Ren et al., 2022; Zhang et al., 2024].

5.2 Graph Convolutional Networks Based Forecasting Model

GCN, a type of GNN specialized for graph-structured data, has wide applications in node classification, link prediction, and graph classification [Wu et al., 2021]. It extracts relationship features between nodes, yielding satisfactory results. Recently, graph learning has been employed in time series forecasting to extract relationships in node or series data. However, traditional GCN relies on a predefined graph as input, which is difficult to obtain and less suitable for dynamic data like traffic flow [Bai et al., 2020]. AGCRN addresses this challenge by capturing node-specific patterns and automatically inferring inter-dependencies among traffic series, which has demonstrated excellent performance in traffic forecasting [Bai et al., 2020]. Nevertheless, the stability of the adjacency matrix learning in AGCRN can be improved further. Research on temporal graphs has been rapidly advancing recently, and many methods are worth considering and drawing inspiration from [Dong et al., 2024; Dong et al., 2023].

6 Conclusion Remarks

In this paper, we propose a multivariate time series forecasting model, Decoupled Invariant Attention Network (DIAN), to decouple invariant and variant patterns in both spatial and temporal dimensions and combine them reasonably for forecasting. First, we use Spatial Invariant-Variant Attention to calculate spatial invariant and variant attention scores for capturing inter-series correlations and propose Decoupled Invariant-Variant Convolution to get spatially-learned representations. Second, we propose Temporal InvariantVariant Attention to capture temporal correlations and Temporal Intervention Mechanism to create intervened samples to simulate potential distribution shift. Extensive experiments on five real-world datasets demonstrate that our model can achieves state-of-the-art performances with higher efficiency and fewer parameters.

Acknowledgments

This research is funded by the Science and Technology Development Fund (FDCT), Macau SAR (file no. 0123/2023/RIA2, 001/2024/SKL), the Start-up Research Grant of University of Macau (File no. SRG2021-00017IOTSC).

Contribution Statement

In this work, Haihua Xu and Wei Fan have an equal contribution, specifically, they led the project, provided theoretical support, and were responsible for the overall model design, code implementation, experimental design and paper writing. Kun Yi provided guidance in solving complex problems. Pengyang Wang provided valuable feedback on the paper drafts. All authors reviewed and approved the final manuscript.