CASA: CNN Autoencoder-based Score Attention for Efficient Multivariate Long-term Time-series Forecasting

1 Introduction

Multivariate Long-Term Time Series Forecasting (LTSF) plays a pivotal role in real-world applications, including weather prediction, traffic flow analysis [Ji et al., 2023], and solar energy forecasting [Lai et al., 2018]. LTSF has seen rapid advancements driven by the emergence of Transformer model [Vaswani, 2017]. Subsequent Transformer variants have further demonstrated the effectiveness of the multi-head self-attention mechanism in capturing temporal dependencies and cross-dimensional correlations. [Zhou et al., 2021; Wu et al., 2021; Liu et al., 2022b; Zhou et al., 2022; Liu et al., 2022c; Zhang and Yan, 2023; Nie et al., 2022].

Despite numerous efforts, Transformer-based models have not consistently outperformed CNN- or MLP-based architectures in the LTSF domain [Wu et al., 2023; Ekambaram et al., 2023; Das et al., 2023; Zeng et al., 2023]. Notably, DLinear

† Corresponding author.

[Zeng et al., 2023], a model constructed with simple linear layers, raises critical questions about the effectiveness and necessity of Transformer-based architectures in this field, especially considering their demanding computational and time resources. To address the aforementioned challenges, diverse tokenization techniques have been introduced into the core architecture of Transformer-family models [Liu et al., 2023; Nie et al., 2022]. We evaluate the effectiveness of three models, each employing a distinct tokenization technique, as illustrated in Figure 2. Figure 1 demonstrates that channel-wise tokenization achieves the best performance among the three, as highlighted in [Yu et al., 2024]. However, it still leads to rapid saturation during training. Moreover, computational cost and memory usage remain significant challenges. Our objective is to develop a more efficient model that delivers superior predictive performance while mitigating these drawbacks. We tackle these issues by refining the self-attention mechanism, the cornerstone of Transformer-based models.

In this paper, we propose CNN Autoencoder-based Score Attention (CASA), a simple yet novel module that addresses the aforementioned gap by effectively capturing correlations and avoiding saturation, thereby facilitating consistent learning, designed to serve as an alternative to the conventional self-attention mechanism. We retain the vanilla Transformer encoder while substituting the attention mechanism with a CNN-based module. CASA approximates QKT √dk rather than directly calculating it, as done in traditional multi-head selfattention. This design addresses a critical limitation of conventional methods by sufficiently accounting for significant correlation between variates in the calculation of attention scores (see Section 3.3 for details).

Our key contributions are as follows:

• We present a simple yet effective CNN Autoencoderbased Score Attention (CASA) module as an alternative to self-attention. It scales linearly with the number of variates, input length, and prediction length. Compared to Transformer-based variants, CASA reduces memory usage by up to 77.7% and improves computational speed by 44.0%, depending on the dataset.
• CASA can be agnostically integrated into Transformer models, regardless of the tokenization technique used (e.g., point-wise, patch-wise, or channelwise). To the best of our knowledge, this is the first module validated across individual tokenization techniques, successfully enhancing cross-dimensional information capture and improving prediction performance.
• CASA achieved first place in 54 out of 64 metrics across 8 real-world datasets and ranked highest in 14 out of 16 average metrics, establishing itself as a highly competitive solution for multivariate LTSF.

2 RelatedWorks

Transformer variants The vanilla Transformer model [Vaswani, 2017], widely recognized for its success in natural language processing, has also achieved notable advancements in time-series forecasting. Diverse Transformer variants have been introduced to enhance forecasting performance, which can be broadly grouped into three approaches. The first approach modifies the traditional self-attention mechanism with alternatives by incorporating specialized modules, or pyramidal attention [Liu et al., 2022b], to reduce memory requirements while capturing multi-resolution representations.

Additional modifications, including the trapezoidal architecture [Zhou et al., 2021] and de-stationary attention [Liu et al., 2022c], aim to improve robustness and address issues like over-stationarization. The second approach leverages frequency-domain techniques, such as Fast Fourier Transform (FFT) [Zhou et al., 2022] and auto-correlation mechanisms [Wu et al., 2021], to better extract temporal patterns. The third approach introduces hierarchical encoderdecoder frameworks [Zhang and Yan, 2023] with routing mechanisms to capture cross-dimensional information, although these methods sometimes encounter challenges such as slower learning and higher computational demands.

Alternatives of Transformers While Transformer variants have significantly advanced the time-series forecasting domain, CNN-based models present promising alternatives. These approaches include methods that model segmented signal interactions [Liu et al., 2022a] and those that reshape 1D time-series data into 2D tensors [Wu et al., 2023], enabling the capture of both inter-period and intra-period dynamics. Similarly, linear models [Zeng et al., 2023] have demonstrated simplicity while achieving high prediction performance. However, these methods generally fall short of explicitly addressing cross-dimensional interactions, which are crucial for improving multivariate time-series forecasting. Other methods have been developed to modify aspects of the Transformer architecture, particularly focusing on tokenization techniques. For instance, PatchTST [Nie et al., 2022] segments input data into patches to extract local information within individual variates, while iTransformer [Liu et al., 2023] treats each variate as a token, enabling the self-attention mechanism to capture multivariate correlations. However, a common drawback of these methods is their reliance on self-attention, which demands substantial computational resources. Furthermore, their prediction performance remains suboptimal, with both models struggling to effectively capture multivariate dependencies, which critically impacts their predictive accuracy. In contrast, our proposed model, CASA, addresses these challenges by significantly reducing resource consumption while achieving superior prediction performance, offering an efficient and effective alternative to traditional self-attention-based approaches.

3 Method

This section provides an overview of CASA, including the motivation behind its architecture, a detailed explanation of its structure, and a complexity analysis. Section 3.1, we formulate the problem we aim to solve and define the necessary notations. Section 3.2 provides an explanation of the overall architecture. Section 3.3 discusses the limitations of conventional Transformers in capturing cross-dimensional interactions. Section 3.4 introduces CNN Autoencoder-based Score Attention (CASA), an improved self-attention module designed to address these issues. We confirm theoretical efficiency of CASA through complexity analysis.

3.1 ProblemFormulation

In multivariate LTSF, given an input length L, the number of variates N, the number of layers M, and a prediction length H, denote the input and the output X ∈ RN×L and Y ∈ RN×H respectively. The hidden dimension is denoted as D, and the intermediate feature after embedding is represented as Zi ∈ RN×D (i ∈ {0, 1, · · ·M}). Since this is not a univariate time-series forecasting problem, the input is consistently represented as a matrix.

3.2 Overall Architecture

The overall framework is depicted in Figure 3. Following prior works [Liu et al., 2023; Yu et al., 2024], we adopt a channel-wise tokenization approach, utilizing the vanilla Transformer encoder as the backbone. To address the challenges outlined in Section 3.3, we enhance the Transformer by replacing only the attention mechanism with a 1D CNN Autoencoder Score Attention (CASA). Although this modification alters only a very small portion of the overall model, it effectively reduces computational cost and memory usage while improving prediction performance.

The input X is linearly embedded to produce the intermediate feature Z0. The final feature ZM, obtained by passing Z0 through M CASA blocks, is then fed into the predictor where the output becomes Y . The pipeline is summarized by the following equation.

3.3 Limitation of Self-Attention

We demonstrate that the existing self-attention mechanism does not sufficiently consider cross-dimension information when embedding queries and keys. the structure of the selfattention mechanism in the conventional Transformer is as follows (fj: affine map):

At this point, since fj is an affine map, queries and keys are computed through the following operations:

Proposition 1. Query and key embeddings are variateindependent operations in the conventional Transformer using channel-wise tokenization.

Proposition 2. Query and key embeddings are timeindependent operations in the conventional Transformer using point-wise and patch-wise tokenization.

Proposition 1 and Proposition 2 imply that each tokenization method does not consider the correlation between variates or time points when embedding the key and query. Since multivariate time series exhibit correlations both between variates and across time points, this reduces the potential of the Transformer architecture. Especially, based on the Proposition 1, the Transformer using channel-wise tokenization does not directly incorporate cross-dimensional information when embedding the r-th variate into queries and keys. In other words, the self-attention mechanism embeds queries and keys through variate-independent feature refinement operations and then computes the attention map using QiKT i √dk . In the LTSF domain, tokens (i.e., variates) exhibit inherent correlations (see Section 4.4), which reduce the effectiveness of variate-independent operations during feature refinement. This limitation can hinder performance in the multivariate LTSF domain, where capturing correlations between variates is important.

3.4 CNNAutoencoder-based Score Attention

To address the issue posed by the self-attention mechanism's variate-independent operation, we treat each variate as a channel and apply a convolution instead of using the affine map from the conventional self-attention mechanism. This approach ensures that the operation becomes variatedependent. In more detail, instead of directly computing QiKT i √dk , we designed a score network Score to approximate this operation using 1D CNN Autoencoder. The modified structure of self-attention is as follows (f: affine map, ⊛: element-wise product):

By approximating QiKT i √dk via a CNN architecture instead of direct computation, we reduce complexity compared to the affine map (see the paragraph below), addressing the limitations of self-attention. This facilitates the development of a linear complexity model with enhanced performance (Section 4.1). To explain the score network in more detail, we adopted an inverted bottleneck autoencoder structure, inspired by previous research [Wilson et al., 2016; Bengio et al., 2013], which demonstrated that embedding low-dimensional features into a high-dimensional latent space can improve expressiveness. In summary, we leverage CNN operations to incorporate information across all variates, embedding them into a high-dimensional feature space before compressing the channels to retain only essential cross-variable information. Consequently, despite its simple architecture, the proposed module outperforms existing self-attention mechanisms while maintaining efficiency, constituting a significant contribution.

Complexity Analysis CASA is an efficient algorithm that exhibits linear complexity not only with respect to the number of tokens, i.e., the number of variates N, but also with respect to the input length L and prediction length H. The detailed complexity calculation is as follows. Let the kernel size of the score network be denoted as k. The complexities of the Reversible Instance Normalization (RevIN), series embedding, and MLP are O(NL), O(NLD), and O(ND2 ), respectively. Additionally, the complexity of the score network, composed of CNN autoencoder blocks, is O(NkD2 ). The predictor has a complexity of O(NDH). Thus, the overall complexity of our method is O(NL + NLD + ND2 + NkD2 +NDH), which scales linearly with respect to N, L, and H. Since the hidden dimension and kernel size are constants in the algorithm, they can be ignored. Consequently, N is dominated by NL and NH(Since L and H typically take on large values in LTSF), leading to the final complexity summarized in Table 1. In addition, the results for memory usage and inference time are included in the table, empirically demonstrating the efficiency of CASA. For details of the implementation, refer to the Appendix D.

4 Experiments

Dataset We conduct our comprehensive experiment on 8 benchmark datasets [Zhou et al., 2021], such as Traffic, ETT series including 4 subsets (ETTh1, ETTh2, ETTm1, ETTm2), Weather, Solar, Electricity. More detailed information on Dataset is described in Appendix A.

Baselines We chose totally 8 comtemporarlly baseline models, including SOFTS [Han et al., 2024], iTransformer [Liu et al., 2023], PatchTST [Nie et al., 2022] , TSMixer

[Ekambaram et al., 2023], Crossformer [Zhang and Yan, 2023], TiDE [Das et al., 2023] , DLinear [Zeng et al., 2023], FEDformer [Zhou et al., 2022].

Setup Our comprehensive experiments results are based on MSE (Mean Squared Error) and MAE (Mean Absolute Error) metrics. Our main experiments are conducted on the conditions with L = 96 and the H ∈ {96, 192, 336, 720}.

4.1 Multivariate Forecasting Results

The main results are presented in Table 2, where red-bold text indicates the best score and blue-underlined text represents the second-best score. CASA demonstrates the lowest MSE and MAE losses across 8 benchmark datasets, surpassing the previous state-of-the-art model, SOFTS, by a substantial margin. Additionally, the second-lowest performance scores exhibit a smaller gap from the best score compared to the others. Notably, the proposed model showcases its robustness on relatively large datasets, such as Traffic, Weather, and Solar, highlighting its ability to capture complex correlations, which significantly enhances the model's predictive performance.

Figure 4 visualizes the prediction performance of weather dataset from CASA, SOFTS, iTransformer, and PatchTST models against the ground truth. CASA shows the closest alignment with the label, while iTransformer shows similar tendency along the ground truth with large deviation. SOFTS and PatchTST prediction slightly detours the label. The full results of different prediction lengths and the visualization results on the rest of the datasets are demonstrated in Appendix B and Appendix C, respectively.

4.2 Superiority Analysis of CASA

Replacing Self-Attention with CASA To ensure the proposed model's adaptability to diverse tokenization techniques, we integrate CASA into various Transformer variants including the vanilla Transformer, PatchTST, and iTransformer. Especially for the vanilla Transformer, we only use the encoder architecture to appropriately compare the effectiveness of CASA.

The results on seven benchmark datasets are presented in Table 3. With the exception of two specific cases-ETTm2 compared to PatchTST and ETTh2 compared to the vanilla Transformer-CASA consistently enhances the performance of the original and variants models, achieving improvements in 40 out of 42 results across the benchmarks. These findings not only demonstrate that replacing self-attention with CASA significantly boosts forecasting accuracy, but also highlight its flexibility and adaptability, as it can seamlessly integrate with diverse tokenization techniques, making it a versatile enhancement for Transformer-based architectures.

Robustness of CASA under varying conditions To validate the robustness of CASA with respect to input length and the number of variates, we conducted experiments on the ETTm1, Electricity, and Traffic datasets, which contain 7, 321, and 862 variates, respectively, with prediction lengths ranging from 48 to 720. As shown in Figure 5, the performance of models utilizing non-channel-wise tokenization declines as the number of variates increases. Specifically, PatchTST experiences a significant performance drop on the Traffic dataset, recording the highest MSE losses. In contrast, models such as iTransformer and SOFTS, which employ channel-wise tokenization, demonstrate greater resilience to increases in the number of variates. However, both models exhibit elevated MSE losses on the ETTm1 dataset, while their performance improves on the Electricity and Traffic datasets. In comparison, our proposed model maintains stable MSE losses across all three datasets, achieving notably lower MSE losses on the ETTm1 dataset. This underscores CASA's ability to deliver high predictive accuracy under varying conditions, ensuring consistent performance even as the input data length increases.

4.3 Model Efficiency Analysis

In this section, we empirically validate the efficiency of CASA, as theoretically outlined in Section 3.4. For comparison, we use iTransformer and SOFTS as baselines. Figure 6 (a) depicts memory usage, revealing that CASA exhibits linear complexity and effectively leverages practical computational resources. This performance is comparable to SOFTS, which also demonstrates similar complexity but increases memory usage significantly. Notably, CASA significantly outperforms iTransformer, which suffers from quadratic memory growth. Figures 6 (b) and (c) present memory footprints, inference time, and MSE for the Traffic and Electricity datasets, using a batch size of 16 and input/inference sequence lengths of 96. CASA consumes fewer computational resources than Transformer variants such as iTransformer, Crossformer, and PatchTST. Regarding MSE, CASA achieves the second-lowest value on Traffic and the lowest on Electricity, all while maintaining fast inference and efficient resource usage. Although CASA slightly exceeds TSMixer in memory usage, it delivers stronger overall performance, striking an effective balance between accuracy and resource efficiency.

4.4 Investigating Cross-Dimensional Interactions: A Correlation Matrix Analysis of CASA

We evaluate CASA's capacity to capture cross-dimensional interactions by examining the correlation matrices derived from each model's predictions (i.e., correlations among all variates) on the Weather Dataset, comparing outcomes from CASA, SOFTS, iTransformer, and PatchTST against the Ground Truth. The Ground Truth correlation matrix shows distinct positive and negative correlation blocks, indicating a clear spatial structure with a grid-like arrangement (Figure 7 (a)). Furthermore, kernel density estimation (KDE) using a Gaussian kernel reveals that correlation values are mostly concentrated in the positive domain (Figure 7 (b)), suggesting many variates rise or fall in sync. Notably, CASA's correlation matrix most closely approximates the Ground Truth, as evidenced by the lowest MSE loss between their probability density functions (PDFs) (Figure 7 (b)). This underscores CASA's ability to preserve intricate relationships essential for capturing temporal and spatial dependencies in weather data. For a more comprehensive evaluation, we compare each model's correlation matrix to the Ground Truth using Mean Squared Error (MSE), cosine similarity, and the Structural Similarity Index Measure (SSIM). CASA outperforms all other models across these metrics, validating its effectiveness in capturing cross-dimensional interactions (Table 4).

5 Conclusion

In this study, we introduce the CASA model, which demonstrates remarkable effectiveness and solid predictive performance, as confirmed by a broad range of experiments. By delivering state-of-the-art results while requiring notably fewer computational resources and less processing time than existing approaches. Moreover, its CNN-based autoencoder module successfully captures cross-dimensional interactions throughout the compress-and-decompress procedure, which contributes to its outstanding performance. Additionally, CASA shows strong potential as a versatile alternative to conventional attention mechanisms in various Transformer configurations, remaining unaffected by different tokenization methods. This further highlights its adaptability, practicality, and efficiency across diverse use cases.

Contribution Statement

Minhyuk Lee and HyeKyung Yoon contributed equally to this manuscript and are recognized as joint first authors.

Acknowledgments

This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [NO.RS2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University)] and the National Research Foundation of Korea (NRF) [RS-2024-00421203, RS-202400406127, RS-2021-NR059802]