FreEformer: Frequency Enhanced Transformer for Multivariate Time Series Forecasting

1 Introduction

Multivariate time series forecasting holds significant importance in real-world domains such as weather [Wu et al., 2023b], energy [Zhou et al., 2021], transportation [He et al., 2022] and finance [Chen et al., 2023]. In recent years, various deep learning models have been proposed, significantly pushing the performance boundaries. Among these models, Recurrent Neural Networks (RNN) [Salinas et al., 2020], Convolutional Neural Networks (CNN) [Bai et al., 2018;

∗ Corresponding Author

Wu et al., 2023a], LLM [Zhou et al., 2023; Jin et al., 2021], Multi-Layer Perceptrons (MLP) [Zeng et al., 2023; Xu et al., 2023], Transformers-based methods [Nie et al., 2023; Liu et al., 2024a; Wang et al., 2024c] have demonstrated great potential due to their strong representation capabilities.

In recent years, frequency-domain-based models have been proposed and have achieved great performance [Yi et al., 2024c; Xu et al., 2023], benefiting from the robust frequency domain modeling. As shown in Figure 1, frequency spectra exhibit strong consistency across different spans of the same series, making them suitable for forecasting. Most existing frequency-domain-based works [Yi et al., 2024a] rely on linear layers to learn frequency-domain representations, resulting in a performance gap. Frequency-domain Transformer-based models remain under-explored. Recently, Fredformer [Piao et al., 2024] applies the vanilla Transformer to patched frequency tokens to address the frequency bias issue. However, the patching technique introduces additional hyper-parameters and undermines the inherent global perspective [Yi et al., 2024c] of frequency-domain modeling.

In this paper, we adopt a simple yet effective approach by applying the Transformer to frequency-domain variate tokens for representation learning. Specifically, we embed the entire frequency spectrum as variate tokens and capture cross-variate dependencies among them. This architecture offers four main advantages: 1) As shown in Section 3, simple frequency-domain operations can correspond to complex temporal operations [Yi et al., 2024c]; 2) Multivariate correlations typically exists (Figure 1), and learning these dependencies facilitates forecasting; 3) Minimal correlations among frequency points [Wang et al., 2024a] hinder the efficacy of cross-frequency dependency learning (Table 5); 4) The permutation invariance of the attention mechanism aligns naturally with the order insensitivity of variates.

Furthermore, we observe that for the frequency-domain representation, the attention matrix of vanilla attention often exhibits a low-rank characteristic, which reduces the diversity of representations. To address this issue, we propose a general solution: adding a learnable matrix to the original softmax attention matrix, followed by row-wise normalization. We term this approach enhanced attention and name the overall model FreEformer. Despite its simplicity, the enhanced attention mechanism is proven effective both theoretically and empirically. The main contributions of this work are summarized as follows:

• This paper presents a simple yet effective model, named FreEformer, for multivariate time series forecasting. FreEformer achieves robust cross-variate representation learning using the enhanced attention mechanism.
• Theoretical analysis and experimental results demonstrate that the enhanced attention mechanism increases the rank of the attention matrix and provides greater flexibility for gradient flow. As a plug-in module, it consistently enhances the performance of existing Transformer-based forecasters.
• Empirically, FreEformer consistently achieves state-ofthe-art forecasting performance across 18 real-world benchmarks spanning diverse domains such as electricity, transportation, weather, healthcare and finance.

2 RelatedWorks

2.1 Transformer-Based Forecasters

Classic works such as Autoformer [Wu et al., 2021], Informer [Zhou et al., 2021], Pyraformer [Liu et al., 2022a], FEDformer [Zhou et al., 2022b], and PatchTST [Nie et al., 2023] represent early Transformer-based time series forecasters. iTransformer [Liu et al., 2024a] introduces the inverted Transformer to capture multivariate dependencies, and achieves accurate forecasts. More recently, research has focused on jointly modeling cross-time and cross-variate dependencies [Zhang and Yan, 2023; Wang et al., 2024c; Han et al., 2024]. Leddam [Yu et al., 2024] uses a dualattention module for decomposed seasonal components and linear layers for trend components. Unlike previous models in the time domain, we shift our focus to the frequency domain to explore dependencies among the frequency spectra of multiple variables for more robust representations.

2.2 Frequency-Domain Forecasters

Frequency analysis is an important tool in time series forecasting [Yi et al., 2023]. FEDformer [Zhou et al., 2022b] performs DFT and sampling prior to Transformer. DEPTS [Fan et al., 2022] uses the DFT to capture periodic patterns for better forecasts. FiLM [Zhou et al., 2022a] applies Fourier analysis to preserve historical information while mitigating noise. FreTS [Yi et al., 2024c] employs frequency-domain MLPs to model channel and temporal dependencies. FourierGNN [Yi et al., 2024b] transfers GNN operations from the time domain to the frequency domain. FITS [Xu et al., 2023] applies a low-pass filter and complex-valued linear projection in the frequency domain. DERITS [Fan et al., 2024] introduces a Fourier derivative operator to address non-stationarity. Fredformer [Piao et al., 2024] addresses frequency bias by dividing the frequency spectrum into patches. FAN [Ye et al., 2024] introduces frequency adaptive normalization for nonstationary data. In this work, we adopt a simple yet effective Transformer-based model to capture multivariate correlations in the frequency domain, outperforming existing methods.

2.3 Transformer Variants

Numerous variants of the vanilla Transformer have been developed to enhance efficiency and performance. Informer [Zhou et al., 2021] introduces a ProbSparse self-attention mechanism with O(𝑁log𝑁) complexity. Flowformer [Wu et al., 2022] proposes Flow-Attention, achieving linear complexity based on flow network theory. Reformer [Kitaev et al., 2020] reduces complexity by replacing dot-product attention with locality-sensitive hashing. Linear Transformers, such as FLatten [Han et al., 2023] and LSoftmax [Yue et al., 2024], achieve linear complexity by precomputing K 𝑇 V and designing various mapping functions. FlashAttention [Dao et al., 2022] accelerates computations by tiling to minimize GPU memory operations. LASER [Surya Duvvuri and Dhillon, 2024] mitigates the gradient vanishing issue using exponential transformations. In this work, we focus on the low-rank issue and adopt a simple yet effective strategy by adding a learnable matrix to the attention matrix. This improves both matrix rank and gradient flow with minimal modifications to the vanilla attention mechanism.

3 Preliminaries

The discrete Fourier transform (DFT) [Palani, 2022] converts a signal x ∈ R 𝑁 into its frequency spectrum F ∈ C 𝑁 . For 𝑘 = 0, 1, · · · , 𝑁 - 1, we have

Here, 𝑗 denotes the imaginary unit. For a real-valued vector x, F [𝑘] is complex-valued and satisfies the property of Hermitian symmetry [Palani, 2022]: F [𝑘] = (F [𝑁 - 𝑘]) ∗ for 𝑘 = 1, · · · , 𝑁 - 1, where (·) ∗ denotes the complex conjugate. The DFT is a linear and reversible transform, with the inverse discrete Fourier transform (IDFT) being:

(c) Enhanced Attention Mechanism

Figure 2: Overall structure of the FreEformer. We leverage the frequency spectrum to capture temporal patterns and employ an enhanced Transformer to model dependencies among multivariate spectra. The enhanced Transformer introduces a learnable matrix to the attention mechanism, which, as shown through theoretical analysis, addresses potential low-rank issues and improves gradient flow.

Linear projections in the frequency domain are widely employed in works such as FreTS [Yi et al., 2024c] and FITS [Xu et al., 2023]. The following theorem establishes their equivalent operations in the time domain.

Theorem 1 (Frequency-domain linear projection and timedomain convolutions). Given the time series x ∈ R 𝑁 and its corresponding frequency spectrum F ∈ C 𝑁 . Let W ∈ C 𝑁×𝑁 denote a weight matrix and b ∈ C 𝑁 a bias vector. Under these definitions, the following DFT pair holds:

where

Here, ⊛ denotes the circular convolution, and ⊙ represents the Hadamard (element-wise) product. The notation [·, ·] indicates the concatenation of two vectors. diag(W, 𝑖) ∈ C 𝑁-|𝑖 | extracts the 𝑖-th diagonal of W. M𝑖 (x) represents the 𝑖-th modulated version of x, with M0 (x) being x itself.

We provide the proof of this theorem in Section A of the appendix. Theorem 1 extends Theorem 2 from FreTS [Yi et al., 2024c] and demonstrates that a linear transformation in the frequency domain is equivalent to the sum of circular convolution operations applied to the series and its modulated versions. This equivalence highlights the computational simplicity of performing such operations in the frequency domain compared to the time domain.

4 Method

In multivariate time series forecasting, we consider historical series within a lookback window of 𝑇, each timestamp with 𝑁 variates: x = {x1, · · · , x𝑇} ∈ R 𝑁×𝑇 . Our task is to predict future 𝜏 timestamps to closely approximate the ground truth y = {x𝑇+1, · · · , x𝑇+𝜏} ∈ R 𝑁×𝜏 .

4.1 Overall Architecture

As shown in Figure 2, FreEformer employs a simple architecture. First, an instance normalization layer, specifically RevIN [Kim et al., 2021], is used to normalize the input data and de-normalize the results at the final stage to mitigate nonstationarity. The constant mean component, represented by the zero-frequency point in the frequency domain, is set to zero during normalization. Subsequently, a dimension extension module is employed to enhance the model's representation capabilities. Specifically, the input x is expanded by a learnable weight vector 𝜙𝑑 ∈ R 𝑑 , yielding higherdimensional and more expressive series data: ¯x = x × 𝜙𝑑 ∈ R 𝑁×𝑑×𝑇 . We refer to 𝑑 as the embedding dimension.

Frequency-Domain Operations Next, we apply the Discrete Fourier Transform (DFT) to convert the time series ¯x into its frequency spectrum along the temporal dimension:

where Re(·) and Im(·) represent the real and imaginary parts, respectively. Due to the conjugate symmetry property of the frequency spectrum of a real-valued signal, only the first ⌈(𝑇 + 1)/2⌉ elements of the real and imaginary parts need to be retained. Here, ⌈·⌉ denotes the ceiling operation.

To process the real and imaginary parts, common strategies include employing complex-valued layers [Yi et al., 2024c; Xu et al., 2023], or concatenating the real and imaginary parts into a real-valued vector and subsequently projecting the results back [Piao et al., 2024]. In this work, we adopt a simple yet effective scheme: processing these two parts independently. As shown in Table 1, this dual-branch scheme yields better performance for FreEformer.

After flattening the last two dimensions of the real and imaginary parts and projecting them into the hidden dimension 𝐷, we construct the frequency-domain variate tokens ˜ Re, ˜ Im ∈ R 𝑁×𝐷 . These tokens are then fed into 𝐿 stacked Transformer blocks to capture multivariate dependencies among the spectra. Subsequently, the tokens are projected back to the lookback length. The real and imaginary parts are then regrouped to reconstruct the frequency spectrum. Then, the time-domain signal ˜x is recovered using the IDFT. The entire process is summarized as follows:

In the above equation, the final step is implemented via the irfft function in PyTorch to ensure real-valued outputs.

PredictionHead A shortcut connection is applied to sum ˜x with the original ¯x. Finally, a flatten layer and a linear head are used to ensure the output matches the desired size. The final result is obtained through a de-normalization step:

4.2 Enhanced Attention

In the Transformer block, as shown in Figure 2(b), we first employ the attention mechanism to capture cross-variate dependencies. Then the LayerNorm and FFN are used to update frequency representations in a variate-independent manner. According to Theorem 1, the FFN corresponds to a series of convolution operations in the time domain for series representations. The vanilla attention mechanism is defined as:

Here, Q, K, V ∈ R 𝑁×𝐷 are the query, key and value matrix, respectively, obtained through linear projections. We denote 𝐷 as the feature dimension and refer to Softmax � QK 𝑇 /√𝐷 � , represented as A, as the attention matrix.

However, as shown in Figure 3, compared to other stateof-the-art forecasters, FreEformer with the vanilla attention mechanism usually exhibits an attention matrix with a lower rank. This could arise from the inherent sparsity of the frequency spectrum [Palani, 2022] and the strong-value-focused properties of the vanilla attention mechanism [Surya Duvvuri and Dhillon, 2024; Xiong et al., 2021]. While patching adjacent frequency points can mitigate sparsity (as in Fredformer), we address the underlying low-rank issue within the attention mechanism itself, offering a more general solution.

In this work, we adopt a straightforward yet effective solution: introducing a learnable matrix B to the attention matrix. The enhanced attention mechanism, denoted as EnhAttn (Q, K, V), is defined as 1 :

where Norm(·) denotes row-wise L1 normalization. The Softplus(·) function ensures positive entries, thereby preventing potential division-by-zero errors in Norm(·).

Theoretical Analysis

Feature Diversity According to Equation (9), feature diversity is directly influenced by the rank of the final attention matrix Norm � A+ ¯B � , where ¯B ≜ Softplus(B). Since rowwise L1 normalization does not alter the rank of a matrix, we have: rank � Norm � A+ ¯B �� = rank � A+ ¯B � . For further analysis, we present the following theorem:

Theorem 2. Let A and B be two matrices of the same size 𝑁 × 𝑁. The rank of their sum satisfies the following bounds:

The proof is provided in Section B of the appendix. As illustrated in Figure 4, the original attention matrix A often exhibits a low rank, whereas the learned matrix ¯B is nearly full-rank. According to Theorem 2, the combined matrix A + ¯B generally achieves a higher rank. This observation aligns with the results shown in Figure 4.

Gradient Flow Let a ∈ R 𝑁 denote a row in QK 𝑇 /√𝐷. For vanilla attention, the transformation is ˜a ≜ Softmax(a). Then the Jacobian matrix of ˜a regarding a can be derived as:

where Diag( ˜a) is a diagonal matrix with ˜a as its diagonal. For the enhanced attention, the transformation is given by:

where b represents a row of Softplus(B). The Jacobian matrices of c with respect to a and b can be derived as:

1 The variants are discussed in Section C.3 of the appendix.

where ˜b = ˜a + b = Softmax(a) + b, I ∈ R 𝑁×𝑁 is the unit matrix; 1 = [1, · · · , 1] 𝑇 ∈ R 𝑁 . The detailed proofs of Equations (11) and (13) are provided in Section C of the appendix.

We can see that 𝜕c/𝜕a in Equation (13) shares the same structure as that of vanilla attention in Equation (11), except for the scaling factor 1/ � � ˜b � � 1 . Since � �˜b � � 1 = 1+ ∥b∥1 > 1 and ˜b is learnable, the gradient is scaled down by a learnable factor, providing extra flexibility in gradient control.

Moreover, as shown in Figure 5, 𝜕c/𝜕b exhibits a pronounced diagonal than 𝜕c/𝜕a, suggesting a stronger dependence of c on b than a. This aligns with the design, as b directly modulates the attention weights.

Combination ofMLP andVanillaAttention We nowprovide a new perspective on the enhanced attention. In Equation (9), the attention matrix is decomposed into two components: the input-independent, dataset-specific term ¯B, and the inputdependent term A. If A is zero, the enhanced attention reduces to a linear transformation of V, effectively functioning as an MLP along the variate dimension. By jointly optimizing A and ¯B, the enhanced attention can be interpreted as an adaptive combination of MLP and vanilla attention.

5 Experiments

Datasets and Implementation Details We extensively evaluate the FreEformer using eighteen real-world datasets: ETT (four subsets), Weather, ECL, Traffic, Exchange, Solar-Energy, PEMS (four subsets), ILI, COVID-19, METR-LA, NASDAQ and Wiki. During training, we adopt the L1 loss function from CARD [Wang et al., 2024c]. The embedding dimension 𝑑 is fixed at 16, and the dimension 𝐷 is selected from {128, 256, 512}. The dataset description and implementation details are provided in the appendix.

5.1 Forecasting Performance

We choose 10 well-acknowledged deep forecasters as our baselines, including (1) Transformer-based models: Leddam [Yu et al., 2024], CARD [Wang et al., 2024c], Fredformer [Piao et al., 2024], iTransformer [Liu et al., 2024a], PatchTST [Nie et al., 2023], Crossformer [Zhang and Yan, 2023]; (2) Linear-based models: TimeMixer [Wang et al., 2024b], FreTS [Yi et al., 2024c] and DLinear [Zeng et al., 2023]; (3) TCN-based model: TimesNet [Wu et al., 2023a].

Comprehensive results for long- and short-term forecasting are presented in Tables 2 and 3, respectively, with the best results highlighted in bold and the second-best underlined. FreEformer consistently outperforms state-of-theart models across various prediction lengths and real-world domains. Compared with sophisticated time-domain-based models, such as Leddam and CARD, FreEformer achieves superior performance with a simpler architecture, benefiting from the global-level property of the frequency domain. Furthermore, its performance advantage over Fredformer, another Transformer- and frequency-based model, suggests that the deliberate patching of band-limited frequency spectra may introduce noise, hindering forecasting accuracy.

Notably, in Table 4, we compare FreEformer with additional frequency-based models, where it also demonstrates a clear performance advantage. The visualization results of FreEformer are presented in Figure 6. Furthermore, as shown in Table 10 of the appendix, FreEformer exhibits state-of-theart performance with variable lookback lengths.

5.2 Model Analysis

Architecture Ablations The FreEformer utilizes an enhanced Transformer architecture to capture cross-variate dependencies in the frequency domain. Table 5 presents a comparison of several FreEformer variants, evaluating the impact of linear and enhanced Transformer layers, different dimensional configurations, and patching along the frequency dimension. To ensure a fair comparison, the enhanced Transformer is used for all Transformer-based settings. The results indicate that: 1) Enhanced Transformer blocks outperform linear layers due to their superior representation capabilities; 2) Multivariate dependency learning generally outperforms inter-frequency learning, aligning with the claim in FreDF [Wang et al., 2024a] that correlations among frequency points are minimal; 3) Furthermore, patching does not improve FreEformer, likely because patching frequencies creates localized receptive fields, thereby limiting access to global information.

Frequency-Domain vs. Temporal Representation To construct the time-domain variant of FreEformer, we remove the DFT and IDFT steps, as well as the imaginary branch.

As shown in Table 6, the frequency-domain representation achieves an average improvement of 8.4% and 10.7% in MSE compared to the time-domain version under the enhanced and vanilla attention settings, respectively. Additionally, we show in Section I.1 of the appendix that Fourier bases generally outperformWavelet and polynomial bases for our model.

Prediction Head In our model, after performing frequency domain representation learning, we apply a temporal prediction head to generate the final predictions. In contrast, some frequency-based forecasters (e.g., FITS and Fredformer) directly predict the future frequency spectrum and transform it back to the time domain as the final step. In FreEformer, the frequency prediction head is formulated as:

where ˜ Re and ˜ Im are defined in Equation (6). As shown in Table 7, the temporal head slightly outperforms the frequencydomain head, highlighting the challenges of accurately forecasting the frequency spectrum. Additionally, Equation (14) incurs higher computational costs in the IDFT step when 𝜏 > 𝑇, as in long-term forecasting scenarios.

5.3 Enhanced Attention Analysis

We compare the enhanced Transformer with vanilla Transformer, state-of-the-art Transformer variants and Mamba [Gu and Dao, 2023] in Table 8. The enhanced Transformer consistently outperforms other models, verifying the effectiveness of the enhanced attention mechanism.

We further apply the enhanced attention mechanism to state-of-the-art forecasters, as shown in Table 9. This yields average MSE improvements of 5.9% for iTransformer, 9.9% for PatchTST, 1.4% for Leddam (with updates only to the 'cross-channel attention' module), and 3.8% for FreEformer. These results demonstrate the versatility and effectiveness of the enhanced attention mechanism. Moreover, comparing Tables 2 and 9, FreEformer consistently outperforms these improved forecasters, underscoring its architectural advantages.

In the appendix, we further demonstrate FreEformer's performance superiority on more metrics (e.g., MASE, correlation coefficient). Remarkably, FreEformer, trained from scratch, achieves superior or comparable performance to a pre-trained model fine-tuned on the same training data.

6 Conclusion

In this work, we present a simple yet effective multivariate time series forecasting model based on a frequency-domain enhanced Transformer. The enhanced attention mechanism is demonstrated to be effective both theoretically and empirically. It can consistently bring performance improvements for state-of-the-art Transformer-based forecasters. We hope that FreEformer will serve as a strong baseline for the time series forecasting community.

Acknowledgments

This work was supported by Beijing Natural Science Foundation under Grant No. L247029, and the National Natural Science Foundation of China (NSFC) under Grant No. 62371009.