Perturbation Matters in Time Series Forecasting: A Wave-attention-aware Transformer Method

1 Introduction

Time Series Forecasting (TSF) refers to the task of predicting future sequential data based on historical observations [Qiu et al., 2024; Kim et al., 2025]. As a fundamental task of machine learning, it has been widely adopted in many real-world applications, such as climate forecasting, epidemic analysis, energy management, financial investment, and traffic planning [Mudelsee, 2019; Deng et al., 2021; Yang et al., 2025; Wang et al., 2025a].

During the past decades, many TSF methods have been developed [Zeng et al., 2023; Ahamed and Cheng, 2024;

∗Corresponding author

Wang et al., 2025b]. Transformer-based TSF methods have recently attracted much attention because of their high performance, especially for long-term forecasting [Nie et al., 2023; Zhang and Yan, 2023; Liu et al., 2024]. Technically, they follow the spirit of problem transformation, whose basic idea is to translate time series into token sequences (i.e. as raw texts) before applying well-established transformers. For example, PatchTST, a representative transformer-based TSF method, applies the patching technique to group segments of time series into tokens [Nie et al., 2023]. In contrast, iTransformer treats each time series of a variate as a single token and models inter-variate dependencies by attention [Liu et al., 2024]. Regarding the attention-based dependency modeling, a simpleyet-effective approach is to treat each variate independently and model the temporal dependencies using attention [Nie et al., 2023]. Building on this, Crossformer introduces a Two-Stage Attention mechanism to simultaneously capture both cross-time and cross-dimension dependencies [Zhang and Yan, 2023], and Pathformer integrates the modeling of intra-dependencies within each token.

Our initial research intention is to investigate the effectiveness of transformer-based TSF methods in various scenarios, and in this paper, our story begins with the empirical study of their stability against noises. Specifically, we apply 4 benchmark time series datasets. For each one, we inject 2 types of noise with different levels (in total 160+ settings) into the training data, and separately train PatchTST models with them. We then evaluate the forecasting performance on test data to analyze how stable it is with different noises and levels. We surprisingly find that, in most cases, injecting noise with appropriate levels can effectively boost the forecasting performance, especially on the longer horizons (see Fig. 1). This phenomena naturally raises the question: why does injecting noises with appropriate levels lead to such a pronounced improvement in time series forecasting?

To explore the underlying reasons, we investigate the token attention weights in transformers, i.e. the critical component to extract temporal dependencies. Interestingly, we observe that noise injection at an appropriate level makes the attention weights exhibit a wave-like pattern, rather than being concentrated on a small set of tokens (see Fig 2). Comparing with highly concentrated attention weights, the wavy attentions may reflect the intrinsic characteristics of time series data, such as temporal continuity, periodicity and multi-scale dependencies, yielding more informative representations to improve the forecasting performance.

Inspired by this phenomena, we propose a novel TSF method to adaptively perturb the training data which constrains the token attention weights to be wavy, namely WaveAttention-aware TransformER (WATER). Specifically, we apply Gaussian perturbations with zero mean and a learnable standard deviation (std) parameter to control the perturbation level. We optimize std parameters by maximizing the secondorder differences on the attention weights of each token. In this way, each token attention weight is encouraged to have locally high-curvature structures and thereby exhibiting a wave-like pattern, rather than highly concentrated on a small set of tokens. By constraining the token attention weights to be wavy, our WATER can adaptively learn perturbations at an appropriate level, which can induce richer temporal dependencies and consequently more informative token representations to promote forecasting performance. In a nutshell, our contributions are summarized as below:

• We conduct extensive preliminary studies on transformerbased TSF methods to evaluate the stability over various noise types. We surprisingly find that appropriate noise levels can promote the forecasting performance and make the token attention weights wavy.
• Inspired by the above phenomena, we propose a novel TSF method WATER which adaptively perturbs the training data by constraining the token attention weights to be wavy via maximizing the corresponding second-order differences.
• We examine WATER by several time series forecasting tasks against SOTA baseline methods on benchmark datasets. And the experimental results demonstrate that our WATER can achieve effective performance improvements.

2 Preliminary Study

We conduct preliminary experiments to evaluate how transformer-based TSF methods are stable with different noises and levels.

2.1 Settings

In this preliminary study, we apply PatchTST because it is acknowledged as a classic transformer-based TSF method. Specifically, we collect 4 benchmark time series datasets (ETTh1, ETTh2, ETTm1, ETTm2), and for each one, we inject noises into the training data. We then train PatchTST with them, and our goal is to evaluate the performance differences on test data between the models trained with and without noises.

We apply 2 types of noises, i.e. structure-agnostic and structure-aware noises, with different levels, totally collecting 160+ settings. Specifically, for structure-agnostic noises, we adopt Gaussian noise, power law noise (Fractional Gaussian noise, FGN), shot noise and impulsive noise (BernoulliGaussian noise, BG) following [Wunderlich and Sklar, 2023]. Noise levels are controlled by varying the signal-to-noise

• (a) Train w/ Gauss. noise.

(b) Train w/ FGN noise.

Figure 1: The MSE and MAE scores of PatchTST on dataset ETTh1 when applying Gaussian and FGN noises with different levels (SNR={0, 10, 20, 30} dB) on training data.

rate (SNR) from {-5, 0, 5, 10, 15, 20, 25, 30} dB and noisespecific parameters, e.g. the Bernoulli mask probability for controlling whether to add Gaussian noise on each time point is set as {0.1, 0.5, 0.7, 1.0}. For structure-aware noises, we adopt Speedup, Cutoff and Scaling transformations following [Obata et al., 2025]. The noise levels are controlled by varying mask ratios from {0.01, 0.1, 0.2, . . . , 0.7}, which controls the proportion of a sequence to be transformed, and scaling parameters from {0.01, 1.0, 2.0, 3.0, 4.0}, which controls noise scaling in noise addition. Statistics of datasets are presented in Table 4 in Appendix. Detailed noise additions and parameter settings are presented in Appendix B.1 and Table 5.

2.2 Empirical Observations

We report the averaged MSE/MAE scores of applying Gaussian and FGN noises on ETTh1 over different historical sequence lengths in Fig. 1a, 1b and full results of different noises are presented in Appendix B.2.

We can surprisingly find that applying noise at appropriate levels can effectively promote the forecasting performance, especially for longer horizons. As the noise level gradually increases frommild to excessive, the overall forecasting performance gain first increases and then deteriorates, and are more significant when horizon=336/720. Specifically, an appropriate noise here refers to noise that slightly blurs local details without altering the overall trend (e.g. the corrupted series with Gaussian noise of 15 dB in Fig. 6). Such noises typically happens at SNR ∈ {15, 20, 25} dB.

2.3 Analysis from Self-Attention Perspective

To explore the underlying reason, we investigate the effect of noise on the token attention weights, which are crucial for modeling complex temporal dependencies in transformerbased methods. We focus on 4 structure-agnostic noises and compare token attention weights between training with and without applying noises on ETTh1 and present a toy attention comparison in Fig. 2. More results of token attention weights are presented in Figs. 18, 19, 20, 21 in Appendix B.2. We can find that applying noise to training data makes the token attention weights wavy and effectively mitigates the extremetoken phenomena.

When training without noise, the extreme-token phenomena is relatively severer than training with noise, i.e. higher attention weights are concentrated on a small set of tokens,

• (a) Train w/o noise.

Figure 2: Output token attention weights comparing training without noise and training with applying Gaussian noise of SNR=20 dB.

and the rest tokens receive only negligible weights. This is a common issue in transformer-based methods and has been widely discussed in previous studies [Guo et al., 2023; Xiao et al., 2024]. Such attention pattern contradicts the complex temporal dependency nature of time series data. It may result in collapsed and less discriminative token representations, thereby harming the overall forecasting performance.

When training with noise, the token attention weights become wavier, where the attention weights of each token vary with temporal distance and exhibiting a rising-or-falling local pattern in a non-monotonic manner. This wave-like pattern may suggest that applying noise encourages the model to aggregate information across different temporal ranges and can effectively explore the intrinsic characteristics of time series data, such as local continuity, periodicity, and multi-scale temporal dependencies. Therefore, the induced attention weights becomes more consistent with the complex characteristics of time series data, leading to more informative token representations and improved forecasting performance.

In a word, applying noise to training data with appropriate levels can induce a wave-like token attention weights, which may reflect the intrinsic characteristics of time series data. This inspires us to adaptively introduce perturbations by constraining the token attention weights to be wavy, thereby improving the overall forecasting performance.

3 Method

For simplicity of notations, we introduce our method from the perspective of univariate time series forecasting. We denote an observed time series sequence by xi = [xi,0, xi,2, . . . , xi,L-1]⊤ ∈ RL, where L denotes the historical sequence length. The time series forecasting task aims at predicting the future sequence yi = [yi,0, yi,2, . . . , yi,F-1]⊤ ∈ RF of length F. Generally, we collect a set of N times series D = {(xi, yi)}Ni=1 as training data to learn a forecasting model f(·) which can accurately predict the future sequence using the observed time series.

Inspired by the preliminary experiments, we propose a novel TSF method WATER to perturb the training data to enhance the overall predicting performance. To induce an appropriate perturbation level, we introduce a second-order difference maximization constraint to encourage the token attention weights to be wavy. We will introduce the details of WATER in the following.

3.1 Learnable Gaussian Perturbations

We consider the Gaussian perturbations and control the noise level by a learnable parameter of standard deviation (std). Specifically, for sequence xi, to exert noise with different levels on each time point, we set the std parameter as σi ∈ RL and the perturbed input is given by:

where ei denotes the Gaussian perturbations and is calculated by the reparameterization trick [Kingma and Welling, 2014]:

The standard deviation parameter is optimized by encouraging the model to output a wavy token attention matrix along with model training, which will be introduced in the following.

3.2 Tokenization and Forecasting

We adopt a transformer-based TSF method which considers token-wise attentions to predict the future sequence ˆyi. We first pre-process the perturbed sequence by Instance Normalization [Kim et al., 2021] and then patch it into several tokens to adapt the input formats of transformers. Specifically, each perturbed sequence ˜x ∈ RH is patched into xi = [xi,1, . . . , xi,Ps ] ∈ RLp×Ps , where xi,j = [˜xi,(j-1)·Lp+1, . . . , ˜xi,j·Lp ]⊤ ∈ RLp denotes the jth token; Lp denotes the token length; Ps = ⌊L/Lp⌋ + 2 denotes the number of patches.

Then we adopt an embedding layer fWe (·), C stacked transformer layers fW1:C t (·) and a flatten-linear predictor fWf (·) to predict the future sequence ˆyi:

We learn the model by optimizing the objective:

where ℓ(·, ·) denote the forecasting loss and we adopt the weighted loss following [Ma et al., 2025].

3.3 Wavy Attention Constraint

Taking inspiration from preliminary studies showing that appropriate noise injection yields wave-like token attention patterns, we adaptively learn input perturbations by directly maximizing the second-order difference of token attention weights to encourage higher curvature.

Specifically, for each token sequence xi, its corresponding attention weight across all tokens is denoted by Ai = {Ac,h i }, where Ac,h i ∈ RLp×Lp , c ∈ [1,C] denotes the transformer layer index and h ∈ [1,H] denotes the attention head index. Specially, it's worth mentioning that the attention matrix Ac,h i can be derived from various attention mechanisms across different backbones, making our method generalizable to any transformer-based TSF model that considers token-wise attention. To induce wavy token attention weights, inspired by [Hodrick and Prescott, 1997], we introduce a second-order difference maximization constraint on the attention weight Algorithm 1 Training procedure of WATER.

Input: Time series data {(xi, yi)}Ni=1 Output: Optimal parameters {σ∗ ,W∗ e , W1:C t ∗ , W∗ f } of

WATER

• 1: Initialize {σ,We,W1:C t ,Wf};
• 2: for epoch e = 1 toMaxEpoch do
• 3: for iter it = 1 to MaxIter do
• 4: Add Gaussian perturbations to each xi by Eq. (1);
• 5: Calculate the training objective of Eq. (6);
• 6: Update {σ,We,W1:C t ,Wf} by back-propagation;
• 7: end for
• 8: end for

distribution of each token to induce higher curvature on each token. Therefore, we set the following maximization objective:

where ∆2 ((Ac,h i )j,k) denotes the second-order difference of Ac,h i along k.

In this way, the local curvatures of each token point are encouraged to be higher, thereby promoting a wave-like pattern in the overall token attention matrix. Each token are encouraged to attend to more tokens across different temporal ranges, which is consistent with the intrinsic data characteristics of time series, such as local continuity, multiple periodicity, etc. In this way, we can utilize more complex temporal dependencies to induce more informative token representations, which contributes to promoted forecasting performance.

By combining the above two objectives, the whole learning objective is formulated as below:

where λ denotes the scaling parameter, A = {Ai}Ni=1 denotes the set of token attentions of all samples. By optimizing the above objective, we can adaptively learn the standard deviation parameter by constraining the token attention weights to be wavy, thereby adaptively adding noise to the input series, further promoting the overall forecasting performance. The overall model training process of WATER is summarized in Algorithm 1.

4 RelatedWork

4.1 Time Series Forecasting

In the past decades, time series forecasting methods have evolved from statistical techniques, such as exponential smoothing and ARIMA models [Box et al., 2015], to advanced machine learning approaches. Recently, deep learning methods have been widely adopted to capture complex temporal patterns, such as MLP, RNN, CNN, Transformers, diffusion models and Mamba, etc.

Transformers has been widely adopted for TSF due to excelling at capturing long-term dependencies. However, early point-wise transformers-based TSF methods suffer from quadratic computational complexity. To address this problem, researchers have devoted into several directions, including sparsity decomposition, tokenization, etc. Informer [Zhou et al., 2021] treats each time series as a single token and models variable dependencies using attention mechanisms. Building upon this paradigm, models such as Autoformer [Wu et al., 2021] and FEDformer [Zhou et al., 2022] successfully reduced complexity using sparsity strategies. More recently, researches integrate tokenization technique to induce tokenwise methods, which significantly reduce the computational complexity and adapt transformers for modeling longer sequences. PatchTST [Nie et al., 2023] is one of the pioneers which adopts the simple channel independency assumption. Crossformer [Zhang and Yan, 2023] develops a two-stage attention mechanism to simultaneously capture the inter- and intra-variate dependencies. Pathformer [Chen et al., 2024] propose to segment time series into patches of different sizes and use a multi-scale router to extract temporal dependencies frommultiple resolutions. TimeXer [Wang et al., 2024b] introduced variable-level tokens to account for bidirectional dependencies.

Building upon transformer-based TSF methods, our work observes that injecting noise at an appropriate level into the input time series consistently improves forecasting performance. Motivated by this, we propose WATER, which adaptively learns the perturbation level by encouraging the token-wise attention weights to be wavy, thereby enabling richer temporal dependencies and promoting the forecasting performance.

4.2 Performance Enhancement via Perturbations

In the machine learning community, introducing controlled perturbations during training has been widely adopted as an effective approach to improve model generalizability and robustness. A prevalent way is to design artificial noise injection as implicit regularizer, which leads into smoother solutions and effectively alleviates overfitting [Orvieto et al., 2022]. In computer vision, [Yin et al., 2025] applies random noise into feature statistics within the model. This strategy significantly improves generalization under unseen noise distributions. Similar ideas have also been explored in TSF. For example, [Yoon et al., 2022] introduces noise during training to achieve certified robustness against bounded adversarial perturbations. [Cheng et al., 2024] systematically studies the effects of different anomaly injection strategies on TSF. Based on their findings, they propose a robust forecasting framework. Overall speaking, injecting noise into training data can enhance the model generalization, leading to more robust and accurate prediction performance.

5 Experiment

5.1 Experimental Settings

Datasets

We conduct our experiments on 8 benchmark time series datasets, including ETTh1, ETTh2, ETTm1, ETTm2, Weather, Solar Energy, Electricity, and Traffic [Zhou et al., 2021]. Statistics are summarized in Table 4.

Baseline Methods

We compare WATER with 9 advanced time series forecasting methods as baselines, comprising MoFo [Ma et al., 2025], PDF [Dai et al., 2024], iTransformer [Liu et al., 2024], Pathformer [Chen et al., 2024], CycleNet [Lin et al., 2024], TimeMixer [Wang et al., 2024a], PatchTST [Nie et al., 2023], Crossformer [Zhang and Yan, 2023] and DLinear [Zeng et al., 2023].

Implementation Settings

We adopt MoFo [Ma et al., 2025] as the backbone forecasting model and set the periodic length as 24. We also evaluate our WATER on other transformer-based backbones which consider token-wise attentions in latter sections. Each σi is initialized by exp(-2.0). The initial learning rate is set as 0.01 and we decay it by half every 5 epochs for each parameter σi and every

2 epochs for rest parameters. The scaling parameter λ of objective Eq. 6 is tuned over {1e-5, 1e-4, 1e-3, 1e-2, 1e-1} and we present the sensitivity analysis of λ in latter sections. The number of backbone layer is set as 1 and we tune the layer dimensions following [Ma et al., 2025]. To ensure evaluation fairness, we disable the "Drop Last" batch-sampling trick. Our experiments are conducted on 2 NVIDIA RTX 4090 GPUs of 128 GB memory in Ubuntu system. We implement our method within the TFB library [Qiu et al., 2024] to ensure fair comparisons. We adopt MSE and MAE for evaluation and report the best performance achieved over historical sequence length L ∈ {96, 336, 512} for each forecasting horizon F ∈ {96, 192, 336, 720}. The baseline results are taken from official implementations or the original papers.

5.2 Result and Analysis

Forecasting Comparison across Baselines

The overall forecasting performance under different prediction horizons are presented in Table 1. We can find that our WATER consistently outperforms most baseline methods in most cases. This results directly indicate the effectiveness of our proposed method in handling long-term time series forecasting. More specifically, the performance improvements tend to be larger on longer prediction horizons. For example, on ETTh2, WATER achieves MSE gains of 0.011 and 0.010

over MoFo at horizon F = 336 and horizon F = 720, respectively, with corresponding MAE improvements of 0.008 and 0.009, indicating that WATER provides increasingly effective corrections as the prediction horizon extends. The result is consistent with the phenomena in our preliminary studies where the perturbations can boost the performance more significantly on longer prediction horizons. The reason may be that perturbations at an appropriate level can drown out minor local shifting patterns and make the overall time series more stationary, where the local shifts may somehow hinder the extraction of long-term global dependencies. In this way, global temporal patterns are amplified and token representations become more statistically consistent, enabling the model to better capture long-range dependencies and leverage global characteristics such as global trend and seasonality.

Generalization across Backbones

To examine the generalization of our WATER, we implement our WATER using different transformer-based TSF backbones, including PatchTST [Nie et al., 2023], Crossformer [Zhang and Yan, 2023] and PDF [Dai et al., 2024]. We compare our method across these backbones on 4 ETT datasets. The results using PatchTST and Crossformer as backbones are presented in Table 2 and results using PDF as backbone are presented in Table 6 in Appendix. We can find that our WATER consistently outperforms the corresponding backbones in most settings, with particularly notable improvements on Crossformer, demonstrating that our method can effectively enhance diverse transformer architectures regardless of their underlying design choices. These results indicate the effectiveness and generalizability of our WATER as a lightweight plug-in module.

Attention Visualization

To examine the effectiveness of our proposed second-order difference maximization on token attention weights in our WATER, we present the visualizations on attention token weights of each self-attention head in Fig. 3. Comparing with MoFo, i.e. the backbone of our WATER, the token attention weights produced by our approach exhibit a clearer wave-like pattern, along with a clearer diagonal correlation structures. Moreover, the contrasts between higher and lower attention weights are enhanced to some extent. The higher weights of each token are more concentrated with its corresponding neighbors and meanwhile the lower weights are slightly suppressed. This indicate that our method can produce clearer and more structural attentions, making it less sensitive to possible noisy temporal patterns. Meanwhile, our WATER also mitigate the extreme-token phenomena in "Head1", further indicating that our WATER is capable of capturing more accurate temporal patterns to promote the forecasting performance.

Forecasting Visualization

We present the forecasting visualizations on ETTh2, ETTm1 and ETTm2 in Fig. 4. We can find that our method can predict sequences which fit the ground-truth trends and key fluctuations well, and the predicted curves are smooth. This indicate that our method can effectively capture the temporal patterns to produce accurate and stable forecasting, making it practical in time series forecasting task.

Sensitivity on Scaling parameter λ

We examine the sensitivity of the scaling parameter λ, which controls the weight of the our proposed second-order difference maximization regularization term. We vary it over {1e - 5, 1e - 4, 1e - 3, 1e - 2, 1e - 1} on all 8 datasets and evaluate the forecasting performance. Results of 4 ETT datasets are presented in Fig. 5 and full results of 8 datasets are presented in Fig. 22 in Appendix. Overall speaking, WATER demonstrates low sensitivity to λ across all datasets, with both MSE and MAE remaining relatively stable over the range of values, indicating the robustness of our method. A slight performance degradation is observed when λ = 0 (i.e., the regularization term is disabled) on several datasets such as ETTh1 and Weather, indicating the effectiveness of the regularization in guiding the noise learning process. Furthermore, excessively large values (e.g., λ = 1e-1) occasionally lead to minor degradation, suggesting that an overly strong regularization may interfere with the dominant forecasting objective. In practice, we recommend setting λ ∈ {1e-4, 1e-3} for stable and consistent performance across diverse datasets.

Sensitivity on Historical Sequence Length L

We evaluate the sensitivity of our WATER to the the historical length by varying the it over {96, 336, 512} on prediction horizons {96, 192, 336, 720} and present the forecasting performance of datasets Weather, Solar, Electricity and Traffic in Fig. 23 in Appendix. Overall, we observe a consistent trend that performance improves as the historical length increases across most settings and datasets. In particular, longer input sequences generally lead to lower prediction errors, especially for longer forecasting horizons, indicating that the model benefits from richer temporal context. This suggests that WATER is able to effectively leverage extended historical information to capture more comprehensive and stable temporal dynamics.

Efficiency Analysis

We present computational efficiency comparisons on dataset Solar of forecasting horizon 96 in Table 3, where we report the amount of all learnable parameters requiring gradient descent (Param.), floating point operations (FLOPs), peak GPU memory usage (Peak GPU) and training time of one epoch (Epoch Time). We compare the methods under the same time series forecasting library TFB [Qiu et al., 2024] and same environment. All results of each model are under the optimal hyper-parameters for fair comparison. The results indicate that our method introduces a small additional computational cost compared to the baseline methods. While optimizing the noise injection parameter may lead to slower convergence, potentially requiring more training time. Overall, our method maintains competitive computational efficiency against the prevalent baseline TSF methods.

6 Conclusion

In this paper, we focus on investigating the effect of perturbations to transformer-based TSF methods. Firstly, we conduct extensive preliminary studies to evaluate the stability of transformer-based TSF methods over various types of noises. We surprisingly find that applying noises at appropriate levels can effectively promote the forecasting performance and make the token attention weights wavy. Inspired by thie phenomena, we propose a novel TSF method WATER which adaptively perturbs the training data by Gaussian perturbations with zero mean and a learnable standard deviation to control the perturbation level. We optimize the standard deviation parameter by maximizing the second-order differences of token attention weights, which encourages each token to exhibit wave-like attention patterns with locally high curvature. Empirical studies on 8 benchmark datasets indicates that our WATER can outperform strong TSF baseline methods and induce attention weights in wave-like patterns.

Acknowledgments

We would like to acknowledge support for this project from the National Natural Science Foundation of China (No.62276113).