Deep Frequency Derivative Learning for Non-stationary Time Series Forecasting

1 Introduction

Time series forecasting has been playing an important role in a variety of real-world industries, such as traffic analysis [Ben-Akiva et al., 1998], weather prediction [Lorenz, 1956], financial estimation [King, 1966; Ariyo et al., 2014], energy planning [Deb et al., 2017], etc. Following by classic statistical methods (e.g., ARIMA [Whittle, 1963]), many

* Corresponding Author.

• (a) Transformation with only the zero frequency component.

Figure 1: Given one time series and its frequency spectrum, the main comparison between existing works (a) and our method (b).

deep machine learning-based time series forecasting methods [Salinas et al., 2020; Oreshkin et al., 2019] have recently achieved superior performance in different scenarios. Despite the remarkable success, the non-stationarity widely existing in time series data has still been a critical but under-addressed challenge for accurate forecasting [Priestley and Rao, 1969; Huang et al., 1998; Brockwell and Davis, 2009].

Since time series data are usually collected at a high frequency over a long duration, such non-stationary sequences with millions of timesteps inevitably let forecasting models face the distribution shifts over time. This would lead to performance degradation at test time [Kim et al., 2022] due to the covariate shift or the conditional shift [Woo et al., 2022a]. For this issue, pioneer works [Ogasawara et al., 2010] propose to normalize time series data with global statistics; one recent work [Kim et al., 2022] proposes to use instance statistics to normalize time series against distribution shifts. Then, some work brings statistical information into self-attention computation [Liu et al., 2022b]; another work [Fan et al., 2023] transform time series with learnable statistics and consider shifts input and output sequences, and [Liu et al., 2023] utilize predicted sliced statistics for adaptive normalization.

Most of these existing works focus on transforming each timestep of time series with certain statistics (usually mean and std.) After our careful theoretical analysis 1 , we have found that these operations can actually be regarded as the normalization towards the zero frequency component of the spectrum in the frequency domain, as shown in Figure 1(a). However, they cannot fully utilize distribution information of time series signals; moreover, this would lead to information utilization bottleneck in normalization and thus hinders the performance of time series forecasting. To address this problem, we propose to utilize the whole frequency spectrum for the transformation of time series, to make full use of distribution information of time series from the frequency perspective and in the meanwhile transform time series into more stationary space thus making more accurate forecasting. Figure 1(b) has shown our method with the whole frequency spectrum.

1 We leave the details of our theoretical analysis in Appendix B.1

Motivated by this view, we then present a deep frequency derivative learning framework, DERITS, for non-stationary time series forecasting. The core idea of DERITS lies in two folds: (i) employing the whole frequency spectrum to take the derivative of time series signals, and (ii) learning frequency dependencies on more stationary transformed representations. Specifically, we first propose a novel transformation for time series signals in DERITS, namely Frequency Derivative Transformation (FDT), which mainly includes two stages. In the first stage, the raw signals in the time domain are transformed into the frequency domain with Fourier transform [Nussbaumer and Nussbaumer, 1982] for further learning. In the second stage, the transformed frequency components are derived with respect to timestamps to get more stationary frequency representations. Inspired by the derivative in mathematics [Hirsa and Neftci, 2013], FDT let models aim for modeling gradients of signals rather than raw input signals, which could mitigate their burden of forecasting with distribution shifts by resolving non-stationary factors (e.g., the shift of trends) in the original time series through one- or high-order derivation.

After acquiring more stationary representations, we further propose a novel architecture, Order-adaptive Fourier Convolution Network (OFCN) in DERITS for the frequency filtering and dependency learning to accomplish the forecasting. Concretely, OFCN is composed of (i) Order-adaptive frequency filter that adaptively extracts meaningful patterns by excluding high-frequency noises for derived signals of different orders, and (ii) Fourier convolutions that conduct dependency mappings and learning for complex values in the frequency domain. Since OFCN is operating in the projection space by FDT, we thus utilize the inverse Frequency Derivative Transformation to recover the predicted frequency components back to the original time domain. Inspired by previous work [Kim et al., 2022], we let all stages of FDT fully reversible and symmetric and make OFCN predict in the more stationary frequency space, which reveals our superiority in enhancing forecasting against distribution shifts. Furthermore, in order for the multi-order derivative learning, we have organized DERITS as a parallel-stacked architecture to fuse representations of different orders. Specifically, DERITS is composed of several parallel branches, each of which represents an order of derivation and prediction corresponding for its FDT and OFCN. Note that the Fourier convolution adapted in each branch is not parameter-sharing and after the distinct processing of different branches, the outputs are fused to achieve the final time series forecasting. In summary, our main contribution can be listed as follows:

• Motivated by our theoretical analysis towards existing time series normalization techniques from the frequency spectrum perspective, we propose to utilize the whole frequency spectrum for the transformation of time series.
• We present a deep frequency derivative learning framework, namely DERITS, built upon our proposed Frequency Derivative Transformation (with its inverse) for non-stationary time series forecasting.
• We introduce the novel Order-adaptive Fourier Convolution Network, for the frequency dependency learning and organize DERITS as a parallel-stacked architecture to fuse multi-order representations for forecasting.
• We have conducted extensive experiments on seven realworld datasets, which have demonstrated the consistent superiority compared with state-of-the-art methods in both time series forecasting and shift alleviation.

2 RelatedWork

2.1 Time Series Forecasting with Non-stationarity

Time series forecasting is a longstanding research topic. Traditionally, researchers have proposed statistical approaches, including exponentially weighted moving averages [Holt, 1957] and ARMA [Whittle, 1951]. Recently, with the advanced development of deep learning, many deep time series forecasting methods have been developed, including RNN-based methods (e.g., deepAR [Salinas et al., 2020], LSTNet [Lai et al., 2018]), CNN-based methods (e.g., SCINet [Liu et al., 2022a], TCN [Bai et al., 2018]), MLP-based Methods (e.g., DLinear [Zeng et al., 2023], N-BEATS [Oreshkin et al., 2020]) and Transformer-based methods (e.g., Autoformer [Wu et al., 2021], PatchTST [Nie et al., 2023]). While time series are non-stationary, existing works try to normalize time series with global statistics [Ogasawara et al., 2010], instance statistics [Kim et al., 2022], learnable statistics [Fan et al., 2023] and sliced statistics [Liu et al., 2023] in order to relieve the influence of distribution shift on forecasting.

2.2 Frequency Analysis in Time Series Modeling

The frequency analysis has been widely used to extract knowledge of the frequency domain in time series modeling and forecasting [Yi et al., 2023a]. Specifically, SFM [Zhang et al., 2017] adopts Discrete Fourier Transform to decompose the hidden state of time series by LSTM into frequency components; StemGNN [Cao et al., 2020] adopts Graph Fourier Transform to perform graph convolutions and FourierGNN [Yi et al., 2024] uses Discrete Fourier Transform to computes series-wise correlations. Autoformer [Wu et al., 2021] replaces self-attention in Transformer [Vaswani et al., 2017] and proposes the auto-correlation mechanism implemented by Fast Fourier Transform. FEDformer [Zhou et al., 2022] introduces Discrete Fourier Transform-based frequency enhanced attention. DEPTS [Fan et al., 2022] utilizes Discrete Cosine Transform to extract periodic patterns. In addition, [Woo et al., 2022b] transforms hidden features of time series into the frequency domain with Discrete Fourier Transform to enhance the time series representation learning. [Yi et al., 2023b] combines Fast Fourier Transform with multi-layer perceptrons for time series forecasting.

3 ProblemFormulation

Time Series Forecasting Let x = [x1; x2; · · · ; xT ] ∈ RT×D be regularly sampled multi-variate time series with T timestamps and D variates, where xt ∈ RD denotes the multi-variate values at timestamp t. In the task of time series forecasting, we use Xt ∈ RL×D to denote the lookback window, a length-L segment of x ending at timestamp t (exclusive), namely Xt = xt-L:t = [xt-L; xt-L+1; · · · ; xt-1]. Similarly, we represent the horizon window as a length-H segment of x starting from timestamp t (inclusive) as Yt, so we have Yt = xt:t+H = [xt; xt+1; · · · ; xt+H-1]. The classic time series forecasting formulation is to project lookback values Xt into horizon values Yt. Specifically, a typical forecasting model Fθ : RL×D → RH×D produces forecasts by ˆYt = fθ(Xt) where ˆYt stands for the forecasting results and θ encapsulates the model parameters.

Non-stationarity and Distribution Shifts In this paper, we aim to study the problem of non-stationarity in deep time series forecasting. As aforementioned in Section 1, long time series with millions of timesteps let forecasting models face distribution shifts over time due to the nonstationarity. The distribution shifts in time series forecasting are usually the covariate shift [Wiles et al., 2021; Woo et al., 2022a]. Specifically, given a stochastic process, let p (xt, xt-1, . . . , xt-L+1) be the unconditional joint distribution of a length L segment where xt is the value of univariate time series at timestamp t. The stochastic process experiences covariate shift if any two segments are drawn from different distributions, i.e. p (xt-L, xt-L+1 . . . , xt-1) ̸= p (xt′-L, xt′-L+1, . . . , xt′-1) , ∀t ̸= t′ . Subsequently, let p (xt | xt-1, . . . , xt-L) represents the conditional distribution of xt, such a stochastic process experiences conditional shift if two segments have different conditional distributions, i.e. p (xt | xt-1, . . . , xt-L+1, xt-L) ̸= p (xt′ | xt′-1, . . . , xt′-L+1, xt′-L) , ∀t ̸= t′ .

4 Methodology

In this section, we elaborate on our proposed deep frequency derivative learning framework, DERITS, designed for nonstationary time series forecasting. First, we introduce our novel reversible transformation, Frequency Derivative Transformation (FDT) in Section 4.1. Then, to fuse multi-order information, we present the parallel-stacked frequency derivative learning architecture in Section 4.2. Finally, we introduce our Order-adaptive Fourier Convolution Network (OFCN) for frequency learning in Section 4.3.

4.1 Frequency Derivative Transformation

As aforementioned in Section 1, to fully utilize the whole frequency spectrum for the transformation of time series with sufficient distribution information, we propose the novel Frequency Derivative Transformation (FDT) to achieve more stationary frequency representations of time series signals. For this aim, FDT mainly includes two distinct stages respectively for domain transformation and frequency derivation.

Domain Transformation

In the first stage, to be specific, we make use of fast Fourier transform [Nussbaumer and Nussbaumer, 1982] to enable the decomposition of time series signals from the time domain into their inherent frequency components. Formally, given the time domain input signals X(t), we convert it into the frequency domain by:

where F is the fast Fourier transform, f is the frequency variable, t is the integral variable, and j is the imaginary unit, defined as the square root of -1; ∫ ∞ -∞ X(t) cos(2πf t)dt is the real part of X and is abbreviated as Re(X); ∫ ∞ -∞ X(t) sin(2πft)dt is the imaginary part and is abbreviated as Im(X). After that we can rewrite X as X = Re(X ) + jIm(X ).

Frequency Derivation

In the second stage, with the transformed frequency components, we propose to utilize the whole frequency spectrum for the signal derivation, in order to represent time series in a more stationary space. The basic idea is to perform our proposed Fourier Derivative Operator in the frequency domain, which is defined as follows:

Definition 1 (Fourier Derivative Operator). Given the time domain input signals X(t) and its corresponding frequency components X(f), we then define R(X(f)) := (j2πf)X(f) as the Fourier Derivative Operator (FDO), where f is the frequency variable and j is the imaginary unit.

In the derivation, different order usually represents different signal representations. We propose to incorporate multiorder information in DERITS to further enhance the forecasting. For this aim, we extend above definition and further define the k-order Fourier Derivative Operator Rk as:

With such two stages, we can finally write the k-order Frequency Derivation Transformation FDTk as:

where X(t) is the time domain input signal; F stands for fast Fourier transform and f is the frequency variable.

Proposition 1. Given X(t) in the time domain and X(f) in the frequency domain correspondingly, the k-order Fourier Derivative Operator on X (f ) is equivalent to k-order derivation on X(t) with respect to t in the time domain, written by:

where F is Fourier transform, dk dtk is k-order derivative with respect to t, and j is the imaginary unit.

We leave the detailed proof in Appendix B.2. With such a equivalence, we can find out FDT can actually achieve more stationary representations in the lower order by derivation. For example, the shifts caused by a single trend signal in time series can nearly degraded be zero. We include the specific analysis in Appendix D. Then, with less distribution shifts and non-stationarity by FDT, the deep networks can have large potential to perform more accurate forecasting.

4.2 Frequency Derivative Learning Architecture

The main architecture of DERITS is depicted in Figure 2, which is built upon the Frequency Derivative Transformation and its inverse for the frequency derivative learning.

FDT/iFDT As mentioned in Section 4.1, DERITS needs to conduct predictions in a more stationary frequency space achieved by frequency derivative transformation. We naturally need to recover the predictions back to the time domain for final forecasting and evaluation. To make FDT fully reversible, we let both stages of FDT reversible, including Fourier transform and Fourier Derivation. Specifically, following Equation (3), we can symmetrically write the inverse frequency derivative transformation (iFDT) of k order as:

where R-1 k is the inverse process of Fourier Derivative Operator of k order; F -1 is the inverse Fourier transform; X(f) is the frequency components that need to be recovered to the time domain. Actually, the inverse process R-1 k is equivalent to an integration operator in the time domain. More details can be found in Appendix D.

The Parallel-Stacked Architecture

To conduct the multi-order frequency derivation transformation and learning, we have organized our DERITS framework as a parallel-stacked architecture, where each branch represents an order of frequency derivation learning, as shown in Figure 2. Let DERITS have K branches in total. For each branch, we first take lookback values Xt to frequency derivative transformation by:

where FDTk is the k-order FDT and X k t is the frequency derivative representation for Xkt at timestamp t. Then, the learned representations for each branch are taken to the Fourier Convolution Network (FCN) for frequency dependency learning. Since our FCN is order-adaptive in each parallel branch, we also take k as input with the computation by:

where Hk t are the predicted frequency components for X k t . Note that FourierConvolution is not parameter-sharing for different branches. After that, we recover the predictions to the time domain by:

where Hkt is the recovered representation of k order in the time domain. After acquiring it, we finally fuse the multiorder representations from parallel branches for the forecasting with MLP layers, which is given by:

where ˆYt are forecasting results by DERITS for evaluation.

4.3 Order-adaptive Fourier Convolution Network

Apart from the frequency derivative transformation, another aim of DERITS is to accomplish the dependency learning with the derived signals in the frequency domain. We thus introduce a novel network architecture, namely Orderadaptive Fourier Convolution Network (OFCN) to enable the frequency learning. Specifically, OFCN is composed of two important components, i.e., order-adaptive frequency filter and Fourier convolutions, which are illustrated as follows:

Order-adaptive Frequency Filter We aim to fuse multiorder derived signal information for forecasting, while it is notable that different order corresponds to different frequency comments. Since time series include not only meaningful patterns but also high-frequency noises, we develop an orderadaptive frequency filter to enhance the learning process.

Supposing there are S frequencies in X k t , we sort X k t on the frequencies in a descending order of amplitude for each frequency to get X ′kt for k order. Then, we design an adaptive mask mk to concentrate on only S 2(K-k) frequency components of X k t for further learning. We write the adaptive frequency filtering process by:

where mk is the mask vector of length S for filtering; vk is a randomly-initialized vector of order k which is learnable; H′kt are the filtered frequency representations. In particular, Equation (10) is inspired by that the low-order derived representations include more noises than more stationary highorder representations and thus should be filtered. To filter frequencies, we design an exponential-masking mechanism for mk to select S 2(K-k) frequencies while filtering others.

Fourier Convolutions Given the filtered signal representations in the frequency domain, the subsequent step involves acquiring the dependencies for time series forecasting. Considering that the representations are complex values, it is intuitive to devise a network in which all operations are conducted in the frequency domain. According to the convolution theorem [Katznelson, 1970], the Fourier transform of a convolution of two signals equals the point-wise product of their Fourier transforms in the frequency domain. Accordingly, we employ Fourier convolution layers that involve performing a product in the frequency domain, to capture these dependencies. Specifically, given H′kt achieved by order-adaptive filtering, we compute it as follows:

where Wk is the weighted matrix to conduct the convolutions in the frequency domain; Hk t is the output by our orderadaptive Fourier convolution network when the order is k. Equation (11) is intuitive which aims to directly learn the dependencies on the filtered components for forecasting.

5 Experiments

In this section, in order to evaluate the performance of our model, we conduct extensive experiments on six real-world time series benchmarks to compare with the state-of-the-art time series forecasting methods. Furthermore, we conduct a detailed analysis of our model for better evaluation.

5.1 Experimental Setup

Datasets We follow previous work [Wu et al., 2021; Zhou et al., 2022; Nie et al., 2023; Yi et al., 2023b] to evaluate our DERITS on different representative datasets from various application scenarios, including Electricity [Asuncion and Newman, 2007], Traffic [Wu et al., 2021], ETT [Zhou et al., 2021], Exchange [Lai et al., 2018], ILI [Wu et al., 2021], and Weather [Wu et al., 2021]. We preprocess all datasets following the recent frequency learning work [Yi et al., 2023b] to normalize the datasets and split the datasets into training, validation, and test sets by the ratio of 7:2:1. We leave more dataset details in Appendix A.1.

Baselines We conduct a comprehensive comparison of the forecasting performance between our model DERITS and several representative and state-of-the-art (SOTA) models on the six datasets, including Transformer-based models: Informer [Zhou et al., 2021], Autoformer [Wu et al., 2021], FEDformer [Zhou et al., 2022], PatchTST [Nie et al., 2023]; MLP-based model: LSTF-Linear [Zeng et al., 2023]; Frequency domain-based model: FreTS [Yi et al., 2023b]. Besides, we also consider the existing normalization methods towards distribution shifts in time series forecasting, including RevIN [Kim et al., 2022], NSTransformer [Liu et al., 2022b]

and Dish-TS [Fan et al., 2023]. All the baselines we reproduced are implemented based on their official code and we leave more baseline details in Appendix A.2.

Implementation Details We conduct our experiments on a single NVIDIA RTX 3090 24GB GPU with PyTorch 1.8 [Paszke et al., 2019]. We take MSE (Mean Squared Error) as the loss function and report the results of MAE (Mean Absolute Errors) and RMSE (Root Mean Squared Errors) as the evaluation metrics. A lower MAE/RMSE indicates better performance of time series forecasting. More detailed information about the implementation is included Appendix A.3.

5.2 Overall Performance

To verify the effectiveness of DERITS, we conduct the performance comparison of multivariate time series forecasting in several benchmark datasets. Table 1 presents the overall forecasting performance in the metrics of MAE and RMSE under different prediction lengths. In brief, the experimental results demonstrate that DERITS achieves the best performances in most cases as shown in Table 1. Quantitatively, compared with the best results of transformer-based models, DERITS has an average decrease of more than 20% in MAE and RMSE. Compared with more recent frequency learning model, FreTS [Yi et al., 2023b] and the state-of-the-art transformer model, PathchTST [Nie et al., 2023], DERITS can still outperform them in general. This has shown the great potential of DERITS in the time series forecasting task.

5.3 Comparison with Normalization Techniques

In this section, we further compare our performance with the recent normalization technique, RevIN [Kim et al., 2022] and Dish-TS [Fan et al., 2023] that handle distribution shifts in time series forecasting. Table 2 has shown the performance comparison in time series forecasting taking the LTSFLinear [Zeng et al., 2023]. Since FDT transforms signals to the frequency domain, we implement a simple single-layer Linear model in the frequency domain. From the results, we can observe that the existing RevIN and Dish-TS can only improve the backbone in some shifted datasets. In some situations, it might lead to worse performances. Nevertheless, our FDT can usually achieve the best performance. A potential explanation is that FDT transforms data with full frequency spectrum and thus achieves stable improvement, while other normalization techniques cannot reveal full data distribution and thus cannot make use of them for transformation.

5.4 Model Analysis

Impact of FrequencyDerivative Transformation It is notable that our proposed FDT plays an important role in DERITS, and we aim to analysis the impact of FDT on the model performance. Thus, we consider a special case of FDT, which is when we set k = 0, the derivation is removed and FDT is degraded to naive Fourier transform. In addition to this setting, we also vary different orders (k) of derivation to test the effectiveness. Table 3 has shown the results on three datasets. We can easily observe that the performance of DeRiTS can beat the variant version without the derivation, which has demonstrate the effectiveness of FDT. Moreover, with the increase of order, the original time series would be derived too much. This might cause the information loss which leads to performance degradation accordingly.

Impact of Multi-order Stacked Architecture As aforementioned in Section 4.2, we organize DERITS as a parallelstacked architecture for multi-order fusion. Thus we aim to study the impact of such a stacked architecture. In contrast to multi-order stacked DERITS, we have considered another situation of individual-order DERITS that removes the parallel-stacked architecture. Figure 3 has shown the performance comparison with individual-order DERITS in the Exchange, ILI, and Weather datasets. It can be easily observed that without the parallel-stacked architecture for multi-order fusion, the individual-order DERITS achieves much worse performance than the original one even under different orders, which signifies the necessity of our multi-order design in the frequency derivative learning architecture.

Lookback Analysis We aim to examine the impact of the lookback window on forecasting performance of DERITS. Figure 4 has demonstrated the experimental results on the Exchange, Weather, and ILI datasets. Specifically, we maintain the prediction length as 96 and and vary the lookback length from 48 to 288 on Exchange and Weather datasets. For the ILI dataset, we keep the prediction length at 36 and alter the lookback window size from 24 to 72. From the results, we can observe that in most cases, larger lookback length would bring up less prediction errors; this is because larger input includes more temporal information. Also, larger input length would also bring more noises hindering forecasting, while our DERITS can achieve comparatively stable performance.

Efficiency Analysis To conduct the efficiency analysis for our framework, we report the training time of DERITS across various prediction lengths, and we also include the training time of the Non-Stationary transformer [Liu et al., 2022b] for comparison, coupled with its corresponding backbones such as FEDformer, Autoformer and Transformer. The experiments are conducted under the prediction length with the same input length of 96 on the Exchange dataset. As shown in Table 4, the results prominently highlight that our model exhibits superior efficiency metrics. Our DERITS significantly reduces the number of parameters thus enhancing the computation speed. In particular, our model showcases an average speed that is several times faster than the baselines. These findings underscore the efficiency gains achieved by our model, positioning it as a compelling choice for nonstationary time series forecasting.

5.5 Visualization Analysis

FDT and Non-stationarity To study the Fourier derivative transformation in DERITS, we visualize the original signals and derived signals for comparison. Since the direct outputs of FDT are complex values that are difficult to visualize completely, we thus transform the derived frequency components back to the time domain via inverse FDT, which allows us to show the corresponding time values for visualization. Specifically, we choose two non-stationary time series from the Weather dataset and Exchange dataset, respectively. As shown in Figure 5, we can observe the original signals have included obvious non-stationary oscillations and trends. In contrast, the derived signals exhibit a larger degree of stationarity compared with raw data. This further reveals that learning in the derivative representation of signals is more stationary and thus can achieve better performance.

Case Study of Forecasting To further analysis the model performance, we carry out the case study for non-stationary time series forecasting. Figure 6 demonstrates the visualization of forecasting results of DERITS and NSTransformer [Liu et al., 2022b] with the prediction length as 96 and lookback length as 96. Upon careful observation of it, it becomes evident that DERITS can be capable of aligning with the ground truth when the time series distribution is largely shifted, while the baseline method deviates from the true values. This demonstrates our model's adaptability to shifts. We include more visualizations in Appendix E.

6 Conclusion Remarks

In this paper, we propose to address non-stationary time series forecasting from the frequency perspective. Specifically, we utilize the whole frequency spectrum for the transformation of time series in order to make full use of time series distribution. Motivated by this point, we propose a deep frequency derivative learning framework DERITS for non-stationary forecasting, which is mainly composed of the Frequency Derivative Transformation and the Order-adaptive Fourier Convolution Network with a parallel-stacked architecture. Extensive experiments on real-world datasets have demonstrated its superiority. Moreover, distribution shifts and non-stationarity are actually a pervasive and crucial topic for time series forecasting. Thus we hope that the new perspective of frequency derivation together with the DERITS framework can facilitate more future related research.

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China (No. 62072153) the Anhui Provincial Key Technologies R&D Program (No. 2022h11020015), the Shanghai Baiyulan Talent Plan Pujiang Project (23PJ1413800), and the National Natural Science Foundation of China (No. 623B2043).