SMLDR: Spectral Memory Learner with Dual-Retrieval for Time Series Forecasting

1 Introduction

Time series forecasting plays an important role in many practical applications, such as energy management [Zhang et al., 2024; Yang et al., 2025], weather monitoring [Price et al., 2025; Conti, 2024], traffic flow control [Qiu et al., 2025; Cao et al., 2025] and economic planning [Wu et al., 2025]. The core challenge of prediction is to model the periodic components intertwined with non-cyclical fluctuations [Zhou et al., 2022b; Yi et al., 2025].

Early statistical models, such as ARIMA [Ho and Xie, 1998] and exponential smoothness [Gardner Jr, 1985], assume that data has stable or linear dynamic characteristics, so it is usually difficult to generalize to complex, nonlinear and multi-cycle situations. In recent years, the deep learning model has alleviated these limitations by learning flexible time representation. In view of the importance of periodicity, a series of studies are committed to separating periodic components from dynamic characteristics [Wu et al., 2021; Zhou et al., 2022b; Liu et al., 2024; Lin et al., 2024a]. These methods are usually decomposed by time-domain smoothing operations, such as moving average [Ansley et al., 1977], STL local weighted regression [Cleveland et al., 1990] or exponential smoothing. This smoothing operation is equivalent to the low-pass filtering in the frequency domain [Oppenheim, 1999; Zhang et al., 2025], which inhibits the high-frequency component. At the same time, frequency domain prediction methods often learn periodicity by selecting, filtering or re-weighting frequency components, such as random sampling, predefined filters or Top-K amplitude selection [Zhou et al., 2022b; Xu et al., 2023; Zhou et al., 2022a; Piao et al., 2024; Yue et al., 2025; Fei et al., 2025]. Both methods rely on the operation of the frequency dimension to model periodicity.

These strategies assume that meaningful periodic information can be separated by selecting frequencies. However, in real scenarios, due to spectral leakage [Oppenheim, 1999; Glorot and Bengio, 2010], multi-period harmonic overlap And periodic gradient, cyclic energy tends to spread to multiple frequency bands and overlap with non-periodic components. As shown in figure 1, this makes the frequency selection ambiguous, that is to say, a single frequency component will contain both periods and residuals. We call this phenomenon spectral non-identifiability. Therefore, strategies based on frequency selection may discard valuable periodic signals or inadvertently retain noise, resulting in performance decline in multi-cycle and long-term prediction scenarios.

We propose SMLDR (spectral memory learner with dual retrieval), which is a new frequency domain prediction framework aimed at overcoming spectral non-identifiability. A key observation is that although the time-domain waveforms with similar periodicity may be completely different, their normalized spectral prototypes are highly similar. Instead of trying to design more complex frequency selection rules, SMLDR uses a learnable spectral memory library to explicitly store these recurring spectral prototypes. Given an input sequence, SMLDR uses its spectral representation as a query to retrieve the most similar cycle patterns, which alleviates the ambiguity introduced by spectral leakage. Components that cannot be explained by the retrieved prototype are regarded as non-periodic residues, and a lightweight time domain branch is used for modeling. Through this double retrieval design, SMLDR explicitly decouples the periodic component and the residual component in the spectral space.

In summary, the contributions of this paper are as follows:

• We point out the core limitations of existing time series decomposition and frequency domain modeling methods, namely the ambiguity in frequency-level selection caused by spectral leakage and harmonic overlap.
• We propose SMLDR, which models periodicity through a learnable spectral memory bank and achieves the decoupling of periodic components and residuals in the frequency domain space.
• The dual-branch design explicitly separates the modeling of reusable periodic prototypes from lightweight timedomain residual modeling, making it highly suitable for prediction in multi-period scenarios.

2 RelatedWork

Time Series Forecasting. Research on Time Series Forecasting (TSF) is extensive, ranging from statistical models such as ARIMA [Ho and Xie, 1998] and exponential smoothing [Gardner Jr, 1985] to various deep learning methods nowadays. Recurrent and convolutional models form an important branch, including LSTNet [Lai et al., 2018], DeepAR [Salinas et al., 2020], SCINet [Liu et al., 2022a], and TimesNet [Wu et al., 2023]. Recently, Transformer-based methods have become a hot topic in time series forecasting, mainly improving prediction performance by enhancing temporal representation and attention mechanisms. Representative models include Informer [Zhou et al., 2021], Autoformer [Wu et al., 2021], FEDformer [Zhou et al., 2022b], Pyraformer [Liu et al., 2022b], PatchTST [Nie et al., 2023], and iTransformer [Liu et al., 2024]. Meanwhile, MLP-based models, such as TIDE [Das et al., 2023], DLinear [Zeng et al., 2023], SparseTSF [Lin et al., 2024b], TTM [Ekambaram et al., 2024] and XLinear [Chen et al., 2026], demonstrate that competitive prediction accuracy can be achieved using simplified models.

Time-series decomposition. Explicit modeling of periodic components is an important research direction in time series forecasting (TSF). A large number of studies have adopted traditional Seasonal-Trend Decomposition (STD) methods [Yi et al., 2025] to decompose time series into seasonal components and trend components, such as Transformer-based models Autoformer [Wu et al., 2021] and FEDformer [Zhou et al., 2022b]. Subsequently, DLinear [Zeng et al., 2023] further adopted fixed decomposition operators combined with linear prediction, while Leddam [Yu et al., 2024] replaced manually designed filters with learnable decomposition kernels. Other methods model periodicity more explicitly, including DEPTS [Fan et al., 2022], which parameterizes periodic functions through time; and CycleNet [Lin et al., 2024a], which learns stable periodic patterns through global periodic indices and uses lightweight Multi-Layer Perceptrons (MLPs) to predict residuals.

Frequency-domain modeling Frequency-domain modeling has become a popular paradigm for capturing global periodic structures in time series. Based on FFT-based spectral representations [Duhamel and Vetterli, 1990], , a series of recent works have explored frequency-domain forecasting from different perspectives. FEDformer [Zhou et al., 2022b] implements a fast attention mechanism through low-rank approximate transformations in the frequency domain. FiLM[Zhou et al., 2022a] applies Fourier analysis to preserve historical information while mitigating noise. FITS[Xu et al., 2023] uses frequency domain interpolation for prediction, essentially acting as a low-pass filter. FreTS[Yi et al., 2023] retains the simplicity and efficiency of the MLP architecture, while combining the global perspective and energy aggregation properties of the frequency domain. FAN[Ye et al., 2024] mitigates the effects of nonstationary by filtering the top K dominant components for each input instance in the Fourier domain. Amplifier[Fei et al., 2025] and Fredformer[Piao et al., 2024] use different approaches to solve the problem that models tend to ignore low-energy components. FreEformer[Yue et al., 2025] solves the low-rank problem of vanilla attention in frequency domain transformers.

Unlike those existing methods that mainly focus on frequency selection or filtering strategies, we re-examine periodic modeling from the perspective of spectral non-identifiability. In our opinion, the difficulty lies in distinguishing reusable and stable periodic components in the highly coupled spectral space of period and residual. Based on this insight, we have proposed a frequency domain prototype retrieval framework, which models the cycle pattern through the learned spectral prototypes and avoids relying on fixed period assumptions or unified frequency rules.

3 Preliminaries

3.1 Time Series Forecasting

Let X → RC→L denote a multivariate time series segment with C variates (channels) and a look-back length L. The forecasting task aims to predict the future horizon Y → RC→T of length T . Given a dataset D = ¶(X(i) ↪ Y(i) )♦Ni=1 , the goal is to learn a mapping function

that minimizes the expected forecasting error under a predefined loss function. Long-term forecasting becomes particularly challenging when the underlying dynamics exhibit multiple coexisting periodicities, gradual period variation, and non-stationary residual perturbations.

3.2 Fast Fourier Transform

Frequency-domain modeling is a common approach for capturing global periodic structures in time series. Given a discretetime signal x(t) of length L, its frequency-domain representation is obtained via the Fast Fourier Transform (FFT):

FFT provides an efficient transformation from the time domain to the spectral domain with a computational complexity of O(L log L), enabling scalable modeling of long sequences. Many frequency-based forecasting methods leverage this transformation to perform filtering, frequency selection, or spectral interpolation, under the assumption that meaningful periodic structures are compactly represented in the spectral space.

3.3 Spectral Non-identifiability

While frequency-domain models typically assume that periodic signals s(t) and residuals r(t) are spectrally orthogonal, this ideal separation is rarely achieved in practice. We formalize this challenge as Spectral Non-identifiability, which stems from two primary factors:

1) Spectral Leakage: Due to the finite look-back window L, the observed spectral Z is a convolution of the true spectrum and the Fourier transform of the window function WL(ω). This spreads energy across neighboring bins, blurring the boundaries of periodic components [Yang and Zhou, 2022].

1. Component Entanglement: Real-world residuals often possess structured spectral content that overlaps with periodicities. The observed power spectrum satisfies:

The non-zero cross-term indicates that individual frequency coefficients are coupled. Consequently, simple energy-based filtering (e.g., Top-k selection) cannot uniquely identify periodic components, leading to accumulated decomposition errors. This motivates SMLDR to bypass frequency selection or filtering and instead employ a memory-based dual-retrieval mechanism to distill identifiable spectral prototypes from a global perspective.

4 Methodology

4.1 Overall Architecture

The SMLDR proposed in this paper adopts a decompositionbased prediction paradigm, which decomposes the input time series into periodic components and residual components in the frequency domain. The design motivation of SMLDR stems from spectral unidentifiability. Traditional time series analysis regards observations as the superposition of periodic patterns and residual changes. However, due to spectral leakage and multi-period superposition, the periods and residuals in the frequency domain are highly coupled, resulting in unstable frequency selection strategies.

Algorithm 1 Forward Pass of SMLDR

Require: Historical series X → RC→L.

Ensure: Predicted future values ˆY → RC→T .

Step 1: Normalize and Frequency Transform

• 1: Xnorm ↔RevIN(X)
• 2: Xfft ↔ rFFT(Xnorm) Step 2: Spectral Representation
• 3: Sraw ↔Concat(Re(Xfft)↪ Im(Xfft))
• 4: S↔Normalize(Sraw)
• 5: Q↔SWQ Step 3: Spectral Memory Dual-Retrieval
• 6: Z↔MemoryRetrieve(Q↪MK) ¶Eq. (15)♦
• 7: Ehist ↔ZMhist V ¶Eq. (16)♦
• 8: Epred ↔ZMpred V ¶Eq. (17)♦ Step 4: Seasonal Reconstruction
• 9: Xsea ↔WhistEhist
• 10: ˆYsea ↔WpredEpred Step 5: Aperiodic Residual Modeling
• 11: Xres ↔Xnorm ↓Xsea
• 12: ˆYres ↔MLP(Xres)

Step 6: Aggregation and Inverse Normalization

• 13: ˆYnorm ↔ ˆYsea + ˆYres
• 14: ˆY↔Denorm( ˆYnorm)
• 15: return ˆY

To solve this problem, SMLDR explicitly separates the periodic components across the entire spectral dimension, represents the periodic components with reusable spectral prototypes, and uses the input context as query to retrieve them, thereby activating multiple periodic patterns simultaneously.

As shown in the figure 2, SMLDR consists of two branches. The spectral memory branch captures globally reusable periodic components and predicted it through dual retrieval. Meanwhile, the aperiodic residual branch models the remaining aperiodic residuals in the time domain. This approach decouples the periodic components from residual changes in the frequency domain and alleviates the entanglement between periodicity and residuals.

To alleviate distribution shift across different time windows, Reversible Instance Normalization (RevIN) is applied at both input and output stages. The final prediction is obtained as:

where ˆYsea → RC→T and ˆYres → RC→T denote the outputs of the spectral and residual branches, respectively.

4.2 Spectral Memory Bank

Spectral Representation

Given the normalized input time series Xnorm → RC→L, we transform it into the frequency domain to obtain a compact representation of its global periodic structure. Specifically, we apply the real-valued Fast Fourier Transform (FFT) [Duhamel and Vetterli, 1990] along the temporal dimension:

where F = ↗L◁2↘ + 1 due to conjugate symmetry. To facilitate learning with real-valued networks, we construct spectral features by concatenating the real and imaginary parts:

where Dfreq = 2F. Frequency-domain magnitude is highly sensitive to amplitude variations and non-stationary scaling [Ye et al., 2024], which tend to mask consistent spectral structures across different instance. To decouple spectral shape from instance-specific energy, we perform channel-wise normalization:

The normalized spectral representation is then given by:

yielding S → RC→Dfreq . The normalization removes amplitudeinduced variance while preserving spectral shape, which is crucial for stable prototype matching under non-stationary conditions. The scale factor εc is retained for adaptive energy restoration.

Finally, the normalized spectral signal is projected into a latent embedding space to serve as a query:

where WQ is a learnable linear projection. For multi-head retrieval, Q is reshaped into:

with dhead = dmodel◁H.

Prototype Retrieval in the Frequency Domain

We introduce a learnable Spectral Memory Bank to store reusable spectral prototypes that summarize recurring periodic patterns across the dataset. The memory consists of N prototypes with keys and values defined as:

For multi-head retrieval, all memory tensors are reshaped into:

Given the query embedding, similarity scores between the input spectrum and all spectral prototypes are computed independently for each channel and head:

Unlike the traditional attention mechanism, we have not applied Softmax normalization to the prototype dimension. This design choice is inspired by the spectral non-identifiability discussed in Section 2: due to spectral leakage and harmonic wave overlap, the real-world spectral is usually composed of multiple coupled periodic components. Forcing competitive normalization will artificially inhibit the coexistence model. On the contrary, the original similarity score Z → RC→H→N acts as a retrieval weight with bias guidance, promoting multiple prototypes to work at the same time. The potential embedding retrieved for historical reconstruction and future forecasting is then calculated as follows:

The outputs from all heads are concatenated to form:

Time-domain Reconstruction with Energy Restoration The retrieved prototype embeddings are projected back to the time domain to reconstruct periodic components:

where Whist and Wpred are learnable linear projections. Finally, the instance-specific energy is restored using the preserved scale factor:

By decoupling spectral shape selection from amplitude modeling, the Spectral Memory Bank enables stable retrieval of periodic prototypes under non-stationary conditions. The reconstructed seasonal component captures reusable periodic structures, while unexplained dynamics are naturally deferred to the residual modeling branch, forming a principled decomposition tailored for long-horizon forecasting.

4.3 Residual Modeling Branch

Once reusable spectral structures are extracted, the remaining signal primarily reflects aperiodic fluctuations, which are more effectively modeled in the time domain. The residual component of the input is computed as:

We employ a lightweight temporal MLP to model residual dynamics:

By explicitly separating reusable spectral prototypes from nonperiodic residuals, SMLDR achieves efficient long-term forecasting under multi-period superposition and non-stationary conditions. The final forecast is obtained by additively combining the seasonal and residual predictions:

This formulation explicitly reflects the assumption that observed time series can be decomposed into reusable periodic prototypes and a complementary aperiodic residual component, both learned jointly in an end-to-end manner.

4.4 Computational Efficiency Analysis

We analyzed the efficiency of the proposed method and focused on the double retrieval mechanism.

Dual Retrieval. Our dual retrieval decouples attention calculation from the sequence length L. The input sequence is projected into a compact query vector, which interacts with N fixed patterns. The complexity of this retrieval process is O(N · C · dmodel). Since N is a small constant (for example, N = 16), and usually N ⇐ L, the retrieval cost is actually O(1) relative to the time dimension L.

Overall Scalability. The rest of the operations (including FFT and linear projection) are logarithmically linear or linearly extended. Specifically, FFT requires O(L log L), and projection requires O(L · dmodel). Therefore, the total time complexity is O(L log L + L · C · dmodel), which is better than the quadratic extension of the standard Transformer.

5 Experiments

5.1 Experimental Setup

Datasets. We evaluate our model on ten widely used realworld benchmarks: ETT [Zhou et al., 2023](ETTh1, ETTh2, ETTm1, ETTm2), Weather [Wu et al., 2021], ILI, Exchange [Lai et al., 2018], Electricity, Covid-19, NN5. These datasets cover diverse domains and exhibit varying levels of periodicity and noise. Following prior work [Nie et al., 2023; Liu et al., 2024], all datasets are preprocessed and normalized using standard normalization. We split each dataset into training, validation, and test sets by the ratio of 6:2:2 (ETT 7:1:2). The look-back window is fixed to L = 96, and prediction horizons are selected from T → ¶96↪ 192↪ 336↪ 720♦

Baselines. We compare SMLDR against a comprehensive set of SOTAmodels: (1) Transformer-based: iTransformer [Liu et al., 2024], FEDformer [Zhou et al., 2022b]; (2) Frequencybased: Amplifier [Yue et al., 2025], Fredformer [Piao et al., 2024], TimesNet [Wu et al., 2023]; (3) Linear/MLP-based: DLinear [Zeng et al., 2023], CycleNet [Lin et al., 2024a], TQNet [Lin et al., 2025]

Implementation Details. All experiments in this study were carried out using PyTorch [Paszke et al., 2019] on one single NVIDIA RTX 3090 GPU with 24GB. Optimization is performed using Adam [Adam and others, 2014] and early stopping patience of 5 epochs. We use Mean Squared Error (MSE) as the loss function and report the results using both MSE and Mean Absolute Error (MAE) as evaluation metrics.

5.2 Long-termForecasting Performance

Table 1 reports the average prediction results of all methods on different datasets. SMLDR achieves lower prediction errors on the vast majority of datasets, indicating its strong effectiveness in modeling complex temporal dynamics. The ETT dataset has significant multi-scale periodicity, while Exchange and NN5 contain complex trends and noise with relatively hidden periodicity. Compared with Transformer-based models, SMLDR explicitly models periodic components in the frequency domain, can capture stable periodic patterns on datasets with different characteristics, and has higher computational efficiency than Transformer-like models. Compared with existing frequency-domain baseline methods, SMLDR does not rely on frequency selection-based strategies. Instead, it retrieves spectral prototypes from a learnable memory bank, which alleviates the ambiguity caused by the coupling between periodicity and residuals. In conclusion, SMLDR can flexibly model multiple coexisting periodic components and maintain stable performance on datasets with different periodicities.

5.3 Ablation Study

To evaluate the effectiveness of the key components in SMLDR, we conducted ablation studies on the ETTh1 and Weather datasets. And we designed three variants: (1) w/o SMB, which removes the spectral memory bank; (2) TimeMatch, which performs prototype retrieval in the time domain without using spectral prototypes; (3) w/o Residual, which removes the residual modeling branch.

As shown in Table 2, removing the Spectral Memory Bank leads to consistent performance degradation, indicating that explicit spectral prototype retrieval is essential for capturing stable periodic structures. Time-domain matching further degrades performance, especially for long-horizon forecasting, demonstrating the advantage of frequency-domain representations in handling phase variation and spectral coupling. Removing the residual branch also harms performance, confirming the necessity of explicitly modeling aperiodic components to prevent interference with periodic prototype learning.

5.4 Spectral Memory Activation Analysis

The activation distribution of spectral memory patterns in Figure 3 illustrates how SMLDR performs content-adaptive periodic modeling in the frequency domain. Each memory entry corresponds to a learned spectral prototype encoding a reusable periodic structure, while activation scores reflect its relevance to the current input. The activation dynamics show that different prototypes are selectively engaged across time, rather than relying on a single dominant frequency, which is consistent with the presence of multi-period structures and their gradual evolution in real-world time series. Moreover, the smoothly varying activation intensities indicate that SMLDR adapts prototype retrieval continuously.

This behavior due to the design of SMLDR: stable periodic patterns are captured by reusable spectral prototypes, while components not explained by these prototypes are handled by the residual branch at the same frequencies. Such a flexible separation mitigates spectral identifiability issues inherent to frequency selection or filtering rules.

5.5 Impact of hyperparameter N

We conducted a sensitivity analysis of the number of memory patterns N to test its impact on predictive performance. The experiment was conducted on three benchmarks including ETTh1, Exchange and Weather. For all datasets, the length of the backwindow is fixed at 96, and the prediction horizons considers H → ¶96↪ 192↪ 336↪ 720♦. The results of the report are obtained by taking an average of all predicted lengths.

As shown in Figure 4, there is no monotonous relationship between predictive performance and the number of memory patterns N. When N = 1, the performance of the model on each data set is generally poor, indicating that a single periodic prototype is not enough to capture the multi-periodic structure in the real-world time series. When N is in the range of 8 to 16, the model achieves good results on different data sets and has low sensitivity to the specific selection of N, which shows that a relatively small set of spectral prototypes is enough to represent the main periodic pattern.

Further increase of N does not bring additional benefits, and may even lead to a slight decline in performance, but the overall result is still close to the best. This phenomenon shows that there is redundancy in the periodic structure in the spectral space, and too many prototypes may introduce unnecessary characterizations and bring noise.

In general, the proposed spectral memory mechanism can effectively model the multi-period structure through a compact prototype set, and the selection of N is robust.

5.6 Performance with varied look-back length

We conduct experiments on the ETTh1 and Weather datasets, in which the look-back length is set to 48 to 720, and the prediction horizon was fixed at 96. Longer input windows often bring redundant or noisy historical information, increasing the difficulty of modeling.

As shown in Figure 5, the performance of many comparison methods fluctuates significantly with the increase of the look-back length, which indicates that they are insufficient in continuously capturing long-range historical dependencies. In contrast, SMLDR achieves the lowest mean square error (MSE) in all look-back lengths. With increasing the look-back length, the performance of SMLDR remains stable and even improves. These phenomena show that SMLDR can effectively extract reusable and valuable cycle patterns from long input sequences while reducing the negative impact of noise.

6 Conclusion

We propose SMLDR, a time-frequency prediction framework that models periodic components through reusable spectral prototypes instead of selecting multiple frequency components. By introducing a spectral memory bank and performing dual retrieval, SMLDR avoids frequency selection strategies and mitigates the strong coupling between periodicity and residuals caused by spectral leakage. Furthermore, SMLDR decouples the modeling of periodic prototypes from a lightweight time-domain residual branch, maintaining high computational efficiency. Extensive experiments on various benchmark datasets demonstrate that this prototype-centered spectral modeling paradigm can capture complex periodicities and exhibit excellent performance across different context lengths.