StreamTimer: Efficient Inference for Long-Context Time Series Transformers

1 Introduction

Long-term time series forecasting has evolved rapidly over the past decades, witnessing a paradigm shift from traditional statistical methods to deep learning architectures [Jin et al., 2024; Wang et al., 2024b]. While these substantial architectural innovations, their fundamental operational protocol remains surprisingly static, characterized by a reliance on a fixed lookback window [Qiu et al., 2024]. Specifically, models are constrained to a pre-determined number of historical time steps (e.g., 96 days or 720 hours) to forecast future values [Liu et al., 2024b]. This rigid sliding window approach persists even as underlying architectures grow increasingly complex, creating a misalignment between the model's intrinsic capacity and its effective utilization of context.

This operational rigidity introduces a paradox in modern TSF. That is, longer historical contexts theoretically harbor richer temporal information, but extending the window to capture it is computationally expensive. Given the quadratic complexity of self-attention, lengthening the window substantially increases memory and compute costs [Zhou et al., 2021]. Besides, empirical evidence suggests that simply enlarging the window often degrades performance due to attention dispersion, insufficient training samples, and overfitting to irrelevant noise [Nie et al., 2023; Liu et al., 2024d]. Consequently, practitioners are trapped in a dilemma: they must either truncate history or risk computational intractability and model instability. This suggests that the fixed-window paradigm is inherently suboptimal for capturing the continuous and unrestricted dynamics of time series data.

To resolve these structural limitations, we propose a transition from the repeated recomputation of fixed windows to an Inference-Only Streaming Autoregressive Framework. Inspired by the efficient caching mechanisms in Large Language Models (LLMs), our approach views forecasting as a continuous, stateful process. By replacing repeated fullcontext recomputation with a one-time context warmup followed by incremental decoding, we reduce the complexity of extending context from quadratic to linear. This allows the model to efficiently leverage extensive historical data without the latency penalties of massive fixed windows. Moreover, as an inference-time optimization, this framework requires no retraining, allowing the trained model to achieve superior long-context performance directly

However, the direct adaptation of NLP-style Key-Value (KV) caching [Xiao et al., 2024] to time series is brittle due to two domain challenges. First, time series data are non-stationary. Current baselines typically employ Reversible Instance Normalization (RevIN) [Kim et al., 2021; Qiu et al., 2024] to mitigate statistical drifts; however, this creates distribution cache mismatches, meaning that the key vectors stored at time t effectively reside in a different feature space than Query vectors generated at t + 1. Second, unlike semantically dense text, time series history is replete with redundancy, where indiscriminately caching noise diminishes the efficacy of the attention mechanism.

To overcome these issues, we introduce StreamTimer, a robust streaming framework tailored for TSF. We propose Cache-Consistent Normalization to dynamically align cached representations with the current statistical regime, and incorporate window attention with attention sinks to maintain temporal coherence while retaining only high-value historical tokens. Our contributions lie in three aspects:

• We propose the first inference-only streaming framework with Timer backbones for TSF that effectively utilizes the lookback window, decoupling context length from computational cost.
• We identify and resolve the normalization mismatch problem in generative trained large time series models, ensuring mathematically consistent distribution during streaming inference.
• Experiments validate that StreamTimer achieves significant speedup while outperforming SOTA baselines, particularly on ultra-long sequences.

2 RelatedWork

2.1 Time Series Forecasting

Contemporary deep learning architectures for time series forecasting predominantly focus on modeling complex global temporal dependencies and multivariate interactions from historical data. Early studies built on Recurrent Neural Networks [Lin et al., 2023] and Temporal Convolutional Networks [Luo and Wang, 2024] to capture local patterns through gated recurrences and dilated convolutions. More recently, Transformers have emerged as the mainstream for long-horizon forecasting due to their capability to capture long-range dependencies. Specifically, Informer [Zhou et al., 2021] introduces sparse attention tailored to long sequences, PatchTST [Nie et al., 2023] and iTransformer [Liu et al., 2024b] leverage patch- and channel-centric designs, and TimeMixer [Wang et al., 2024a] and TimeMoE [Shi et al., 2025] enhance the exploitation of multivariate structures and multi-scale temporal patterns. While these backbones provide robust representation capabilities, they typically rely on a fixed-lookback-fixed-forecast recomputation paradigm.

2.2 Generative Pre-training for Large Time Series Models

The field of TSF has transitioned from small-scale, taskspecific deep learning models [Wu et al., 2025; Lin et al., 2024; Wu et al., 2024; Zeng et al., 2023] to Large Time Series Models (LTSMs) [Zhou et al., 2023; Jin et al., 2024; Liu et al., 2024a; Woo et al., 2024; Ansari and Stella, 2024] powered by unsupervised pre-training on massive datasets. Inspired by the success of LLMs, recent works have explored generative pre-training paradigms to build foundation models for time series.

Early efforts, such as Time-LLM [Jin et al., 2024] and LLM4TS [Chang et al., 2024], repurposed pre-trained LLMs (e.g., LLaMA, GPT-2) by aligning time series modalities with textual tokens. However, these methods often suffer from high computational overhead due to cross-modality alignment. More recently, native LTSMs like Timer [Liu et al., 2024d] have emerged, employing a decoder-only Transformer architecture trained via next-token prediction on largescale heterogeneous datasets. Timer unifies various forecasting tasks into a single generative framework, demonstrating strong zero-shot generalization capabilities. Building on this, Timer-XL [Liu et al., 2024c] addresses the context bottleneck by introducing TimeAttention to capture fine-grained intraand inter-series dependencies, expanding the context window to thousands of tokens.

Despite these architectural advancements, both Timer and Timer-XL adhere to a recomputation-based inference protocol. To forecast a future horizon, these models require processing a fixed lookback window from scratch at each step. This rigid mechanism incurs quadratic computational complexity O(TL2 ) with respect to the context length L and the forecast horizon T, severely limiting their efficiency when handling extremely long historical contexts in real-time streaming scenarios.

3 Approach

3.1 Preliminaries

We construct our streaming framework upon the foundation of LTSMs, specifically adopting the decoder-only Transformer architecture employed by Timer [Liu et al., 2024d] and Timer-XL [Liu et al., 2024c]. Unlike traditional deep forecasters that rely on encoder-decoder structures, Timer and Timer-XL treat time series forecasting as a native sequence generation task.

Patch-basedTokenization. Given a rawunivariate time series X = {x1, . . . , xL} ∈ RL, the model first transforms the continuous data points into a sequence of discrete embeddings. We adopt a patching strategy where non-overlapping windows of size P are aggregated into tokens. The resulting sequence S = (s1, . . . , sN) consists of N = L/P tokens, where each st ∈ RP represents a local temporal segment. These segments are then projected into a high-dimensional latent space RD via a learnable linear embeddingWemb:

where PEt denotes the positional encoding required to preserve temporal ordering.

Causal Contextualization. The core computation occurs within a stack of K decoder-only Transformer blocks. To ensure the autoregressive property, a strict causal mask M is applied to the self-attention mechanism. For the k-th layer, the representation h(k) t is updated by attending solely to historical tokens:

where Mij = -∞ if j > i. This mechanism guarantees that the information flow at step t is strictly isolated from future signals.

Generative Objective. The training paradigm mirrors that of LLMs, formulating forecasting as a next token prediction (NTP) problem. The joint probability of the time series is factorized into a product of conditional probabilities:

In the continuous domain of time series, this probabilistic objective manifests as minimizing the reconstruction error between the predicted next patch and the ground truth. The model outputs the prediction ˆst+1 via a linear projection head

Whead applied to the final hidden state h(K) t . The optimization objective is defined as the Mean Squared Error (MSE) over the sequence:

Through this formulation, the model learns to synthesize future trajectories based on the unified history representations, establishing the theoretical basis for our subsequent streaming optimizations.

3.2 Hybrid-Scale Normalization for Patch-based Generators

RevIN [Kim et al., 2021] effectively mitigates distribution shifts by normalizing the input using the mean and variance over the entire lookback window. However, for patch-based autoregressive models like Timer, relying solely on global statistics may overlook rapid and high-frequency fluctuations within individual patches. Conversely, normalizing strictly at the patch level risks discarding the global amplitude information necessary for accurate trajectory reconstruction.

To address this dilemma, we introduce a Hybrid-Scale Normalization strategy that dynamically calibrates the distribution by fusing global and local statistics. Let X ∈ RL×C denote the input lookback window with length L and C variables. We first compute the global statistics across the temporal dimension:

Simultaneously, for the i-th patch si ∈ RP×C , where P is the patch size, we compute its local statistics µ(i) local and σ(i) local .

To balance the stability of global trends with the sensitivity to local variations, we define the normalization statistics as a convex combination of both scales, controlled by a mixing coefficient α:˜µ(i) = α · µ(i)

Each input patch si is then normalized as ¯si = (si - ˜µ(i) )/˜σ(i) . Crucially, to ensure consistency, the generative output ˆsi is denormalized using the same hybrid statistics:

In our implementation, α is a learnable parameter initialized to 0.5. This design allows the model to remain robust to non-stationary drifts while retaining essential global context, thereby enhancing the generation stability of Timer and Timer-XL. We provide detailed empirical and theoretical verification of this choice in Appendix C.

3.3 Linear-Complexity Streaming via KVCaching

The standard inference paradigm of Transformers typically involves a recomputation strategy, where the entire history is processed repeatedly to generate the next token. For a sequence of length L, this results in a computational complexity of O(L2 ), rendering long-context forecasting prohibitively expensive. To address this, we implement a Streaming Inference Mechanism based on KV Caching, adapted for the patch-based architecture of Timer.

Mechanism. Instead of recalculating the representations of historical patches at every step, we cache the Key K and Value V matrices computed in previous iterations. Let st denote the input patch at step t. During the streaming process, the model only computes the Query qt, Key kt, and Value vt for the current patch. The attention mechanism then attends to the concatenation of the cached states and the current states:

Here, [·; ·] denotes concatenation along the temporal dimension. By maintaining this persistent memory state, the computation for the current step t becomes independent of the history length t - 1 in terms of feature extraction, involving only a single forward pass of the Transformer block.

Implementation Protocol. Operationally, we reformulate the inference process as a stateful autoregressive protocol over the discretized time horizon. The procedure commences by initializing an empty Key-Value memory. Unlike standard batch processing, the model subsequently enters a streaming phase where it ingests exclusively the current patch st at each time step t. During this incremental pass, the transformer blocks query the historical context persisted in the cache to derive the current representation yt, Crucially, to constrain memory growth, we employ an attention-based eviction policy: post-computation, we retain only the [REG] token serving as an attention sink and the subset of historical tokens with the highest attention scores. Low-scoring tokens are discarded from the memory bank. This selective retention strategy allows us to reconstruct the coherent forecast trajectory with a bounded memory overhead per step.

4 Experiments

We evaluate StreamTimer across three dimensions: (1) supervised training as a task-specific forecaster, (2) transfering as a zero-shot forecaster, and (3) the effectiveness of normalization and model efficiency. Since performance saturation on previous benchmarks [Qiu et al., 2024; Wang et al., 2024b] often conceals the challenges of the long-context forecasting paradigm, we establish new long-context forecasting benchmarks by following the experimental settings of TimerXL [Liu et al., 2024c]. Detailed experimental configurations are provided in Appendix A.

4.1 Univariate Time Series Forecasting

Setups To address the issue of data scarcity often encountered when extending context lengths in univariate time series [Makridakis et al., 2020], we follow the experimental protocol of Timer-XL [Liu et al., 2024c]. Specifically, we leverage the multivariate datasets from [Liu et al., 2024b] and formulate univariate forecasting tasks by applying channel independence. The core objective of our experiments is to evaluate model robustness given long historical inputs, rather than merely pursuing extended forecasting horizons. Consequently, we fix the forecast horizon and focus on expanding the lookback window to monthly and yearly scales. Furthermore, we establish a challenging long-context univariate benchmark named ERA5-S, derived from the ECMWF Reanalysis v5 (ERA5) dataset [Mu˜noz-Sabater et al., 2021]. Leveraging its 40-year temporal span, this benchmark is specifically designed to assess land-surface temperature forecasting capabilities based on yearly contexts.

Results As illustrated in Figure 3, increasing the context length generally leads to improved prediction accuracy across univariate datasets. While previous state-of-the-art models like Timer-XL show promising results, they tend to encounter performance saturation or even deterioration in excessively long contexts (e.g., monthly and yearly levels), likely due to the accumulation of noise. In contrast, our proposed StreamTimer exhibits superior robustness. As shown in the curves, StreamTimer consistently outperforms Timer-XL and DLinear, particularly achieving lower MSE in the longest lookback settings across all datasets. This indicates that StreamTimer effectively mitigates the impact of irrelevant history while capturing critical long-term dependencies. A similar conclusion can be drawn from the ERA5-S benchmark (Table 1), where StreamTimer surpasses both PatchTST and Timer-XL

on all sites, demonstrating its efficacy in handling complex meteorological data with extended contexts.

Revisiting Non-stationary Forecasting We delve into the widespread non-stationarity in univariate tasks. While normalization techniques like RevIN [Kim et al., 2021] generally benefit Transformer performance by aligning statistical distributions, their limitations become pronounced in ultralong contexts. We argue that standard normalization oversimplifies the time-varying statistics inherent in long history. Specifically, as the context length increases, both the internal distribution shifts within the input and the statistical discrepancy between input and output are significantly exacerbated. Forcing such complex sequences into a unified distribution constrains the model capacity of Transformers. The model is compelled to handle these distorted distributions rather than learning distinct temporal variations, leading to side effects such as mode collapse and oversmoothed predictions. In contrast, our proposed normalization optimizes the distributional properties, thereby liberating the model's capacity to focus on the forecasting task itself rather than struggling with distribution alignment. As evidenced in Table 1, this approach allows Timer-XL and Timer to achieve superior results on ERA5-S without the pitfalls of standard normalization.

4.2 Multivariate Time Series Forecasting

Setups Following the experimental protocols of TimerXL [Liu et al., 2024b], we evaluate multivariate forecasting performance. To advance the development of generated transformers, we focus on the rolling forecast setting, where a single trained model adapts to various prediction horizons by recursively integrating previous predictions into the lookback window for the next iteration. Furthermore, we establish a long-context multivariate benchmark, the ERA5 multistation land-surface temperature prediction (ERA5-MS). This benchmark is designed to facilitate the learning of complex temporal dynamics and inter-variable correlations with sufficient training samples.

Results As shown in Table 2, StreamTimer achieves the best results on the established benchmarks. Our comparison focuses primarily on its predecessors, Timer and Timer-XL.

While Timer-XL already demonstrates strong capabilities over Timer by leveraging extended contexts, StreamTimer pushes the performance boundary even further. It is particularly noteworthy that StreamTimer significantly outperforms Timer-XL, especially on non-stationary time series (improving by 8.2% in ETTh1). We attribute this substantial improvement over Timer-XL to two key innovations: first, our proposed normalization effectively resolves the distribution shift issues that hinder Timer-XL in ultra-long contexts; second, the streaming forecasting mechanism enhances stability during continuous inference. Furthermore, we find StreamTimer achieves 69 wins out of 80 tests under different metrics and conditions, and maintains strong performance even on lowdimensional datasets with low predictability (e.g., ETT), underscoring its effectiveness and superiority.

Performance on ERA5-MS To evaluate the model's capability in multivariate time series forecasting, we conducted experiments on the real-world ERA5-MS benchmark as shown in Table 3. We compared StreamTimer with strong baselines ranging from channel-independent approaches PatchTST, UniTST to large-scale generative models Timer, Timer-XL. The results indicate that StreamTimer effectively combines the benefits of advanced tokenization and architectural design, surpassing the Timer-XL. By consistently achieving the best metrics in both MSE and MAE across diverse stations, StreamTimer proves to be the most effective solution for multivariate forecasting in this setting.

Zero Shot Forecasting Results To evaluate the generalization capability of StreamTimer on unseen datasets, we conducted zero-shot forecasting experiments. Following previous protocols [Jin et al., 2024; Kudrat et al., 2025], we selected ETTh1, ETTh2, ETTm1, and ETTm2 as source datasets and performed direct inference on the remaining datasets as targets. Table 5 presents the results for a prediction horizon of 192, where StreamTimer demonstrates consistent superiority across transfer tasks, significantly outperforming PatchTST and the baseline models, Timer and Timer-XL. This robustness across different granularities and domains is primarily attributed to StreamTimer's streaming dynamic adaptation mechanism. This mechanism enables the model to keenly capture and adapt to the distribution evolution of the target domain during the inference phase, thereby effectively mitigating the distribution shift challenges between the source and unseen target domains.

Ablation onHybrid-scaleNormalization To verify the effectiveness of our proposed normalization strategy, we conducted a comprehensive ablation study comparing it with several state-of-the-art normalization techniques, including RevIN [Kim et al., 2021], SAN [Liu et al., 2023], DishTS [Fan et al., 2023], and MPNorm [Zhang and Dong, 2025]. Table 4 reports the results on ETTh1 and Weather datasets using both Timer and Timer-XL as backbones. The results demonstrate that our proposed module consistently achieves the lowest MSE and MAE across almost all prediction lengths. While RevIN and MPNorm serve as strong baselines, our method further boosts performance, particularly on the ETTh1 dataset where non-stationarity is significant. Notably, while SAN and Dish-TS employ model-based statistics prediction that necessitates complex pre-processing and auxiliary modules, this added complexity yields negligible performance gains.

Ablation on KV Cache As shown in Table 6, our ablation study uncovers that RevIN induces severe distribution shift during KV cache streaming. Specifically, RevIN's error rises markedly as the horizon extends to 720, reflecting its failure to manage statistical variations in long streams. On the contrary, our method maintains a stable and low MSE, verifying that it causes almost no harm to model performance in streaming scenarios.

4.3 Model Analysis

Model Efficiency To comprehensively evaluate model efficiency, it is essential to consider the input context length T, the number of variables N, and crucially, the prediction horizon L. Unlike 1D sequences, the computational overhead in multivariate time series forecasting is dictated by the interplay of these three dimensions. We provide a theoretical complexity analysis comparing StreamTimer against the baseline models, Timer and Timer-XL. Specifically, the complexity of Timer, which adopts channel independence, scales as O(NT 2L). Meanwhile, Timer-XL, which incorporates channel dependence to capture inter-variable correlations, sees its complexity escalate to O(N 2T 2L). Crucially, the computational burden of both architectures involves a multiplicative coupling with the prediction horizon L and quadratic terms of T, leading to significant overhead accumulation in longterm forecasting scenarios. In contrast, leveraging the streaming forecasting mechanism, StreamTimer effectively avoids the redundant re-computation of Keys and Values. As derived in Table 10, StreamTimer significantly optimizes the overall complexity to O(NTL). This indicates that the computational cost of StreamTimer scales linearly with the total sequence length, rather than multiplicatively. As shown in Figure 4, by eliminating these redundant operations, StreamTimer demonstrates an order-of-magnitude efficiency advantage, particularly in tasks involving extensive contexts and extended prediction horizons.

Visualization of Hybrid-Scale Normalization Figure 5 visualizes the effectiveness of the proposed normalization strategy. The blue and orange solid lines represent the ground truth and the model prediction, respectively. While RevIN relies on a static global mean (purple dotted line), which fails to capture temporal variations, our Hybrid-Scale Normalization utilizes local means (green and red dashed lines) that dynamically track local trends. Specifically, the red dashed line indicates that the model successfully infers the future distribution mean based on historical patterns. The alignment between the local statistics and the actual data demonstrates the method's precision in mitigating the impact of non-stationarity.

5 Conclusions

In this paper, we propose StreamTimer, a streaming inference framework designed to address the limitations of fixed lookback windows in time series forecasting. By reformulating the forecasting process into a continuous state update mechanism, we effectively leverage extremely long historical information without incurring additional computational burden. Our core technical innovations-particularly Cache-Consistent Normalization tailored for non-stationary data-effectively overcome the domain-specific barriers to applying KV caching in time series. Extensive experiments validate the efficiency of StreamTimer, demonstrating substantial reductions in both computational and memory costs while matching or even exceeding predictive performance. These results prove that decoupling context length from computational cost via streaming is both feasible and efficient, paving a new direction for the development of ultra-long sequence modeling and Time Series Foundation Models.

Acknowledgments

This work was supported in part by the Zhejiang Provincial Science and Technology Plan Project under Grant 2024C03261 and in part by Key Scientific Research Base for Digital Conservation of Cave Temples(Zhejiang University), National Cultural Heritage Administration.

Contribution Statement

Xiyu Meng and Yuhan Wu contributed equally to this work. Yabo Dong is the corresponding author.