From Values to Tokens: An LLM-Driven Framework for Context-Aware Time Series Forecasting via Symbolic Discretization

1 Introduction

Time series forecasting (TSF) is critical for decision-making in domains such as energy [Cheng and others, 2025a], healthcare [Qiu et al., 2024], and finance [Feng et al., 2019]. In practice, forecasting requires not only modeling temporal dependencies, but also understanding how they interact with external contextual factors-such as static attributes or dynamic events [Liu et al., 2024]. Fundamentally, TSF can be viewed as learning a mapping from past values and contextual features to future outcomes [Jiang et al., 2025].

∗Corresponding Author.

To learn this mapping, researchers have proposed a comprehensive range of methods, ranging from classical statistical models to modern data-driven approaches. Traditional methods, such as ARIMA [Hyndman and Khandakar, 2008] and state-space models [Winters, 1960], rely on strong assumptions about data generation and often incorporate domain-specific priors. In contrast, recent data-driven approaches such as deep learning models aim to learn patterns directly from data without handcrafted assumptions. Architectures based on RNNs [Lai et al., 2018], CNNs [Cheng et al., 2025b], Transformers [Zhou et al., 2022], and MLPs [Zeng et al., 2023] have been widely adopted, each capturing different aspects of temporal dependencies. However, most of these models assume homogeneous numerical inputs and struggle to effectively incorporate complex contextual features, particularly those with heterogeneous modalities.

Beyond capturing temporal dependencies, there is an increasing emphasis in recent research on incorporating contextual features to enhance forecasting performance [Williams et al., 2025]. These features typically fall into two categories: dynamic exogenous variables (e.g., weather conditions, event indicators) and static attributes (e.g., product types, patient demographics, market segments). When contextual features share the same numerical modality as the target series, they can be directly modeled as additional input channels. However, many particularly high-value contextual features, such as clinical notes, policy texts, or user logs, are expressed in unstructured textual form. This heterogeneity poses significant challenges for aligning and integrating information across modalities.

To address these challenges, some studies have explored shallow fusion strategies to incorporate contextual features. Models such as DeepAR [Salinas et al., 2020] and Temporal Fusion Transformer (TFT) [Lim et al., 2021] typically concatenate external variables with time series or introduce gating mechanisms. While offering basic integration, these methods often rely on weak alignment and struggle to capture deep semantic interactions across modalities. More recently, LLMs have been introduced into time series forecasting [Sun et al., 2024]. Methods like Time-LLM [Jin et al., 2024] inject time series features into LLMs using linear adapters (Figure 1 (a)) or soft prompts (Figure 1 (b)). Although promising, these approaches fall short in resolving the structural discrepancies between numerical sequences and contextual features. Moreover, they fail to fully leverage the generative capabilities of LLMs, which are pretrained on large-scale corpora. This observation raises a fundamental question: Can time series be effectively modeled in a discrete token space to unlock the potential of LLMs?

Motivated by this question, and as illustrated in Figure 1(c), we explore a more expressive paradigm that remains insufficiently explored in prior TSF literature, which formulates time series forecasting as a multimodal discrete context understanding and generation problem powered by pre-trained LLMs. The key idea is to transform continuous numerical sequences into discrete tokens and embed them into the same semantic space as contextual language inputs. This formulation enables the full use of LLMs' capabilities in semantic understanding, contextual reasoning, and autoregressive generation. However, this paradigm introduces several non-trivial challenges. First, discretizing dynamic time series is more difficult than compressing static data, as it requires preserving temporal dependencies while reducing granularity. Second, even with symbolic representations, semantic misalignment between temporal tokens and contextual features may hinder effective fusion. Finally, it remains unclear whether time series forecasting can be effectively addressed through autoregressive generation over discrete tokens.

Based on the above analysis, we propose TokenCast, an LLM-driven framework for context-aware time series forecasting via symbolic discretization. TokenCast begins with a time series tokenizer that converts continuous sequences into discrete tokens, mitigating structural discrepancies across data modalities. To bridge the semantic gap, temporal and contextual tokens are jointly embedded into a shared representation space using a pre-trained LLM, optimized via a generative objective while keeping the backbone frozen and tuning only the embedding layer. Building on this unified semantic space, the aligned LLM is further fine-tuned with supervised forecasting signals to enhance predictive performance. We evaluate TokenCast on diverse real-world datasets enriched with contextual features. Experimental results show that TokenCast achieves strong accuracy and generalization across domains. We also conduct comprehensive ablation and qualitative studies, offering insights into the flexibility of symbolic, LLM-based time series forecasting.

2 RelatedWork

Time series forecasting (TSF) is a fundamental task across various domains. Traditional approaches typically rely on statistical assumptions such as stationarity and linearity, and often depend on handcrafted assumptions that limit their flexibility [Holt, 2004; Kalekar and others, 2004]. Alternatively, data-driven methods [Chen and Guestrin, 2016], particularly those based on deep learning, have advanced TSF by learning temporal patterns directly from data. RNN-based models [Wang et al., 2019] capture dependencies through recurrence, CNN-based models [Wang et al., 2023] enhance local pattern extraction, and Transformer-based architectures [Shi et al., 2025] are well-suited for modeling long-range interactions. Furthermore, MLP-based approaches [Wang et al., 2024] demonstrate that simple architectures can achieve competitive performance with improved computational efficiency. These models mainly focus on numerical data, with less emphasis on unstructured contextual features.

In addition to modeling temporal dependencies, recent research increasingly emphasizes the integration of contextual features for accurate forecasting [Chang et al., 2025; Hu et al., 2025]. Two major lines of research have emerged in this direction. One line of research focuses on deep learning architectures that model feature interactions [Gasthaus et al., 2019]. Another line of research leverages pre-trained LLMs for multimodal modeling [Cheng et al., 2025a; Cheng and others, 2025b]. Some approaches, such as TEMPO [Cao et al., 2024], utilize linear adapters to project temporal features into the LLM's semantic space. Others, like Promptcast [Xue and Salim, 2023], employ soft prompts to guide the frozen LLM's behavior. However, these promising approaches fail to bridge the structural gap between numerical and textual modalities [Jia et al., 2024].

3 The Proposed TokenCast

In this section, we present the formal problem definition, clarify the key concepts and notations used consistently throughout the paper, and provide an overview of the TokenCast.

3.1 ProblemFormulation

We consider a dataset D = {(Hi, Ti, Pi)}Ni=1 of N multimodal instances, where H ∈ RL×C denotes the historical multivariate time series, T denotes the contextual features, and P ∈ RLP×C denotes the future time series. The contextual features T are tokenized into language tokens Y by a pretrained LLM tokenizer, while H is mapped to discrete time series tokens Zq through a learnable function fθ : H ↦→ Zq. These tokens are concatenated as Z = [Zq; Y ] ∈ V T ′ . With boundary markers delimiting the generated temporal tokens ˆ Z, a decoding function gϕ : ˆ Z ↦→ ˆ P reconstructs the predicted time series ˆ P ∈ RLP×C .

3.2 Framework Overview

Figure 2 provides an overview of the TokenCast, which consists of three main stages. The process begins with the discretization, which transforms continuous sequences into a sequence of discrete tokens via a decoupled and dynamical vector quantization tokenizer. Subsequently, both the temporal and contextual tokens are then jointly processed by a pre-trained LLM, which performs cross-modality alignment under generative objectives. Following this alignment, the aligned LLM is adapted to the downstream task via generative fine-tuning. The predicted tokens are decoded to raw time series using a frozen de-tokenizer. The following sections elaborate on the principal stages of the TokenCast.

3.3 Time Series Discretization

Time Series Tokenizer. To fully harness the generative and reasoning capabilities of language models, symbolic representation naturally arises as an effective intermediary. Accordingly, we employ time series discretization as a simple yet powerful approach to establish this bridge. It is worth noting that existing approaches, such as Symbolic Aggregate Approximation (SAX) [Lin et al., 2007], have achieved progress in time series discretization but often suffer from significant information loss due to dimensionality reduction. In contrast, reconstruction-based methods map subsequences to discrete codes from a predefined codebook and achieve more precise representations through reconstruction optimization. While preserving the original information is advantageous, previous reconstruction-based methods typically encode the entire sequence, overlooking the statistical properties of time series. Reversible Instance Normalization (RevIN) [Kim et al., 2021] is widely used in forecasting, but its reliance on cached normalization statistics can lead to future information leakage when applied over prediction horizons. To mitigate this issue, we introduce a decoupled and dynamic tokenizer.

As illustrated in Figure 2 (a), similar to the forecasting phase, we divide the multivariate time series into a historical time series H ∈ RLH×C and a predicted time series P ∈ RLP×C , which can be formally represented as X = [H; P] ∈ RL×C . The process begins with a reversible instance normalization (RIN) layer. We compute the mean µ(H) and standard deviation σ(H) solely from the historical time series H, and apply them to normalize the time series X, thereby preventing future information leakage. These statistics are retained for inverse transformation during decoding. Instead of employing separate encoders, we adopt a shared encoder, which facilitates the joint modeling of both local and global information. The normalized time series is then passed through a causal encoder fenc, yielding a sequence of continuous latent representations Z = fenc(X) ∈ RT×d , where T is the number of latent vectors and d is the feature dimension. To discretize the latent representations, we apply a vector quantization layer. For domain i, a learnable codebook Ci = {ei,k}Kk=1 ⊂ Rd is maintained, containing K embedding vectors. Each latent vector zt ∈ Rd is mapped to its nearest neighbor in the codebook as zq t = ei,k∗, where k∗ = argmink ∥zt - ei,k∥22 . The output of this layer is a quantized sequence Zq = (zq 1 , . . . , z qT ), and the corresponding sequence of indices {k∗ } serves as the discrete tokens for downstream modeling. These tokens are subsequently decoded by a shared causal decoder fdec, rather than by separate decoders, which ensures consistent reconstruction and enables the predicted part to dynamically exploit richer historical features. Then, the final reconstruction ˆX is obtained by applying the inverse RIN operation using the stored statistics µ(H) and σ(H), i.e., ˆX = fdenorm(fdec(Zq)).

Training Objective. The tokenizer is optimized by minimizing the objective function defined as follows:

where Lrecon = ∥ ˆX - X∥22 is the reconstruction loss that optimizes both the encoder and decoder. Due to the nondifferentiability of the argmin operation in quantization, we employ the straight-through estimator (STE) during backpropagation. To train the vector quantizer, we include: Lcodebook = ∥sg[Z] - Zq∥22 , Lcommit = ∥Z - sg[Zq]∥22 , where sg[·] denotes the stop-gradient operator, which prevents gradients from flowing into its argument during backpropagation. To promote diverse usage of codebook entries, we further add a diversity loss Ldiversity = 1N ∑Ni=1 1 di+ϵ , where di = minj̸=i ∥ei - ej∥2 denotes the nearest-neighbor distance between codebook embeddings. This penalty discourages vectors from clustering too closely and encourages more uniform utilization of the codebook.

3.4 Pre-trained LLMBackbone Formulation

Following the discretization of time series into discrete tokens, the next challenge is to model the complex dependencies embedded in these sequences. While architectures like TCNs or Transformers can be trained from scratch, we argue that a pre-trained LLM serves as a more effective backbone. This is supported by two observations: (1) a pre-trained LLM possesses strong semantic understanding and contextual reasoning capabilities acquired from large-scale corpora, and (2) the structure of discrete time series tokens closely resembles that of language tokens [Zhao et al., 2023]. By casting forecasting as a generative task, we directly leverage the LLM's autoregressive generation ability. To guide LLM reasoning and incorporate contextual features, we employ a structured prompt template, as shown in Figure 2 (b). This prompt template consists of four essential components: domain knowledge, task instructions, statistical properties, and discrete time series tokens. This design ensures token-level consistency with language tokens and introduces task-specific descriptions alongside statistical attributes, enabling the LLM to perform instruction-driven generation.

3.5 Cross-Modality Alignment of Time Series and Contextual Features

While discretization aligns time series structurally with language tokens, a semantic gap remains between time series and contextual features. Existing methods often introduce projection modules (e.g., MLPs) to map time series into the LLM's latent space for fusion [Jia et al., 2024]. Although effective in downstream tasks, these strategies rely on external transformation modules for alignment, which bypass the language model's native vocabulary modeling mechanism. To this end, we implement a more explicit vocabulary-level alignment strategy. As illustrated in Figure 2 (b), we construct a unified vocabulary by directly appending K temporal tokens and S task-specific special tokens to the original vocabulary Vorig of the pre-trained LLM, forming an extended vocabulary V . Correspondingly, a shared embedding matrix E ∈ R|V |×d is used to encode all tokens, regardless of their modality origin. This unified embedding mechanism enables seamless fusion of time series and contextual features while maintaining alignment with the pretrained model. To ensure distributional alignment with pretrained embeddings for fine-tuning, the embedding of the newly introduced time series tokens is initialized by sampling from a multivariate gaussian distribution defined by the mean µ and covariance Σ of the original word embeddings. Then, temporal tokens Zq and contextual tokens Y are concatenated at the token level and jointly transformed into embeddings via the shared embedding layer: E([Zq, Y ]) = [E(z1), . . . , E(zn), E(y1), . . . , E(ym)], where E denotes the unified embedding matrix. This unified embedding process enables the LLM to reason over concatenated sequences without requiring architectural modification.

To optimize cross-modality token representations within the shared embedding space, we adopt an autoregressive training objective. Specifically, we freeze all parameters of the pre-trained LLM and update only the shared embedding matrix E, which is responsible for encoding both temporal and contextual tokens. Given a concatenated token sequence [Zq, Y ], the training objective is formulated as a next-token prediction task over the combined sequence:

where zt ∈ V denotes the t-th token in the sequence, and p(·) is the conditional probability predicted by the frozen language model given the embedding vectors from E.

3.6 Generative Fine-tuning and Context-aware Time Series Forecasting

We now detail the procedure for adapting the aligned LLM for forecasting tasks. As illustrated in Figure 2 (c), we employ a generative fine-tuning strategy to specialize the model for context-aware time series forecasting. This process consists of two primary stages: (1) structured prompt-based generative fine-tuning; and (2) context-aware time series forecasting with token-based decoding. In the first stage, promptbased generative fine-tuning is introduced to explicitly transfer the pretrained language modeling capability into the forecasting domain. Instead of relying on external mapping modules, generative fine-tuning directly formulates forecasting as a generation task, where the model is supervised to output both natural language reasoning and sequences of future time series tokens. This paradigm fosters a fast-thinking behavior: by optimizing an autoregressive objective against groundtruth structured responses, the model learns to rapidly recognize patterns, associate contextual features with temporal dynamics, and produce coherent outputs without engaging in deep deliberation. As a result, the aligned LLM acquires the ability to generate fluent and context-aware predictions. In the second stage, the fine-tuned model is utilized for contextaware forecasting and decoding. During inference, the model receives a prompt with historical data and contextual features, and autoregressively generates a complete response. The key component of this generated output is the sequence of discrete tokens, which represents the model's prediction of future time series values. To translate this symbolic representation back into a continuous predicted time series, these tokens are processed by a frozen time series de-tokenizer. We use boundary markers to delimit the temporal tokens.

4 Experiments

In this section, we conduct comprehensive experiments to evaluate our TokenCast's performance on diverse real-world datasets enriched with contextual features for time series forecasting. Additionally, we perform extensive ablation studies and exploration analysis to demonstrate the effectiveness of its individual components.

4.1 Experimental Settings

Datasets. As shown in Table 2, we evaluate our framework on six real-world datasets from diverse domains enriched with contextual features: Economic [McCracken and Ng, 2016], Health [Panagopoulos et al., 2021], Web [Gasthaus et al., 2019], two subsets of Stock data [Feng et al., 2019] and Nature [Poyatos et al., 2020]. These datasets, spanning various temporal patterns and contextual dependencies, serve as a comprehensive benchmark for context-aware forecasting. Data preparation involves imputing missing values and applying z-score normalization to all datasets, ensuring stable convergence and comparability.

Baselines. We compare our proposed framework against eight strong baselines, grouped into four representative categories for comprehensive evaluation. For LLM-based models, we include Time-LLM [Jin et al., 2024] and GPT4TS [Zhou et al., 2023], which adapt pre-trained LLMs for time series forecasting using modality-aware prompting and reprogramming. In the self-supervised frameworks category, we evaluate TimeDART [Wang et al., 2025] and SimMTM [Dong et al., 2023]. Additionally, we include Transformerbased methods like Autoformer [Wu et al., 2021] and Crossformer [Zhang and Yan, 2023]. Finally, we consider the MLP-based method DLinear [Zeng et al., 2023]. Further details are provided in the Appendix.

ImplementationDetails. For each baseline, we search over multiple input lengths and report the best performance to avoid underestimating its capability. The historical length is set to L = 96 for the Nature dataset and L = 36 for the other five datasets, based on the data volume and temporal resolution. The forecasting horizons are set to {24, 48, 96, 192} for Nature and {24, 36, 48, 60} for the other dataset. We adopt two widely used evaluation metrics in time series forecasting: mean absolute error (MAE) and mean squared error (MSE). We report average results for the main and ablation studies. For exploratory analysis, we use 96-to-24 on Nature and 36-to-24 on the other datasets. Complete results for the main experiments, ablation studies, and exploratory analysis are included in the Appendix. All experiments are implemented in PyTorch and conducted on a distributed setup with 8 NVIDIA A100 GPUs.

4.2 Forecasting Performance Analysis

Table 1 comprehensively compares forecasting performance across six benchmark datasets. TokenCast demonstrates superior performance in most scenarios, further confirming previous empirical findings [Zhou et al., 2023] that no single model performs best across all settings. Notably, LLMbased baselines like Time-LLM also show competitive results, particularly on context-rich datasets such as Economic and Stock-NY. This further validates the potential of leveraging large language models in time series forecasting. However, these models often lack the structural alignment mechanisms introduced by our framework, limiting their consistent performance. Conventional baselines such as TimeDART perform well on datasets with strong periodicity and weak contextual dependence (e.g., Nature), but their performance drops significantly on complex datasets rich in contextual features (e.g., Economic and Web). This contrast underscores the importance of contextual feature modeling and crossmodal interaction. In summary, our framework delivers stateof-the-art results with high consistency. This is attributed to its core design: discretizing time series into discrete tokens and aligning them with contextual features. This unified token-based paradigm effectively addresses real-world context-aware time series forecasting challenges.

4.3 Ablation Studies

Ablation on Alignment and Fine-tuning. We conduct the ablation study on two crucial training steps: the crossmodality alignment and generative fine-tuning. The comprehensive results in Table 3 clearly demonstrate their indispensable contribution to the overall framework performance. The model equipped with the cross-modality alignment stage consistently achieves lower MSE scores across all six datasets. Without this alignment, contextual features risk being misinterpreted by the time series backbone, leading to suboptimal forecasts. This highlights its critical role in effectively integrating contextual information by bridging structural and semantic discrepancies between time series and contextual features, thus facilitating meaningful feature interaction. This alignment thus acts as a foundational step, ensuring the subsequent fine-tuning stage operates on a semantically rich and coherent feature space.

Concurrently, Table 3 vividly illustrates the pivotal contribution of the generative fine-tuning stage. Across all six benchmark datasets, the model employing generative finetuning consistently and substantially outperforms its counterpart that omits this crucial step. The performance degradation when omitting this stage is notable across various datasets, underscoring the general applicability and importance of the fine-tuning process. This drop is particularly stark on datasets like Stock-NA, where the complex, non-stationary patterns demand task-specific adaptation. Ultimately, these findings emphasize that generative fine-tuning is essential for adapting the pre-trained LLM's general capabilities to generative time series forecasting.

Ablation on Multimodal Contributions. Fig. 3 analyzes how different types of contextual features affect forecasting performance. Incorporating any contextual features yields substantial improvements, as the variant without contextual input consistently performs worse. We further divide the input into general info, which provides high-level context such as domain knowledge and task instructions, and local info, which offers event-specific details like static statistical attributes. While both types contribute to performance, general info typically brings larger improvements. The strong results of the full model indicate its ability to effectively combine the broad context from general info and the specific cues from local info to enhance prediction accuracy.

4.4 Exploration Analysis

Codebook Size. We conduct a study to assess the impact of codebook size on model performance, as summarized in Table 4. The results highlight the importance of selecting an appropriate codebook size for time series forecasting. Specifically, a size of 128 achieves state-of-the-art results on the Nature and Stock-NA datasets, while a smaller size of 64 excels on the Economic dataset. Interestingly, both smaller (32) and larger (256) codebook sizes fail to produce better results and often lead to significant performance degradation. This suggests that for our framework, simply increasing token granularity is not always beneficial. Instead, a moderate codebook size strikes a balance between reconstruction fidelity and the complexity of the downstream task.

Generative Uncertainty. To validate the uncertainty modeling capabilities of our TokenCast, we conduct experiments on both the Economic and Stock-NY datasets. As shown in Fig. 4, our method produces predictive distributions that closely track the ground truth, with 50% and 80% prediction intervals capturing the inherent variability in the data. By adjusting the temperature during sampling, we observe that the model can flexibly modulate the spread of the predictive intervals, indicating its potential for controllable uncertaintyaware forecasting. This demonstrates that our model not only provides accurate mean predictions but also yields wellcalibrated uncertainty estimates.

LLM Backbone. We evaluate four LLM backbones to identify the optimal architecture for our forecasting framework. As summarized in Table 5, the Qwen2.5-0.5Bbase models consistently demonstrate superior performance. Specifically, the base version achieves state-of-the-art results on the Nature and Stock-NA datasets, while the instruct-tuned version excels on the more complex Economic dataset. Interestingly, larger models like Qwen2.5-1.5B-inst. fail to yield further gains and often underperform. This suggests that for our tasks, simply scaling up model size is not beneficial. Instead, the 0.5B models strike a balance between representational capacity and generalization.

Embedding Layer Initialization. We investigate three initialization strategies for our model's embedding layer to identify the most effective approach. As shown in Table 6, mean initialization consistently provides the most consistent performance. Specifically, it achieves the best results on the Nature and Economic datasets. While word initialization is superior on the Stock-NA dataset, its performance is less consistent across other domains. Notably, standard random initialization suffers a significant performance degradation on Stock-NA, highlighting its instability. These findings suggest that initializing embeddings with meaningful prior information provides a better starting point for optimization. Therefore, we adopt mean initialization as the default.

Qualitative Analysis of Tokenization. As shown in Figures 5 and 6, we conduct a comprehensive evaluation of the proposed discretization module on the Nature dataset from three complementary perspectives. The token usage heatmap (Figure 5) indicates that all 64 tokens are actively and consistently utilized across samples, demonstrating effective miti- Token Usage Heatmap gation of codebook collapse and a strong capacity to capture diverse and heterogeneous temporal structures. The codebook clustering visualization (Figure 6, left) further reveals that tokens organize into coherent and well-separated groups in the latent space, suggesting that the learned discrete vocabulary preserves meaningful structural relationships among different temporal patterns. Moreover, the dynamic reconstruction results (Figure 6, right) highlight the tokenizer's context-adaptive decoding behavior: the same token id (e.g., ID = 18) can generate distinct decoded segments under different contextual conditions, thereby enabling faithful and flexible alignment with the original time series. Overall, these empirical findings confirm that the proposed discretization process learns a diverse, semantically organized, and structurally meaningful vocabulary, while effectively supporting contextaware and adaptive decoding for time series forecasting.

Figure 6: Illustration of codebook clustering in the latent space and dynamic reconstruction.

5 Conclusion

We proposed TokenCast, a context-aware TSF framework based on a pretrained LLM. This approach first converts a continuous time series into discrete tokens. Leveraging a pretrained LLM, it aligns the temporal and contextual tokens through an autoregressive objective, achieving unified modeling of both modalities. The model is then further finetuned to generate future token sequences. We evaluate TokenCast on multiple real-world datasets rich in contextual information. Experimental results demonstrate that TokenCast achieves superior accuracy. We also conduct comprehensive ablation experiments and qualitative analysis to validate the framework's adaptability and flexibility for symbolic, LLMdriven TSF. Looking ahead, we believe that leveraging language as a symbolic intermediary will have the potential to advance TSF towards a multimodal and multi-task level.

Acknowledgments

This research was supported by grants from the National Natural Science Foundation of China (No. 62502486, 62376193), the grants of the Provincial Natural Science Foundation of Anhui Province (No. 2408085QF193), Guangdong S&T Programme (No. 2025B0101120004), USTC Research Funds of the Double First-Class Initiative (No. YD2150002501), the Fundamental Research Funds for the Central Universities of China (No. WK2150110032).