LeRet: Language-Empowered Retentive Network for Time Series Forecasting

1 Introduction

Time series forecasting (TSF) is fundamental in many realworld applications [Zhou et al., 2020b; Wang et al., 2021a; Zhang et al., 2022; Wang et al., 2024; Wang et al., 2023d], including weather forecasting [Wu et al., 2021], traffic planning [Tian et al., 2021; Wang et al., 2023b; Zhou et al., 2020a; Zhang et al., 2024] and electricity scheduling [Zhou et al., 2022]. In the past decade, various deep learning models have been applied to TSF, such as convolutional neural networks (CNN) [Luo and Wang, 2024; Wang et al., 2023c; Wu et al., 2023], recurrent neural networks (RNN) [Lin et al., 2023b], graph neural networks (GNN) [Wang et al., 2023a; Huang et al., 2024b; Zhao et al., 2024a; Zhou et al., 2023b; Yang et al., 2023; Zhao et al., 2024b], and Transformers [Zhou et al., 2021; Nie et al., 2023; Zhang and Yan, 2023; Shabani et al., 2023; Liu et al., 2021; Lin et al., 2023a], achieving excellent predictive performance. Despite the excellent performance, these task-specific models are confined to single time series modality, concentrating on the modeling of either intra-sequence dependence [Wu et al., 2021; Zhou et al., 2022], or inter-sequence dependence [Zhang and Yan, 2023; Huang et al., 2024a]. At a cost of focusing on time series modeling, the task-specific models are only valid for data with limited size and thus lacking generalization, posing challenges to achieve few-shot or zero-shot forecasting.

∗Yang Wang and Zhengyang Zhou are corresponding authors.

To gain the generalization capacity and general crossdomain forecasting, numerous Large Language Models (LLM) based TSF methods [Zhou et al., 2023a; Cao et al., 2023; Gruver et al., 2023; Jin et al., 2023; Chang et al., 2023; Yu et al., 2023] have emerged. These models fine-tune pretrained LLM (e.g., GPT [Radford et al., 2018], LLaMa [Touvron et al., 2023]) to embed extensive language knowledge into time series, transferring pre-trained domain knowledge to temporal data. Generally, the success of LLM-based methods stem from two aspects, i) Language knowledge empowering [Zhou et al., 2023a]. LLM is provided with abundant language knowledge, including a nuanced understanding of time series, notably enhancing the model to recognize sparse and complex temporal data. With the well understanding of series patterns, many studies [Gruver et al., 2023; Yu et al., 2023] further reveal the exceptional capability of LLM in few-shot and even zero-shot forecasting scenes. ii) Causal dependence learning [Chang et al., 2023]. In contrast to existing bidirectional attention models [Nie et al., 2023; Zhang and Yan, 2023; Shabani et al., 2023; Liu et al., 2021; Lin et al., 2023a], LLM is based on causal attention where the hidden state of a token is only related to itself and preceding tokens, following inherent sequential causality in time series. Such encoding strategy enhances the comprehension of the temporal evolution process and has recently been effectively exploited in following studies [Lin et al., 2023b].

While fine-tuning LLM for TSF leverages the crossdomain transferability fromNLP pre-training, it suffers from two serious issues from the practicality and interpretability perspectives. Regarding practicality, although a large number of parameters can enhance the fitting capacity, directly optimizing such large network, even with techniques like LoRA [Hu et al., 2022] and SoRA [Ding et al., 2023], is resource-intensive. Besides, such large size of parameters hinders the feasibility for lightweight applications at the edge devices, limiting the accommodation of LLM-based methods to real-world forecasting scenarios. Regarding interpretability, LLM is pre-trained on discrete token-based text, while time series data accounts for numerical continuous data, resulting in a significant gap between natural language and numerics in representation space. This gap poses difficulties for LLM in achieving interpretable forecasting.

Hence, efficiently and effectively leveraging time series related knowledge from LLM to enhance the generalization and forecastability of TSF remains challenging. Fortunately, the success of LLM has prompted us to simultaneously consider the superiority of model structure design, the inherent sequential causality in time series modeling, and the task-oriented advantage, the excellent knowledge representation of natural languages with the capacity of series-level pattern transfer.

In this work, we propose a LeRet, a Language-empowered Retentive network for Time Series Forecasting. The differences between our LeRet and other TSF solutions are shown in Figure 1. In terms of causal dependence, we introduce a Causal Dependence Learner for time series, emphasizing the informativeness of historical information to capture causalrelated temporal features. We further design a patch-level autoregressive forecasting task with a pre-training objective for the learner, to enhance the nuanced understanding of the causal evolution of series. For language knowledge, to efficiently utilize LLM, we propose a TS-Related Language Knowledge Extractor to extract general time series characteristics from pre-trained LLM. With Language-TS Modality Integrator, we map text embedding to time series feature space, then empower time series feature by actively receiving extracted language knowledge, and project the languageempowered representation to sequence-level forecasting. Our comprehensive evaluations demonstrate that LeRet is a robust time series learner that outperforms state-of-the-art forecasting models. As to generalization, with language model empowered, LeRet also excels in both few-shot and zero-shot learning scenarios. The main contributions of this work can be summarized as follows,

• We comprehensively dissect the advantages and limitations of LLM-based TSF methods, highlighting the strengths of model structure in sequential causality extraction and the language-oriented knowledge representation, and pointing out its shortcomings on the aspects of practicality and interpretability.
• We propose a novel framework that leverages the strengths of both LLM-based methods and task-specific models. Our LeRet especially gains the zero-shot and few-shot learning ability, from pre-trained language knowledge and our design of modality alignment.
• LeRet demonstrates outstanding predictive performance across various TSF tasks, including long-term, shortterm, few-shot, and zero-shot forecasting. Quantitatively, LeRet outperforms 8 state-of-the-art models for long-term forecasting, achieving top-1 performance in 57 settings and top-2 in 5 settings out of a total of 64 settings.

2 RelatedWork

2.1 Task-sepcific ForecastingMethods

Benefiting from the advancements in deep learning, various models, including CNN-, GNN-, RNN-, and Transformerbased architectures, have been designed for task-specific forecasting [Liu et al., 2021; Miao et al., 2024; Wang et al., 2021b; Lin et al., 2024]. Notably, Transformer-based models have gained widespread acknowledgment for their global modeling capacity, allowing capturing long-term temporal dependencies through self-attention. Autoformer [Wu et al., 2021] presents a series decomposition block based on a moving average to decompose complex temporal data into seasonal and trend components. FEDformer [Zhou et al., 2022] leverages a frequency enhanced decomposition mechanism to obtain more efficient forecasting. Crossformer [Zhang and Yan, 2023] advocates for not only focusing on temporal dependencies but also considering relationships among variables. It introduces a routing mechanism to efficiently model cross-variable dependencies. PatchTST [Nie et al., 2023] introduces patching and channel independence for TSF, reducing model complexity while dramatically enhancing forecasting performance. However, these models focus solely on a single time series modality, lacking strong predictive generalization. Additionally, their bidirectional attention mechanisms overlook the causal evolution inherent in time series. In response to this, in LeRet, we refine such causal features through the Causal Dependence Learner and extract language knowledge by TS-Related Language Knowledge Extractor to expand forecasting generalization.

2.2 LLM-based ForecastingMethods

The recent emergence of Large Language Models (LLMs) has introduced new possibilities for time series modeling, leading to a growing interest in the application of LLM to Time Series Forecasting (TSF). GPT4TS [Zhou et al., 2023a] utilizs pre-trained language model without updating its selfattention and feedforward layers. The model undergoes finetuning and evaluation across various time series analysis tasks, demonstrating comparable or state-of-the-art performance by leveraging knowledge transfer from natural language pre-training. LLM4TS [Chang et al., 2023] adopts a two-stage fine-tuning approach on the LLM to fully leverage time series data. Tempo [Cao et al., 2023] decomposes the trend, seasonality, and residual components of time series, and dynamically selects prompts to ease the comprehension challenges for LLM. However, these LLM-based methods directly feed time series data into language pre-trained LLM, lacking interpretability. Additionally, the large parameter sizes limit their application scenarios. Therefore, we design the TS-Related Language Knowledge Extractor to extract lightweight language knowledge from the extensive LLM, and Language-TS Modality Integrator for multimodality fusion, significantly improving the interpretability and practicality of LLM-based models.

3 Method

3.1 ProblemDefinition

The objective of time series forecasting (TSF) task is to predict the future values based on historical observations. Given historical L-steps input Xinput = [x1, x2, ..., xL] ∈ RL. We aim to learn a function F(·) to accurately forecast ˆX, where ˆX = [ˆxL+1, ˆxL+2, . . . , ˆxL+T ] ∈ RT represents predicted T - steps future values. The optimization objective is to minimize the discrepancy between predicted values and actual future values over time.

3.2 Overall Architecture

The overall framework of LeRet is illustrated in Figure 2. Initially, a multivariate time series is decomposed into multiple univariate time series, which are then treated independently [Nie et al., 2023]. This transforms the task of multivariate time series forecasting into a set of univariate time series forecasting tasks.

Subsequently, for a time series Xinput ∈ RL, We partition it into non-overlapping patches of length P, resulting in a total of N = ⌊ L P ⌋ + 1 input patches Xpatch ∈ RN×P . These patches are embedded as Xpe ∈ RN×dp using a simple linear layer:

Based on Xpe, we firsly apply Retentive Network (RetNet) [Sun et al., ] as a Causal Dependence Learner (CDL) to encode features of the patched time series, remaining temporal causal nature and obtaining features H ∈ RN×dm . It can be given by:H = CausalLearner(Xpe). (2)

In the causal dependence representation, previous patch features is independent of next patch. Thus, conducting a patchlevel autoregressive task as pre-training helps the model understand causal growth patterns in the time series by predicting the next patch's values based on the preceding patch.

Moreover, in TS-Related Language Knowledge Extractor, we input K text descriptions of fundamental time series features TDs, such as trend, period, stability, and noise level, into a pre-trained Large Language Model (LLM) to extract language knowledge related to the time series, denoted as:

where δ = {S1, S2, ..., SK} ∈ RK×ds is the extracted time series related text embedding and K is the number of texts.

Subsequently, through the Language-TS Modality Integrator (LTMI), text embedding are firstly mapped to the time series space, and the fusion of language knowledge and time series produces language-empowered sequential features Z ∈ RN×dm :

LeRet employs sequence forecasting with patch-level enhanced. (a) Patch-level Pre-training. LeRet predict the temporal values of the next patch based on the causal temporal features H ∈ RN×dm to pre-train Causal Dependent Learner; (b) Sequence-level Forecasting. The language-empowered sequential features Z ∈ RN×dm is finally used to predict T future time steps.

3.3 Causal Dependence Learner

To preserve the inherent causal dependence of time series and improve computational efficiency, LeRet utilizes the Retentive Network (RetNet) [Sun et al., ] as the backbone of Causal Dependence Learner.

Retention Mechanism The input of model is Xpe = {x1pe , x2pe , ..., xNpe }. Without employing bidirectional attention, RetNet utilizes a retention mechanism for sequence modeling. We use qn = Wqxnpe ∈ , km = Wkxmpe and vm = Wvxmpe as query, key and value of corresponding patch embedding, respectively. Denote on as the output feature of xnpe by rentention metchanism. It can be expressed as:

where qn (γeiθ )n , km (γeiθ )-m is known as xPos [Sun et al., 2022], i.e., a relative position embedding proposed for Transformer. We further simplify γ as a pr-defined scalar to formulate Eq(5) becomes:

where γ serves as a pre-defined decay factor, replacing the initial calculation of the attention map, and † denotes the conjugate transpose. The formulation is easily parallelizable within training instances and can be represented as:

where ¯Θ is the complex conjugate of Θ, and D ∈ RN×N combines causal masking and pre-defined exponential decay. Similar to self-attention, the parallel representation enables us to train the models with GPUs efficiently.

Gated Multi-scale Retention The model employs h = dm/d retention heads in each layer, where d is the head dimension. Multi-scale retention (MSR) assigns different γ for each head, adding a swish gate to increase non-linearity. The layer of Multi-scale Retention (MSR) is defined as:

Here, WG,WO ∈ Rdm×dm are learnable parameters. GN, Concat and SG are group normalization, concatenation and swish gate, respectively.

Causal Dependence Representation The forward propagation process of CDL is illustrated in Figure 3(a), which includes the MSR layer, feed forward layer and residual operation to enhance the fitting ability for extracting time series features. The entire forward process is expressed as,

where LN and FF are normalization layer and feed forward layer. H ∈ RN×dm is the final output of CDL, capturing temporal causal dependencies, where each subsequent time series patch can only attend to the ones preceding it.

3.4 TS-Related Language Knowledge Extractor

The extensive language knowledge in LLM is redundant, and it is difficult to directly form targeted knowledge related to time series based on LLM. Additionally, using LLM during as a part of forecasting consumes significant computational and storage resources in both training and inference phases. Therefore, we propose TS-Related Language Knowledge Extractor, which efficiently extracts language knowledge about time series from LLM.

Language Description Protocol for Time Series As shown in Table 1, we select four significant time series characteristics (i.e., trend, period, stability, and noise level) and form several text descriptions based on these features. Formally, we represent a collection of K text descriptions about time series characteristics as TD = {td1, td2, ..., tdK}, where tdi(1 ≤ i ≤ K) is an independent text with a length of θi.

Series Knowledge Representation from Natural Language For the description tdi, we first input it into the LLM for feature encoding, obtaining the text embedding tei ∈ R(θi+2)×ds , where ds is the feature dimension of each token, and θi + 2 is the number of tokens (adding start token [BOS] and end token [EOS] to the original segmentation). Since we choose LLaMa as the LLM, which is a decoderonly architecture, under this causal encoding, each token can only perceive itself and the tokens before it. As only the last state can store all the information of the sentence, we select the embedding of the [EOS] token of each text embedding as the extracted time series related language knowledge, denoted as δ = {S1, S2, ..., SK} ∈ RK×ds .

3.5 Language-TSModality Integrator

As language knowledge γ and time series features H belong to two different and distinct feature spaces, directly feeding language knowledge into the time series forecasting model is not feasible. This would make it challenging for the model to understand the originally captured temporal patterns, increasing the difficulty in fitting forecasting task. In the LanguageTS Modality Integrator, we design a two-stage modality fusion to achieve language knowledge empowered time series as shown in Figure 3(b).

In the first stage, we need to map language knowledge to the time series feature space. An easy-to-implement method is cross-attention, allowing language knowledge to adaptively aggregate time series features, forming language knowledge expressed in the time series feature:

where Qδ = δ · W Q δ , KH = H · W K H , VH = H · W VH , are language and time series features linearly mapped, and δ′ ∈ RK×dm is the mapping of language features into the time series feature space.

In the second stage, we need to integrate this aligned language knowledge δ′ with time series features H:

where QH = δ ·W H δ , Kδ′ = δ′ ·W K δ′ , Vδ′ = δ′ ·W V δ′ , are language and time series features linearly mapped, Z ∈ RK×dm represents the language-empowered time series features.

3.6 Sequence Forecasting with Patch-level Enhancement

Patch-Level Pre-training To enhance the model in understanding the causal evolution of time series, we devise a patch-level forecasting as a pre-training task to warm up Causal Dependence Learner. For example, given an input sequence of patches such as the 1st patch, 2nd patch, 3rd patch, this task is expected to generate outputs corresponding to the 2nd patch, 3rd patch, 4th patch based on the preceding patch. Since the language-empowered Z obtained after modality fusion may disrupt the causality of time series features, we use causal temporal features H to make patch-level forecasting :

where Ypatch ∈ RN×P represents the patch-level predicted time series values, and HeadP is a linear layer as patch-level pre-training head.

Sequence-Level Forecasting For sequence-level forecasting, we use language-empowered time series features Z to make predictions:

where Yseq ∈ RT is the forecasting results for the future L steps, and HeadS is the prediction head consisting of a reshape block and a linear layer.

4 Experiments

4.1 Datasets and Experimental Setups

We evaluate performance of long-term forecasting on Weather, Traffic, Solar, Electricity and four ETT datasets (i.e., ETTh1, ETTh2, ETTm1, and ETTm2), which have been extensively adopted for benchmarking long-term forecasting models. The input time series length L is set as 336 for all baselines, and we use four different prediction horizons T ∈ {96, 192, 336, 720}. For short-term forecasting, we adopt the PeMS which contains four public traffic network datasets (PEMS03, PEMS04, PEMS07, PEMS08). All the models are following the same experimental setup with input length L = 96 and prediction length T = 12.

4.2 Main Results

Long-term Forecasting

Our results are presented in Table 2, where LeRet demonstrates superior performance across different prediction length again all baselines. Quantitatively, LeRet achieves 57 first-place and 5 second-place rankings out of 64 settings. In contrast to the effective linear model DLinear, LeRet achieves performance gains of 21.3% and 18.3% in MSE and MAE metrics. Compared to the state-of-the-art task-specific TSF model ModernTCN, LeRet exhibits a relative reduction of 7.6% and 5.6% in MSE and MAE metrics, respectively. When compared with the cutting-edge LLM-based TSF model LLM4TS, LeRet exhibits superiority in 59 out of 64 experimental settings, with a performance improvement of 7.1% and 5.0% in MSE and MAE metrics, respectively.

Short-term Forecasting

As illustrated in Table 3, for short-term forecasting, LeRet consistently maintains a leading predictive performance. Compared to SOTA short-term forecasting model SCINet, LeRet achieves significant reductions in MAE, MAPE, and RMSE, respectively. The comprehensive experimental results underscore the efficacy of LeRet in short-term forecasting.

Few-shot Learning

In few-shot learning, only 10% of the training data timesteps are utilized, and the outcomes are presented in Table 5. Evidently, LLM-based methods outperform other benchmark TSF models. Quantitativly, LeRet achieves an average 4.4% reduction in MSE and 2.1% reduction in MAE compared to the top-performing LLM4TS.

Zero-shot Learning

This task is to evaluate how effectively a model can perform on target dataset when it has been trained on source dataset, and the results are presented in Table 4. LeRet outperforms all state-of-the-art models, achieving a performance improvement of over 10% compared to other models.

4.3 Ablation Study

To assess the effectiveness of each component in LeRet, we conduct a comprehensive ablation study on Causal Dependence Learner (CDL), Time-Language Modality Integrator (TLMI), Patch-level Pre-training (PLP) and TS-Related Language Knowledge Extractor (TRLE). In the corresponding ablations, CDL is replaced with self-attention, TLMI is substituted with a simple concatenation operation, PLP is removed, and excluding TRLE prevents the model from receiving knowledge from the language modality. Our observations from Table 6 are as follows: Obs.1) The combination of CDL and PLP is necessary. Removing these two modules results in approximately a 8.6% decrease in performance; Obs.2) Integrating TRLE with TLMI contributes to the model's understanding of temporal features, resulting in an average performance improvement of 5.7%.

5 Discussion

Compared to other LLM-based TSF models, LeRet exhibits significant advantages in interpretability and parameter scales. Actually, we focus on ensuring the interpretability of using LLM (i.e., the interpretability of the model input). Usually, LLM-based TSF models directly fine-tune pre-trained language models with time series as inputs, achieving good predictive performance but lacking explanation for why discrete texts can be arbitrarily replaced by continuous numerical values. In contrast, LeRet enhances the interpretability of LLM utilization by firstly mapping text embeddings to the time series feature space, and then integrating the aligned text features with time series features for language-empowered forecasting. In terms of parameter scales, compared to other LLM-based models like LLM4TS with 7B parameters per forward pass, our LeRet requires only 0.15M parameters during both training and inference. This reduction is achieved by utilizing pre-trained LLM solely for ts-related text embeddings. TS-related text embeddings can be stored in memory for quick retrieval, without involving LLM in training, significantly reducing the parameter scales by several thousand times.

6 Conclusion

We propose a language-empowered time-series learning framework, LeRet, which inherits the structure of sequential causality extraction from LLMs and exploits pre-trained language knowledge for effective and semantic interpretable series forecasting. We introduce a novel paradigm for LLMbased methods. Empirical evaluations reveal the effectiveness of our LeRet, especially show superiority in few- and zero-shot scenarios.

Acknowledgements

This paper is partially supported by the National Natural Science Foundation of China (No.62072427, No.12227901), the Project of Stable Support for Youth Team in Basic Research Field, CAS (No.YSBR-005), Academic Leaders Cultivation Program, USTC, and the grant from State Key Laboratory of Resources and Environmental Information System.