RePST: Language Model Empowered Spatio-Temporal Forecasting via Semantic-Oriented Reprogramming

1 Introduction

Spatio-temporal forecasting aims to predict future states of real-world complex systems by simultaneously learning spatial and temporal dependencies of historical observations, which plays a pivotal role in diverse real-world applications, such as traffic management [Li et al., 2018; Wu et al., 2019], environmental monitoring [Han et al., 2023], and resource optimization [Geng et al., 2019]. In the past decade, deep learning has demonstrated great predictive power and led to a surge in deep spatio-temporal forecasting models [Jin et al., 2023a]. For example, Recurrent Neural Networks (RNNs) and Graph Neural Networks (GNNs) are frequently combined to capture complex patterns for spatio-temporal forecasting [Li et al., 2018; Han et al., 2020]. Despite fruitful progress made so far, such approaches are typically confined to the one-task-one-model setting, which lacks general-purpose utility and inevitably falls short in handling widespread data-scarcity issue in real-world scenarios, e.g., newly deployed monitoring services.

In recent years, PLMs like GPT-3 [Brown, 2020] and the LLaMA family [Touvron et al., 2023] have achieved groundbreaking success in the Natural Language Processing (NLP) domain. PLMs exhibit exceptional contextual understanding, reasoning, and few-shot generalization capabilities across a wide range of tasks due to their pre-training on extensive text corpora. Although originally designed for textual data, the versatility and power of PLMs have inspired their application to numerically correlated data [Zhou et al., 2024; Jin et al., 2023b; Jin et al., 2024]. For example, [Zhou et al., 2024] pioneers research in this direction and showcases the promise of fine-tuning PLMs as generic time series feature extractors. Besides, model reprogramming [Jin et al., 2023b] has considered the modality differences between time series and natural language, solving time series forecasting tasks by learning an input transformation function that maps time series to a compressed vocabulary.

However, two significant challenges remain in directly applying the aforementioned reprogramming techniques to spatio-temporal forecasting. The foremost issue lies in the underutilization of PLMs' full potential. Recent work [Tan et al., 2024] suggests that existing PLM-based approaches for time series forecasting fail to leverage the generative and reasoning abilities of PLMs. This limitation becomes even more apparent when handling more complex spatially correlated time series data. To this end, a crucial question arises: how can we better explain this shortcoming and unlock the potential of PLMs for spatio-temporal forecasting? Another challenge is PLMs' limited capacity to model the intricate correlations present in spatio-temporal data. While PLMs excel at capturing dependencies within one-dimensional sequential data, they fall short in comprehending spatio-temporal data, which often has more complex structures like grids or graphs [Li et al., 2024]. This gap poses a significant obstacle to PLMs' effective use in this domain.

To address these challenges, we argue that the primary limitation of existing approaches lies in their oversimplified treatment of spatio-temporal data, which prevents PLMs from fully understanding the underlying semantics. Instead of merely serving as a one-dimensional encoder, PLMs need a more sophisticated understanding to handle spatio-temporal data effectively. As depicted in Figure 1, our explorative experiments uncover that even applying simple decomposition techniques can significantly facilitate PLMs to better understand spatiotemporal data and lead to improved performance.

Building on this insight, we propose REPST, a reprogramming framework specifically designed for spatio-temporal forecasting using PLMs. Specifically, we first propose a semantic-oriented spatio-temporal decomposer, which adaptively disentangles spatio-temporal dynamics into components that represent interpretable sub-processes within the system. This is achieved through a Koopman theory-based evolutionary matrix, which results in decomposed components rich in spatio-temporal semantics that PLMs can more easily comprehend. This decomposition-based approach enables PLMs to capture both spatial and temporal dynamics more effectively. Moreover, we introduce a selective reprogramming strategy to tackle the complexity of spatio-temporal structures, which differ fundamentally from the one-dimensional sequence-like structure of textual data. Our strategy constructs an expanded spatio-temporal vocabulary by selecting the most relevant spatio-temporal word tokens from the PLM's vocabulary through a differentiable reparameterization process. Unlike previous works that use compressed vocabularies, which can lead to ambiguous semantics, our approach reconstructs the reprogramming space with a rich, semantically distinct spatio-temporal vocabulary. By leveraging pretrained spatiotemporal correlations, this strategy enables PLMs to focus on relationships among tokens in a 3D spatio-temporal space, significantly enhancing their ability to model complex spatiotemporal dynamics. We evaluate REPST on a variety of spatiotemporal forecasting tasks, including energy management, air quality prediction, and traffic forecasting. Extensive experimental results highlight the framework's superior performance compared to state-of-the-art models, particularly in few-shot and zero-shot learning contexts. Our main contributions are summarized as:

• We identify the underlying reason for the underperformance of existing PLM-based approaches for spatio-temporal forecasting, highlighting the need to decompose spatio-temporal dynamics into interpretable components to fully leverage PLMs' potential.
• We propose REPST, a spatio-temporal forecasting framework that enables PLMs to grasp complex spatio-temporal patterns via semantic-oriented decomposition-based reprogramming. The reprogramming module reconstructs an expanded spatio-temporal vocabulary using a selective strategy, allowing PLMs to model spatio-temporal dynamics without altering their pre-trained parameters.
• We show that REPST consistently achieves superior performance across real-world datasets, particularly in data-scarce settings, demonstrating strong generalization capabilities in few-shot and zero-shot learning scenarios.

2 Preliminaries

Spatio-temporal data can be considered as observations of the state of a dynamical system. It is typically represented as a two-dimensional matrix X ∈ RN×T , which captures the states of a set of N nodes V, where each node in V corresponds to an entity (e.g., grids, regions, and sensors) in space. Specifically, we denote xit-T+1:t = [xit-T+1 , xit-T , ..., xit ]⊤ ∈ RT×1 as the observations of node i from time step t - T + 1 to t, where T represents the look-back window length. The goal of spatio-temporal forecasting problem is to predict future states for all nodes i ∈ V over the next τ time steps based on a sequence of historical observations. This involves uncovering the complex spatial and temporal patterns inherent in spatiotemporal data to reveal the hidden principles governing the system's dynamics:

where Xt-T+1:t = [x0t-T+1:t , x1t-T+1:t , ..., xN-1 t-T+1:t ]⊤ ∈ RN×T denotes the historical observations in previous T time steps, and fθ(·) is the spatio-temporal forecasting model parameterized by θ. Yˆ t+1:t+τ = {y ˆit+1:t+τ }Ni=0 and Yt+1:t+τ = {yit+1:t+τ }Ni=0 denote the estimated future states and the ground truth in the next τ time steps, where Yˆ t+1:t+τ ,Yt+1:t+τ ∈ RN×τ . For convenience, we omit the lower corner mark and represent Xt-T+1:t, Yt+1:t+τ as X, Y and xit-T+1:t , yit+1:t+τ as xi , yi .

(1) Overall Framework

(2) Spatio-Temporal Decomposer

3 Methodology

As illustrated in Figure 2, REPST consists of three components: a semantic-oriented spatio-temporal decomposer, a selective reprogrammed language model, and a learnable mapping function.

3.1 Semantic-Oriented Evolutional Spatio-Temporal Decomposition

Recent studies have revealed that PLMs possess rich spatiotemporal knowledge and reasoning capabilities [Gurnee and Tegmark, 2023; Jin et al., 2024]. However, existing methods failed to fully leverage the capabilities of PLMs, which raises challenges for spatio-temporal data forecasting as well. As aforementioned, reasons for this shortcoming lies in their over simplistic encoding to time series. PLMs requires further process of spatio-temporal data to enhance their comprehensibility to such complex structure. In this section, we address this shortcoming through a carefully designed semanticoriented spatio-temporal decomposer. Previous works [Liu et al., 2024c; Yi et al., 2024; Shao et al., 2022b] decouple the time series in Fourier space and handle the decoupled signals separately for better use of the hidden information of time series. Simply decomposing time series solely based on frequency intensity is not interpretable and cannot effectively capture the highly coupled spatio-temporal dynamics. Furthermore, this cannot be easily realized by language models as well due to their limited semantics information.

To fully unlock the spatio-temporal knowledge, inspired by dynamic mode decomposition [Schmid, 2010], we propose to capture the underlying dynamic signals in an interpretable manner by leveraging the dynamic system's evolution matrix A. To be specific, considering two state observations X1:t-1 and X2:t, it satisfies X2:t = AX1:t-1. This evolution matrix A is sought in a low-rank setting to capture the modes governing the system's dynamics. By applying a series of mathematical process such as singular value decomposition (SVD) to X1:t-1 and X2:t, we obtain the eigenvectors Ω = [ω1, ω2, ..., ωC] and corresponding eigenvalues V = [v1, v2, . . . , vC] , which can be leveraged to decompose spatio-temporal dynamic systems into different components. Each ωi, referred to as a mode of the dynamical system, reflects certain evolution dynamics of the system. We provide detailed description for calculation process in Appendix A.5.

Specifically, we first obtain Xnorm by normalizing the input X for each node to have zero mean and unit standard deviation using reversible instance normalization (RevIN) [Kim et al., 2021]. Then, we disentangle a set of interpretable dynamic components Xdyn ∈ RN×T×C from intricate spatio-temporal data through reconstructing the system dynamics via modes ωi from the system's evolution matrix's eigenvectors Ω and the corresponding eigenvalues vi. By explicitly decoupling the interpretable nature of the spatio-temporal system, our approach is well-suited to capture the various dynamic behaviors of the system, providing PLMs with a series of components enriched with spatio-temporal semantic information that is significantly easier to comprehend compared to the originally densely coupled dynamic signals.

where Xdyn is a set of spatio-temporal dynamics calculating based on the modes ωi and eigenvalues vi. εi is based on the input observation (see Appendix A.5). Since the dynamics of the system is disentangled, we can distinguish the noise from the dominant dynamic signals in the original data. If only the most significant information is retained during reconstruction, the reconstruction results can remove noise, thus obtaining a smoother state evolutionary information. Therefore, we further reconstruct the whole system Xrec ∈ RN×T with most dominant modes to enhance prediction:

where α represents a set of indices stands for top-k most dominant modes, constructed based on each mode's contribution to the overall system [Schmid, 2010], which is calculated through the analysis of ωi and vi (see Appendix A.5 in supplementary material). Compared to existing Fourier-based methods, the system's evolution matrix Ω is derived from data representing the true dynamics of the system. It can separate modes corresponding to specific intepretable processes, enabling us to capture various aspects of the system's evolution, such as periodic oscillations in traffic flows caused by traffic signals or slow changes in air pollution driven by wind direction [Chen et al., 2012].

Additionally, to enhance the information density of decoupled signals, we employ patching strategy [Nie et al., 2022] to construct patches as the input tokens for PLMs. Given the decoupled signals Xdec = Xrec ∥ Xdyn ∈ RN×T×(C+1) , we divide the observations of each node as a series of non-overlapped patches XPdec ∈ RN×P×TP×(C+1) , where P = [T/TP ] + 1 represents the number of the resulting patches, and TP denotes the patch length. Next, we encode the patched signals as patched embeddings : Xenc = Conv(XPdec , θp) ∈ RN×P×D , where N stands for the number of nodes, and D is the embedding dimension. Conv(·) denotes the patch-wise convolution operator and θp represents the learnable parameters of the patchwise convolution. Unlike previous works [Liu et al., 2024a; Liu et al., 2024b] that simply regard each node as a token, our model treats each patch as one token, allowing to construct fine-grained relationships among both spatial and temporal patterns. By doing so, our model can preserve representations rich in semantic information, allowing PLM's comprehension in both spatial and temporal dynamics more effectively.

3.2 Selective Reprogrammed Language Models

Based on the decoupled signal patches Xenc, how to tackle the complexity of spatio-temporal structures raises another question. Compared to directly handling the spatio-temporal embeddings, representations in natural language space are inherently suitable for PLMs. To enrich spatio-temporal semantics and enable more comprehensive modeling of hidden spatio-temporal relationships, as well as unlocking the reasoning capabilities of PLMs, we further reprogram the components into the textual embedding place via an expanded spatio-temporal vocabulary. When handling textual-based components, the rich semantic information can boost the pretrained knowledge of PLMs, resulting in an adequate modeling of the hidden interactions between disentangled components.

Specifically, we introduce our selective reprogramming strategy, which further constructs an expanded spatio-temporal vocabulary in a differentiable reparameterization process. We begin with E ∈ RV ×d , the pretrained vocabulary of the PLMs, where V is the vocabulary size and d is the dimension of the embedding. We introduce a learnable word mask vector m ∈ RV×1 to adaptively select the most relevant words, where m[i] ∈ {0, 1}. In specific, we first obtain m through a linear layer followed by a Softmax activation, denoted as m = Softmax(EW), where W is a learnable matrix. Afterward, we sample Top-K word embeddings from E based on probability m[i] associated with word i for reprogramming. Since the sampling process is non-differentiable, we employ Gumbel-Softmax trick [Jang et al., 2016] to enable gradient calculation with back-propagation, defined as:

where m′ is a continuous relaxation of binary mask vector m for word selection, τ is temperature coefficient, gi and gj are i.i.d random variables sampled from distribution Gumbel(0, 1). Concretely, the Gumbel distribution can be derived by first sampling u ∼ Uniform(0, 1) and then computing gi = -log(-log(u)). By doing so, we can expand vocabulary space while preserving the semantic meaning of each word.

After obtaining the sampled word embeddings E′ ∈ RK×d , we performmodality alignment by using cross-attention. In particular, we define the query matrix Xq = XencWq, key matrix Xk = E′Wk and value matrix Xv = E′Wv , where Wq , Wk, and Wv. After that, we calculate the reprogrammed patch embedding as follows: Z = Attn(Xq,Xk,Xv) = Softmax( XqX⊤k √d )Xv, where Z ∈ RN×P×d denotes the aligned textual representations for the input spatio-temporal data.

Based on the aligned representation, we utilize the frozen PLMs as the backbone for further processing. Roughly, PLMs consist of three components: self-attention, Feedforward Neural Networks, and layer normalization layer, which contain most of the learned semantic knowledge from pre-training. The reprogrammed patch embedding Z is encoded by this frozen language model to further process the semantic information and generates hidden textual representations Ztext. A learnable mapping function Projection(·) is then used to generate the desired target outputs, which map the textual representations into feature prediction: Yˆ = Projection(Ztext).

4 Experiments

4.1 Experimental Settings

Datasets. We conducted experiments on six commonly used real-world datasets [Lai et al., 2018]*, varying in the fields of traffic [Zhang et al., 2017; Li et al., 2018], solar energy, and air quality † . Each dataset comprises tens of thousands of time steps and hundreds of nodes, offering a robust foundation for evaluating spatio-temporal forecasting models. The detailed statistics of the datasets are summarized in Appendix in supplementary material.

* https://github.com/LibCity/Bigscity-LibCity

† https://www.biendata.xyz/competition/kdd_2018/data/

Baselines. We extensively compare our proposed REPST with the state-of-the-art (sota) forecasting approaches, including (1) the GNN-based methods: [Wu et al., 2019; Shao et al., 2022b; Wu et al., 2020] (2) non-GNN-based sota models: [Shao et al., 2022a; Liu et al., 2023a; Deng et al., 2021] which emphasizes the integration of spatial and temporal identities; (3) sota time series models: [Zhou et al., 2021; Liu et al., 2023b; Nie et al., 2022] (4) PLM-based time series forecasting models: [Zhou et al., 2024]; (5) methods with no trainable parameters: [Cui et al., 2021]. We reproduce all of the baselines based on the original paper or official code.

4.2 REPST Generalization Performance

Few-shot performance. PLMs were trained using large amounts of data that cover various fields, equipping them with cross-domain knowledge. Therefore, PLMs can utilize specific spatio-temporal related textual representations to unlock their capabilities for spatio-temporal reasoning, which can handle the difficulties caused by data sparsity. To verify this, we further conduct experiments on each field to evaluate the predictive performance of our proposed REPST in datasparse scenarios. Our evaluation results are listed in Table 1 with the best in bold and the second underlined. Concretely, all models are trained on 1-day data from the train datasets and tested on the whole test dataset. REPST consistently outperforms other deep models and PLM-based time series forecasters. This illustrates REPST can perform well on a new downstream dataset and is suitable for spatio-temporal forecasting tasks with the problem of data sparsity.

Specifically, our REPST show competitive performance over other baselines in few-shot experiments, demonstrating that PLMs contain a wealth of spatio-temporal related knowledge from pre-training. Moreover, the capabilities of spatiotemporal reasoning can be enhanced by limited data. This shows a reliable performance of REPST when transferred to data-sparse scenarios.

Zero-shot performance. In this part, we focus on evaluating the zero-shot generalization capabilities of REPST within cross-domain and cross-region scenarios following the experiment setting of [Jin et al., 2023b]. Specifically, we test the performance of a model on dataset A after training under a supervised learning framework on another dataset B, where dataset A and dataset B have no overlapped data samples. We use the similar experiment settings to full training experiments and evaluate on various cross-domain and region datasets. The datasets includes NYC Bike, CHI Bike [Jiang et al., 2023], Solar Energy and Air Quality (NYC, CHI, Solar and Air). We compare our performance with recent works in time series or spatio-temporal data with opensourced model weights [Das et al., 2023; Li et al., 2024; Ekambaram et al., 2024].

Our results in Figure 3 show that REPST consistently secure top positions on all settings. This outstanding zero-shot prediction performance indicates REPST's versatility and adaptability in handling diverse scenarios. It does obtain transferable knowledge for dynamic systems by unlocking the reasoning capabilities of PLMs. Its excellent adaptation to brand new scenarios significantly reduces the time and computational resources typically required by traditional approaches. Although our REPST falls a little short to OpenCity-base in Solar Energy→CHI_Bike, it is because of the large amount of traffic related datasets included by OpenCity's pretrain datasets. Compared to it, our REPST is trained on Solar Energy dataset which has almost no connection with such traffic datasets. This relatively comparable performance demonstrate REPST's excellent generative capability in cross-domain settings. The numerical results are shown in Appendix B.3 in supplementary material.

Increasing predicted length. In this part, we analyze the model performance across varying prediction horizons τ ∈ {6, 12, 24, 36, 48}, with a fixed input length T = 48. Figure 4 showcases the MAE across two datasets: Air Quality and NYC Bike, for four models. The REPST model demonstrates the most stable performance across both MAE and RMSE metrics, particularly in longer prediction horizons (τ = 36, 48). In contrast, previous state-of-the-art models exhibit notable performance degradation as the prediction horizon increases. The performance of REPST, on the other hand, remains relatively stable and robust, demonstrating its efficacy in leveraging PLM to improve performance over longer-term predictions, which can also be attributed to its ability to capture both spatial and temporal correlations effectively, making it highly suited for few-shot learning tasks in spatio-temporal forecasting.

4.3 Full Training Performance of REPST

Table 2 reports the overall performance of our proposed REPST as well as baselines in 6 real-world datasets with the best in bold and the second underlined. As can be seen, REPST consistently achieves either the best or second-best results in terms of MAE and RMSE.

Notably, REPST surpasses the state-of-the-art PLM-based time series forecaster FPT [Zhou et al., 2024] by a large margin in spatio-temporal forecasting tasks, which can demonstrate that simply leveraging the PLMs cannot handle problems with complex spatial dependencies. Additionally, the performance of our REPST reaches either the best or second-best results in METR-LA and PEMS-BAY datasets. Previous stateof-the-art models, STAEformer and STID, learn global shared embeddings both in spatial structure and temporal patterns tailored for certain datasets, which is harmful to their generalization abilities but benefits their capabilities to handle single datasets. Our spatio-temporal reprogramming block leverages a wide range of vocabulary and sample words that can adequately capture the spatio-temporal patterns, which do make an impact on unlocking the capabilities of PLMs to capture fine-grained spatio-temporal dynamics.

4.4 Ablation Study

To figure out the effectiveness of each component in REPST, we further conduct detailed ablation studies on Air Quality and Solar Energy datasets with three variants as follows: r/p PLM: it replaces the PLM backbone with transformer layers, following the setting of [Tan et al., 2024]; r/pDecomposition: it replaces the semantic-oriented spatio-temporal decomposer with a transformer encoder; w/o expanded vocabulary: it removes our selective spatio-temporal vocabulary and utilizes the dense mapping function to enhance reprogramming.

Figure 5 shows the comparative performance of the variants above on Air Quality, Solar Energy and Beijing Taxi. Based on the results, we can make the conclusions as follows: (1) Our REPST actually leverage the pretrain knowledge and generative capabilities of PLMs. When we replace the PLM backbone with transformer layers, the performance of all the datasets decline obviously, indicating that the pretrain knowledge makes an effect to handle spatio-temporal dynamics. (2) The semantic-oriented spatio-temporal decomposer which adaptively disentangles input spatio-temporal data into interpretable components can actually enable PLMs to better understand spatio-temporal dynamics. When constructing spatio-temporal dependencies by a transformer encoder layer, it is still unclear for PLMs to comprehend. (3) The impressive performance in w/o expanded vocabulary demonstrates that the selectively reconstructed vocabulary achieves accurate reprogramming which enables PLMs to focus on relationships among tokens in 3D spatio-temporal space.

5 RelatedWorks

5.1 Spatio-Temporal Forecasting

Spatio-temporal forecasting has been playing a critical role in various smart city services, such as traffic flow prediction [Liu et al., 2023a; Shao et al., 2022a; Fang et al., 2023], air quality monitoring [Han et al., 2023; Han et al., 2021], and energy management [Geng et al., 2019]. Unlike traditional time series forecasting, the forecasting challenges associated with spatio-temporal data are often characterized by the unique properties of strong correlation and heterogeneity along the spatial dimension, which are inherently more complex.

Early studies usually capture spatial dependencies through a predefined graph structure [Li et al., 2018; Han et al., 2020;

Shao et al., 2022b; Wu et al., 2020], which decribes the explicit relationships among different spatial locations. In recent years, there is a growing trend toward the utilization of adaptive spatio-temporal graph neural networks, which can automatically capture dynamic spatial graph structures from data [Wu et al., 2019]. Besides, attention mechanism is also widely employed in existing models, as seen in examples like [Guo et al., 2021; Liu et al., 2023a; Fang et al., 2024]. In contrast, [Shao et al., 2022a], a model based on Multi-Layer Perceptrons, achieves state-of-the-art results by utilizing multiple embedding techniques to memorize stable spatial and temporal patterns.

More recently, inspired by the huge success of PLMs in NLP field, there is increasing interest in building pre-trained models for spatio-temporal forecasting tasks. Several studies [Liu et al., 2024a; Liu et al., 2024b; Yan et al., 2023] explore the application of PLMs for handling spatio-temporal data. Furthermore, the strong power of Transformer offers an opportunity to build spatio-temporal foundation models, such as OpenCity [Li et al., 2024] and UniST [Yuan et al., 2024]. Trained on numerous spatio-temporal data, these models demonstrate strong capabilities across diverse forecasting scenarios. However, due to the problems of data-sparsity in multiple spatio-temporal scenarios, it is difficult for these models to gather large amount of data to perform pretraining comprehensively. In addition, the PLM-based spatio-temporal forecasting approaches do not fully leverage PLM's potential. To address these gaps, this paper introduces a new reprogramming framework to leverage PLM's generative and reasoning capabilities for spatio-temporal forecasting, particularly in data-sparse scenarios.

5.2 Pretrained Language Models for Time Series

In recent years, PLMs have demonstrated remarkable performance across various time series analysis tasks [Zhou et al., 2024; Gruver et al., 2024; Sun et al., 2023]. A significant body of research has focused on leveraging PLMs to address these challenges [Gruver et al., 2023]. However, a persistent issue in these efforts is the modality gap between time series data and natural language. To address this challenge, [Jin et al., 2023b] develops a time series reprogramming approach, which can effectively bridges the modality gap between time series and text data. The objective of reprogramming is to learn a trainable transformation function that can be applied to the patched time series data, enabling it to be mapped into the textual embedding space of the PLM.

Nevertheless, [Tan et al., 2024] conducted numerous experiments showing that existing PLM-based approaches do not fully unlock the reasoning or generative capabilities of PLMs. The reason these approaches achieve high performance lies in the similar sequential formulation shared by time series and natural language [Liu et al., 2024d]. Although PLMs can handle one-dimensional sequential data like text and time series, they fall short in capturing dependencies among complex spatio-temporal structure, leading to suboptimal performance for spatio-temporal forecasting tasks. In this work, we propose REPST, which enables PLMs to comprehend complex spatiotemporal dynamics via a semantic-oriented decompositionbased reprogramming strategy.

6 Conclusion

In this paper, we highlight the underlying reason for the poor performance of previous PLM-based approaches in spatiotemporal forecasting, emphasizing the need for the interpretability to fully leverage PLM's potential. To address this problem, we developed REPST, a tailored spatio-temporal forecasting framework that enables PLMs to comprehend the complex spatio-temporal patterns via a semantic-oriented decomposition-based reprogramming strategy. As a result, PLM's potential is full unlocked to handle spatio-temporal forecasting tasks. Extensive experiments demonstrate that our proposed framework, REPST, achieves state-of-the-art performance on real-world datasets and exhibits exceptional capabilities in few-shot and zero-shot scenarios.

Acknowledgments

This work was supported by the National Key R&D Program of China (Grant No.2023YFF0725004), National Natural Science Foundation of China (Grant No.92370204), the Guangzhou Basic and Applied Basic Research Program under Grant No. 2024A04J3279, Education Bureau of Guangzhou Municipality.