Towards Cross-Modality Modeling for Time Series Analytics: A Survey in the LLM Era

1 Introduction

With the proliferation of edge devices and the development of mobile sensing techniques, a large amount of time series data has been generated, enabling a variety of realworld applications [Liu et al., 2025d; Pettersson et al., 2023; Liu et al., 2024b; Cai et al., 2024; Liu et al., 2021b]. Time series data typically take the format of sequential observations with varying features [Liu et al., 2025c; Alnegheimish et al., 2024; Liu et al., 2024i; Liu et al., 2022]. Considerable research efforts have been made to design time series modeling and analytics methods, which enables different downstream tasks, such as time series forecasting [Liu et al., 2021c; Chen et al., 2020; Jin et al., 2022], imputation [Chen et al., 2024; Xiao et al., 2022], classification [Liu et al., 2024d; Liu et al., 2021a], and anomaly detection [Xu et al., 2024].

∗Corresponding author

Recently, large language model (LLM)-based methods [Touvron et al., 2023; Radford et al., 2019] have emerged as a new paradigm for time series modeling. These methods are inspired by time series and natural text exhibit similar formats (i.e., sequence) [Liu et al., 2025a], and assume that the generic knowledge learned by LLMs can be easily transferred to time series [Xue and Salim, 2023]. Although existing surveys have introduced broad overviews of LLM-based time series methods [Jin et al., 2024b; Zhang et al., 2024; Jiang et al., 2024], they overlook the critical challenge posed by the cross-modality gap [Liu et al., 2024d] between time series and textual data. To be specific, LLMs are pre-trained on textual corpora and are not inherently designed for time series, there is a pressing need to develop cross-modality modeling strategies that effectively integrate textual knowledge into time series analytics.

This survey makes a unique contribution to the existing literature by addressing the cross-modality gap between time series and textual data, thereby enhancing LLM-based time series analytics. Figure 1 shows a general framework for LLM-based time series modeling. In this paper, we divide textual data into four types: numerical prompt, statistical prompt, contextual prompt, and word token embedding. To contend with the cross-modality modeling, we summarize two overarching strategies according to recent studies [Jin et al., 2024a; Liu et al., 2025e], i.e., alignment and fusion, to integrate time series with different textual data. For alignment, we identify four key methods: unidirectional retrieval, bidirectional retrieval, contrastive learning, and knowledge distillation. In addition, the fusion strategy primarily relies on concatenation and/or addition to integrate textual information into time series embeddings.

Furthermore, this taxonomy covers forecasting, classification, anomaly detection, as well as multiple combinations across diverse application domains, including general analysis, healthcare, traffic, and event-oriented scenarios. Finally, we conduct experimental evaluations on multi-domain multimodal datasets to assess the effective combinations of textual data and cross-modality strategies for effective time series forecasting, providing practical insights for future research.

The major contributions are summarized as follows.

• We present a comprehensive catalog of literature on LLM-based cross-modality modeling for time series analytics, highlighting recent representative methods.
• We propose a taxonomy that classifies related studies into four groups based on the type of textual data. Additionally, we explore cross-modality modeling strategies, including alignment and fusion, and discuss their applications across various tasks and domains.
• We perform experimental evaluations on multi-domain multimodal datasets to explore effective combinations of additional textual data and strategies that facilitate time series analytics.

2 Formulation

2.1 Definitions

Time Series. We define a time series as an ordered sequence, denoted by X = {x1, . . . , xS} ∈ RS×N , where S represents the sequence length, and N is the number of variables. Each observation xi is an N-dimensional vector at time step i. The scalar vi refers to the numerical value of a specific variable in the time series at time step i.

Textual Data. The textual data T = {P,W} in time series modeling can be categorized into four types: numerical prompt [Gruver et al., 2023], statistical prompt [Liu et al., 2024c], contextual prompt [Liu et al., 2024f], and word token embedding [Pan et al., 2024]. Some studies [Liu et al., 2025e; Jin et al., 2024a] utilize a combination of prompts, denoted as P = {PN,PS,PC}, while others directly adopt word token embeddings W [Pan et al., 2024; Liu et al., 2024e] extracted from LLMs. In this survey, we unify the definitions of these textual data types as follows:

• Numerical Prompt transforms the numerical data of X into a textual format, denoted as PN. Each prompt consists of M words, primarily representing the numerical values of the time series.
• Statistical Prompt encodes statistical features of the time series, such as mean, maximum, minimum, median, top-k, or trend values. These statistics are typically expressed in textual format and denoted as PS.
• Contextual Prompt provides auxiliary descriptions, including dataset metadata, media news, or event-related information. We denote contextual instructions as PC.
• Word Token Embedding refers to the pre-trained weights within LLMs. Instead of using textual prompts, the textual representations can be directly captured from the word token embeddings, denoted as W.

2.2 Cross-ModalityModeling for Time Series Analytics

Given time series X and textual data T, cross-modality modeling aims to learn a function that integrates X with T to generate the target output Y for downstream tasks, such as long-term forecasting, short-term forecasting, classification, imputation, and anomaly detection. Formally, the objective is to learn a mapping function:

where f(·) is the method that aligns and fuses both modalities to enhance time series modeling.

3 Cross-Modality Alignment

We highlight three widely adopted alignment methods: retrieval, contrastive learning, and knowledge distillation.

3.1 Retrieval

Retrieval is the method of leveraging data from one modality to access relevant information in another. Based on the retrieval direction, we categorize it into two types: unidirectional retrieval, where information flows from one modality to another, and bidirectional retrieval, where both modalities can retrieve information from each other.

Unidirectional Retrieval

This method has been applied to forecasting tasks across general domains. For example, TimeCMA [Liu et al., 2025e] introduces hybrid prompts that integrate numerical, statistical, and contextual information to improve time series forecasting. These hybrid prompts are processed by an LLM to generate prompt embeddings, which are then aligned with the original time series through unidirectional similarity-based retrieval. This retrieval leverages time series embeddings to extract disentangled and robust representations from the LLM-empowered prompt embeddings.

Similarly, Time-LLM [Jin et al., 2024a], Time-FFM [Liu et al., 2024e], S2 IP-LLM [Pan et al., 2024], and CALF [Liu et al., 2024d] employ unidirectional retrieval by aligning time series embeddings with word token embeddings in pretrained LLMs, using the former as queries. In contrast, TEMPO [Cao et al., 2024] takes an inverse approach, utilizing prompt embeddings as queries to retrieve the top-K corresponding values from the patched time series input.

Overall, the unidirectional retrieval method typically involves using the time series embedding EX to retrieve relevant information from the LLM-enhanced textual embedding ET , implemented via cross-attention:

where EA is the aligned time series embedding.

Conversely, if the prompt embeddings act as the query:

19%

8%

8%

58%

19%

58%

13%

13%

39%

39%

3%

3%

Here, WQ,WK,WV are learnable projection matrices that transform the features into query, key, and value spaces.

Bidirectional Retrieval

This method extends unidirectional retrieval by allowing both time series and textual embeddings to retrieve information from each other. For instance, LeRet [Huang et al., 2024a] introduces a bidirectional retrieval method for time series forecasting. Instead of relying solely on time series embeddings as queries, LeRet allows textual embeddings to retrieve relevant time series features, creating a dynamic exchange between the two modalities. This bidirectional retrieval strategy improves forecasting accuracy by leveraging the strengths of both data sources.

In the healthcare domain, MedualTime [Zhang et al., 2025] employs bidirectional retrieval for forecasting and classification, integrating textual and time series embeddings to enhance predictive modeling in clinical applications. Formally, bidirectional retrieval can be expressed as an extension of unidirectional retrieval, where either modality can act as the query. For example, textual knowledge is mapped to the time series feature space in the first stage:

Second stage aims to integrate this aligned textual knowledge with time series features:

where E′T denotes the aligned textual embeddings.

3.2 Contrastive Learning

Contrastive learning aims to establish a shared representation by maximizing the agreement between corresponding time Alignment Fusion Alignment Fusion series and textual embeddings while minimizing the similarity between non-corresponding pairs [ ¨Ozyurt et al., 2023]. For example, Chen et al. [Chen et al., 2024] propose a contrastive module to align time series and textual prompts by maximizing the mutual information between a small model's time series representation and LLM's textual representation.

Forecasting Classification Detection Multiple Forecasting Classification Detection Multiple Similarly, METS [Li et al., 2024] utilizes the autogenerated clinical reports to guide electrocardiogram (ECG) self-supervised pre-training. The contrastive strategy aims to maximize the similarity between paired and report while minimizing the similarity between ECG and other reports. TEST [Sun et al., 2024] builds an encoder to embed TS via instance-wise, feature-wise, and text-prototype-aligned contrast, where the TS embedding space is aligned to LLM's embedding layer space.

Formally, contrastive learning for cross-modality alignment can be defined as follows. Given a time series embeddings EX and textual embeddings ET , the contrastive loss function is formulated as:

where sim(·, ·) denotes the similarity function (e.g., cosine similarity), τ is a temperature hyperparameter, and N represents a set of negative samples (i.e., unrelated textual embeddings). The objective is to maximize the similarity between aligned pairs (EX,ET ) while minimizing the similarity between mismatched pairs (EX, ¯ET ).

3.3 Knowledge Distillation

The LLM-based knowledge distillation (KD) achieves a small student model from an LLM, enabling efficient inference solely on the distilled student model. Recent works have been proposed to address the cross-modal misalignment problem with knowledge distillation [Liu and Zhang, 2025], which can generally be categorized into black-box distillation [Liu et al., 2024a] and white-box distillation [Liu et al., 2025b] based on the accessibility of the teacher model's internal information during the distillation process.

CALF [Liu et al., 2024d] is a black-box KD method that aligns LLMs for time series forecasting via cross-modal finetuning. To adapt the word token embeddings to time series data, they align the outputs of each intermediate layer l in time series-based LLM with those of the textual LLM, also align the output consistency between these two modalities to maintain a coherent semantic representation:

where FlX and FlT are the outputs of the l-th Transformer block in time series-based and textual LLMs, respectively. L is the total number of layers in the LLM. γ is the hyperparameter that controls the loss scale from different layers.

In contrast, TimeKD [Liu et al., 2025b] is a white-box KD method that benefits from the design of privileged correlation distillation, the student model explicitly aligns its internal attention maps with those of the teacher model to mimic their behavior.

4 Cross-Modality Fusion

Fusion strategy allows models to leverage complementary information from different modalities [Zhao et al., 2024], enhancing their ability to capture richer contextual dependencies. We summarize two common fusion methods: concatenation and addition of embeddings. Unlike alignment strategies, fusion-based methods often introduce data entanglement issues [Liu et al., 2025e], which may lead to suboptimal performance compared to alignment-based methods.

4.1 Addition-based Fusion

Addition-based fusion integrates textual embeddings with time series representations by summing their feature vectors. This method allows models to incorporate textual information without significantly increasing the dimensionality of the feature space, making it a computationally efficient alternative to concatenation. Unlike concatenation, addition-based fusion maintains a compact representation, ensuring that the model does not introduce unnecessary complexity while still leveraging multimodal information.

Several studies have adopted addition-based fusion for time series analysis. Time-MMD [Liu et al., 2024c], GPT4MTS [Jia et al., 2024], AutoTimes [Liu et al., 2024h], and T3 [Han et al., 2024] add the textual embedding with time series embedding for time series analysis, respectively. Formally, the addition can be expressed as follows:

where EF is the fused embeddings and + denotes addition.

4.2 Concatenation-based Fusion

Concatenation-based fusion directly merges textual embeddings with time series features to create a joint representation. This method enables models to incorporate textual information alongside time series data, allowing for a more comprehensive feature space. While concatenation provides a straightforward way to multimodal integration, it can increase the dimensionality of the feature space, leading to greater computational complexity. Moreover, the lack of explicit alignment mechanisms between modalities may introduce noise, reducing the effectiveness of downstream tasks.

Some studies directly concatenate time series and textual embeddings. For instance, UniTime [Liu et al., 2024f] concatenates the contextual prompt embedding with time series embedding to retain a LLM-based unified model for crossdomain time series forecasting. SIGLLM [Alnegheimish et al., 2024] concatenates the contextual prompt embedding with time series embedding for the zero-shot anomaly detection task. TEMPO [Cao et al., 2024] concatenates the word token embedding with different time series features, such as trend, seasonal, and residual, for time series forecasting. Time-FFM [Liu et al., 2024e] concatenates word token embedding with time series embedding for time series forecasting. The concatenation can be formulated as follows:

where || denotes concatenation.

Other works utilize multiple strategies to integrate data embeddings. Beyond retrieval-based alignment, Time-LLM [Jin et al., 2024a] further enhances the adaptability of LLMs for time series forecasting by concatenating the textual prompt embedding as a prefix to the general time series embedding. S2 IP-LLM [Pan et al., 2024] concatenates time series embedding and retrieved embedding to avoid the data entanglement issue. FCSA [Hu et al., 2025] concatenates the time series embeddings and textual prompt embeddings to extract fine-grained features for further alignment. After additionbased fusion, T3 [Han et al., 2024] concatenates the contextual prompt and traffic data embeddings to maximize the utilization of the training data for the traffic forecasting task.

5 Experiments

We perform extensive experimental evaluations on multidomain, multimodal datasets. We employ four types of textual data in Figure 3 (a) as well as cross-modality alignment and fusion strategies in Figure 3 (b). Specifically, we select the three most common methods from the literature: unidirectional retrieval-based alignment (21%), concatenation-based fusion (36%), and addition-based fusion (18%) in Figure 3 (c). We focus on the time series forecasting task, accounting for 67% of reported tasks across multiple domains in Figure 3 (d). We also implement a single-modality model without textual inputs. Our code and datasets are available 1 .

5.1 Dataset Description

We utilize datasets from five domains [Liu et al., 2024c], spanning agriculture, climate, economy, energy, and health. Each dataset consists of time series with relevant textual data. The textual data includes expert reports and news summaries, each annotated with timestamps corresponding to the periods they describe.

Time Series Data is summarized in Table 1, covering five domains: agriculture, climate, economy, energy, and health. The datasets are recorded at varying temporal resolutions, including weekly and monthly frequencies, with records spanning from the 1980s to 2024. Each dataset consists of univariate or multivariate time series, with the number of dimensions ranging from 1 to 11.

Textual Data consists of expert reports and news summaries. Expert reports are categorized as statistical prompts, as they provide insights into averages and trends within specific timeframes. News summaries can serve as contextual prompts, offering cues about future trends. We follow TimeCMA [Liu et al., 2025e] to wrap time series values into numerical prompts. The word token embeddings are obtained from the pre-trained GPT-2 [Pan et al., 2024].

1 https://github.com/ChenxiLiu-HNU/CM2TS

5.2 Implementation Details

Figure 4 presents an overview of our cross-modality modeling framework for time series analysis, comprising three key components: a pre-trained LLM, an alignment or fusion layer, and an adapter, detailed below.

Pre-trained LLM. This module includes a tokenizer and a pre-trained GPT-2 model with frozen weights, efficiently embeds textual data.

Encoder. It can be a Transformer-based encoder than embeds time series data and captures temporal dynamics [Liu et al., 2024g], with reversible instance normalization applied to normalize time series values.

Alignment or Fusion. The alignment strategy employs a unidirectional retrieval method. The fusion strategy includes concatenation and addition methods.

Task Adapter. The adapter is designed by the downstream task. In this section, we conduct the forecasting task. We use a linear projection layer to predict the future time series. We also de-normalize the values to restore the actual prediction.

5.3 Experiment Settings

We configure the lookback window size to 36 for weekly data and 8 for monthly data, while setting the prediction horizon to 24 time steps across all datasets. We evaluate models' performance using Mean Squared Error (MSE) and Mean Absolute Error (MAE). The model is optimized using the Adam optimizer, with Cosine Annealing as the learning rate scheduler. Training is conducted on NVIDIA A100 GPUs, with 20 epochs for weekly data and 50 epochs for monthly data.

5.4 Results and Discussion

Table 2 presents the results of cross-modality time series forecasting across five domains. The findings highlight the impact of incorporating textual data and provide insights into the effectiveness of different alignment and fusion methods.

Textual Data Enhances Foresting. Overall, textual data significantly enhance time series forecasting performance. For example, in the climate domain, the retrieval-based numerical prompt reduces MSE by 22.6% compared to the time series-only baseline. Among the different types of textual information, numerical prompts and statistical prompts lead to the most notable improvements. These text types contain numerical values that directly correlate with time series patterns, providing structured signals for forecasting. In contrast, contextual prompts and word token embeddings show relatively weaker performance, as they contain less structured information, which may not align as closely with numerical trends.

Numerical Prompts Perform Better. Numerical prompts consistently deliver the best performance across most domains. This is particularly evident in climate, energy, and health forecasting. For example, in the climate and health domains, numerical prompts reduce MSE by 21.15% and 17.95%, respectively, compared to contextual prompts. Statistical prompts show strong results in economy forecasting. Word token embeddings perform the worst, indicating that general semantic representations may not effectively capture time series-relevant information.

Alignment Outperforms Fusion. We evaluate three methods for each textual data type: retrieval-based alignment, addition-based fusion, and concatenation-based fusion. Retrieval-based alignment consistently performs well, particularly for numerical and statistical prompts. In the the economy domain, when using numerical prompts as textual data, retrieval reduces MSE by 16.96% compared to concatenation. Addition-based fusion often outperforms concatenation with the three textual prompts but underperforms it when using word token embeddings.

Domain-SpecificObservations. The impact of textual data varies across different domains. Economy and health forecasting benefit considerably from numerical and statistical prompts, where structured reports align well with economic indicators and public health trends. Climate and agriculture forecasting show more moderate improvements. The reason is that these domains may rely on more external factors that are not always well captured by textual descriptions. Energy forecasting achieves the better results with addition-based fusion of numerical prompts.

6 Future Directions

Multi-ModalityModeling. Expanding beyond the integration of time series and textual data, future research could delve into additional modalities such as images [Huang et al., 2024b], video [Wang et al., 2024a], and audio [Huang et al., 2021]. In this context, it is essential to explore the capability of LLMs for enhancing multimodal representation. Recent advancements in multi-modal LLMs exemplify the potential of such integrations. For instance, Meta's LlaMa 3.2 processes both images and textual data, enabling applications ranging from augmented reality to document summarization. These developments underscore the importance of investigating how LLMs can be leveraged to create effective multimodal modeling.

Improving Effectiveness. While LLM-based crossmodality methods have demonstrated strong capabilities, they do not always surpass smaller, task-specific models [Wang et al., 2024c]. In some cases, employing an LLM with an excessive number of parameters can lead to overfitting, particularly on specialized tasks across several domains. Future research could focus on techniques such as dynamic model selection, meta-learning, and continual adaptation that can help improve model effectiveness by allowing models to adjust to changing data distribution.

Efficient Optimization. Despite their success, existing studies still meet the challenge of high computational costs, particularly when processing long sequences, more tokens, or handling multivariate data. This is due to the high dimensionality of multivariate time series (i.e., multiple variables over timestamps) and the multi-head attention mechanism within LLMs Recent advancements have explored strategies to mitigate these challenges, such as last token storage [Liu et al., 2025e], knowledge distillation [Gu et al., 2024]. Future research could focus on developing lightweight architectures, efficient attention mechanisms, and adaptive computation frameworks to optimize efficiency and scalability.

Transparency of LLMs. LLMs have demonstrated remarkable performance in textual-time series analytics [Wang et al., 2024b], yet they often operate as "black-box" systems, raising concerns about their reasoning processes and overall transparency. Much of the current research primarily applies or fine-tunes LLMs without an explicit focus on exposing their internal reasoning processes. This lack of interpretability can hinder trust, particularly in high-stakes applications such as healthcare and finance. Moreover, LLMs are prone to generating hallucinations, seemingly plausible but incorrect outputs, which further complicates their deployment in realworld scenarios. Future research on textual-time series analysis could prioritize enhancing the transparency of LLMs, ensuring that these models operate more reliably during subsequent alignment or fusion processes.

7 Conclusion

This paper aims to highlight the importance of cross-modality modeling for time series analytics in the LLM era. We propose a novel taxonomy from a textual data-centric perspective, categorizing existing studies by key data types, namely numerical prompts, statistical prompts, contextual prompts, and word token embeddings. Our main premise is that through cross-modality alignment and fusion, textual data can significantly enhance time series analytics tasks across diverse domains. To validate this viewpoint, we perform multidomain multimodal experiments to systematically evaluate the effectiveness of various alignment and fusion strategies in key time series tasks. Finally, we explore open challenges and promising directions for future research.

Acknowledgments

This study is supported under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contributions from the industry partner(s).