MedualTime: A Dual-Adapter Language Model for Medical Time Series-Text Multimodal Learning

1 Introduction

In the medical domain, time series and text modalities are pivotal in representing patient information. Time series, such as electrocardiograms (ECG), electroencephalograms (EEG), and vital signs, provide continuous measurements that reflect a patient's physiological states over time [Wang et al., 2024].

On the other hand, text modalities encompass various clinical documentation, such as radiology reports and physicians' notes, which offer contextual insights for understanding the broader scope of a patient's clinical condition [Davuluri, 2024]. Traditional methods typically involve designing models tailored for single data types and tasks [Fries et al., 2021; Xie et al., 2022; Weimann and Conrad, 2021]. However, jointly modeling time series with text modalities offers richer insights for clinical decision-making. For instance, EEG signals capture electrical activity in the brain, while clinical reports usually provide health history. Analyzing only the symptom history might suggest epilepsy but struggles to specify seizure types, while EEGs detect abnormal brain activity but lack individual medical context. Integrating both modalities can improve diagnostic precision and rationality. A key challenge for jointly modeling is to effectively represent and exploit the complementarity and interactions between different modalities [Guo et al., 2019].

Recently, large-scale pre-trained language models (LMs) have shown exceptional proficiency in understanding sequential data [Chang et al., 2023; Gruver et al., 2024], sparking interest in leveraging them to integrate time series and text modalities. Several contrastive learning-based works in medical domain utilize language models as textual encoders to extract meaningful representations from clinical knowledge, guiding the pre-training of time series encoder [Liu et al., ; Yu et al., 2024; King et al., 2023]. However, the clinical knowledge is not present during the inference stage, leading to a potential loss of critical information. For instance, METS [Li et al., 2024] utilizes a frozen LM to derive embeddings from clinical reports, aligning them with ECG embedding through contrastive learning to augment ECG representation. And only ECG encoder provides decision for inference. Other prompt-based works integrate the text modality into a prompt to guide LM to comprehend time series input [Jia et al., 2024; Liu et al., 2024]. Specifically, MedTsLLM [Chan et al., 2024] augments the text prompt with patientspecific clinical information to adapt LLM for medical signal processing.

In these LM-based multimodal time series-text works, time series is typically considered as the primary modality, being more relevant for decision-making, while text serves as an auxiliary modality to enhance the time series embedding, either by projecting textual knowledge into the time series encoder using contrastive learning [Liu et al., ] or by guiding LM with a textual prompt to comprehend temporal inputs [Chan et al., 2024]. We classify these approaches as temporal-primary multimodal models (Figure 2 (a)). However, in some cases, the textual information is no less important than temporal information. As shown in Figure 1, we conduct a unimodal classification experiment on the PTB-XL ECG dataset and find that 18.8% of samples are correctly classified by the text modality but misclassified by the time series modality, while 13.1% shows the reverse. This highlights the complementarity of these two modalities and suggests that the text modality contains even more unique task relevant information in some cases. Therefore, viewing text modality as auxiliary may introduce bias and fail to capture essential information while a text-primary perspective could enable a more comprehensive understanding of the informative textual content.

To fully exploit the complementarity and mutual benefits of different modalities, we propose a novel textualtemporal multimodal learning paradigm to integrate both temporal-primary and textual-primary perspectives (Figure 2 (a)). However, to effectively construct LM-based approach of such paradigm is technically non-trivial. The most straightforward solution is to train two LM-based submodels separately. Nevertheless, there remain two-fold challenges: First, considering LMs involved, two separately trained submodels suffer non-negligible computational costs. Second, the integration and design of submodels should fuse the two modalities from different perspectives to capture both shared and unique information effectively. Note that the naive multimodal concatenation at LM input layer of existing works [Chan et al., 2024] is difficult to extract the intertwined and high-level multimodal semantics.

To address the aforementioned challenges, we propose MedualTime, a language model for medical time series-text multimodal learning, consisting of a temporal-primary multimodal adapter and a textual-primary multimodal adapter to effectively explore the complementary information in multimodal medical input. Under dual adapter design, each modality has the chance to serve as the primary modality and get improved by the other modality. Within each adapter, multimodal fusion is achieved by injecting learnable adaptation tokens into the top layers to extract high-level multimodal semantics. Furthermore, both adapters share the same LM backbone to reduce computational resources. Meanwhile, we keep the majority of LM's parameters frozen to make different modalities benefit from its pre-trained knowledge. We update only a small portion of LM's parameters, adapting it to our task while enabling efficient fine-tuning. In addition, by pipeline sharing, the modality alignment of different adapters could be accomplished. Our main contributions are as below:

• We are the first to propose a textual-temporal multimodal learning paradigm that treats both time series and text modality equally in medical scenarios. This paradigm fully leverages the rich complementary semantics and captures the intricate interaction between different modalities.
• We propose MedualTime, a dual-adapter language model for medical time series-text multimodal learning. Each adapter performs the mutual integration of time series and text modalities by introducing learnable tokens into the top layers of the LM backbone, facilitating highlevel multimodal semantic fusion. The frozen shared LM pipeline allows both adapters to leverage the pretrained knowledge and achieve alignment in the same semantic space, largely reducing the computational resources with adapter tuning.
• MedualTime demonstrates superior performance on public medical multimodal datasets, showing its strong generalization and transferability in medical diagnosis tasks. Notably, it achieves an average improvement of 8% in accuracy and 12% in F1 score under supervised learning. The code is in https://github.com/start2020/ MedualTime.

2 RelatedWork

In this section, we discuss LM based approaches for time series-text multimodal learning in both medical and general domains. We categorize them into two branches based on how they derive multimodal representation.

Contrastive Learning based Approaches adopt contrastive learning to align time series and text modalities into separate but coordinated spaces to enforce shared information between modalities [Liang et al., 2024]. This branch includes METS [Li et al., 2024], MERL [Liu et al., ], ESI [Yu et al., 2024] and [King et al., 2023]. They leverage LMs to obtain embedding representations of the text modality, which then guide the pre-training of time series encoder, enhancing the quality and robustness of time series representation. For instance, MERL employs contrastive learning to enhance ECG signal representations under the supervision of clinical reports. However, during training, contrastive learning tends to focus on capturing shared semantics between modalities, often overlooking modality-specific information. Furthermore, in the inference stage, only the time series modality is available, while the text modality is absent. As a result, this framework relies solely on time series data for decision-making, potentially disregarding unique and task-relevant information embedded in the text. This limitation can lead to suboptimal model performance.

Prompt-based Approaches incorporate text modality into a textual prompt, which is processed by a language model to generate embeddings. Such embedding is then prepended as a prefix of temporal embedding to enhance LLM's reasoning capacities on time series modality. This branch includes Time-LLM [Jin et al., 2023], UniTime [Liu et al., 2024], GPT4MTS [Jia et al., 2024], InstructTime [Cheng et al., 2024], MedTsLLM [Chan et al., 2024]. For instance, Time-LLM assembles dataset description, task instruction, and data statistics into a textual prompt. The prompt is turned to textual embedding by a frozen LLM, concatenated before the temporal embedding as a prefix to facilitate LLM's understanding of time series data. However, such sequential concatenation implies that the concatenated modalities are not equally important, making LLM focus more on time series.

All these LM-based multimodal works consider time series as the primary modality for decision-making, with text serving as an auxiliary modality to enhance time series modeling. In contrast, MedualTime allows each modality to act as a primary modality, which can comprehensively capture the unique and shared semantics in different modalities.

3 Methodology

In this work, we focus on sample-level medical time series-text data. Specifically, each sample is a timetext pair (e.g., ECG signal and its coupled clinical report). The whole dataset is denoted as S = {(X1, S1) , (X2, S2) , ..., (XN, SN)}, where Xi ∈ RT×d denotes a d-dimension multivariate time series modality with length T and Si denotes the paired textual modality. For simplicity, we omit the sample indicator subscript in the following. To fully utilize the complementary information of different modalities, MedualTime consists of two multimodal adapters, namely a textual-primary multimodal adapter and a temporal-primary multimodal adapter. Each adapter treats one modality as the primary modality and enhances it with the other modality. Both adapters share the same frozen pretrained language model with L layers. Each adapter implements multimodal fusion in the topmost M (M ≤ L) transformer blocks of the language model. The shared language model backbone facilitates efficient fine-tuning and encourages the dual adapters' embedding space alignment.

3.1 Textual-primaryMultimodal Adapter

Processed by the textual tokenizer, the text input can be modeled by I s -length word tokens with embedding Es ∈ RI s×D , where D is the hidden dimension. For the first L -M transformer layers, which are standard transformer layers, the forward process of layer-l is:

where Hls is the output of layer-l with H0 s = Es, MHA, LN,MLP denote the multi-head attention, the layer normalization, and the multi-layer perception, respectively. To obtain the query, key, value matrics at layer-l, W l q , W lk , W l v are parameterized by the pre-trained language model. Meanwhile, the attention operation Attention is defined by:

where Q,K, V are corresponding query, key, and value matrices, dk is the dimension of key.

Furthermore, inspired by the adapter architecture in [Zhang et al., 2023a], we utilize a lightweight adapter mechanism to achieve multimodal modeling at the topmost M transformer blocks. Specifically, we adopt learnable length-P adaptation tokens T l s at each multimodal fusion layer l (L-M+1 ≤ l ≤ L) , where the adaptation tokens T l s ∈ RP×D have the same dimension as language model. As to the secondary temporal modality, a trainable temporal encoder and a cross-modal projector are utilized to transform the time series input into the language model embedding space:

The temporal encoder can be any time-series encoder that best fits the specific datasets, while the projector is a linear layer responsible for dimension transformation. For decreasing the computational cost, different multimodal fusion layers share the same temporal embedding. Thus, the adaptation tokens of textual-primary multimodal adapter is calculated by:

For the topmost M transformer layers, the multimodal forward process is formalized as:

In particular, combined with the pre-trained projection matrices W lk , W l v , the learnable adaptation tokens serve as key, value matrices of the multi-head attention layer. In Equation (8), we perform a zero-initialized gating strategy to achieve multimodal adaptation token fusion [Zhang et al., 2023a]. Gating parameter Gatel will be initialized as zero at the beginning of training, the multimodal adaptation tokens will be injected gradually, which can preserve the pre-trained knowledge and capacities of LMs.

3.2 Temporal-primaryMultimodal Adapter

Considering the sequential property of time series, the temporal-primary multimodal adapter takes the time series data as the language model input. We utilize the common patching strategy for time series modeling in related works [Nie et al., 2022; Zhou et al., 2024]. Several adjacent timestamps will be assembled as a token, which can provide local semantic information within a time series. For a pre-defined patch size p and stride s, the time series input X ∈ RT×d can be reorganized as ˜X ∈ RTs×(p×d) , where Ts = ⌈ T-p s ⌉ + 1 is the number of temporal tokens. Subsequently, we utilize a projector (i.e. linear layer) to adjust the dimension of temporal tokens. The adjusted temporal token can be denoted as Et (Et ∈ RTs×D ). With H0 t = Et as the input of the first transformer layer, the model forward process will be similar to the ones introduced in Section 3.1, e.g., Equation (1-2) and Equation (5 - 8).

For the secondary text input, we use a frozen pre-trained language model (LM) as a text encoder (similar to the temporal encoder in Equation (4)) to extract textual information:

3.3 Pre-trained Language Model Parameters Sharing

Aided by our dual-adapter model design, most of the parameters in the pre-trained language model (e.g., the attention weight matrices Wq,Wk,Wv, and the MLP layer of each transformer block) can be shared between the textualprimary and temporal-primary multimodal adapters. On the one hand, freezing these parameters preserves the language model's knowledge and sequential modeling capabilities. On the other hand, since most of the parameters in our proposed adapters are shared, the increase in training parameters is minimal compared to using a single adapter. This ensures complementary modeling between the two modalities while maintaining efficient fine-tuning. Additionally, by sharing the same LM pipeline, the embedding spaces of the different adapters align easily, further facilitating the integration of dual adapters.

3.4 Training Loss

Supervised Learning. For supervised classification, we add the last transformer layer output of each adapter together to obtain the final multimodal representation. Then, an extra linear classifier and the cross-entropy loss are used for supervised training.

Unsupervised Representation Learning. For unsupervised representation learning, following previous works [Zhang et al., 2023b], we adopt the contrastive learning paradigm. In particular, for data augmentation, we add random Gaussian noise to the original input. The noisecorrupted sample and its original sample are a positive pair within each adapter. We denote H′L s as the augmentation of HL s , and H′L t as the augmentation of HL t . The contrastive loss could be divided into two parts: within-adapter contrastive loss and cross-adapter contrastive loss.

Formally, by maximizing the agreement between positive pairs and minimizing the similarity between negative pairs (i.e., different input instances), in a mini-batch with size B, the within-adapter contrastive losses are

where 1[k̸=i] is the indicator function and τ is the temperature parameter, sim(·, ·) is the dot product between two ℓ2normalized vectors.

The cross-adapter contrastive learning assumes that the embeddings from two adapters for one temporal-textual input pair should be similar. Concurrently, embeddings from different instances are considered negative pairs. In this vein, the cross-adapter contrastive loss is given by:

The overall loss function of unsupervised representation learning is given by:

Note that for the variants of MedualTime, namely MedualTime (Time) and MedualTime (Text), we only adopt the within-adapter contrastive loss for training.

4 Experiments

4.1 Experimental Setup

Datasets Since our approach focuses on medical tasks involving coupled time-series and text modalities, where each modality provides valuable information for decision-making, we conduct experiments using publicly available datasets that meet the necessary criteria above: (1) PTB-XL 1 [Wagner et al., 2020]: This electrocardiogram (ECG) corpus consists of 12-lead ECG signals, which capture the electrical activity of the heart, along with clinical reports describing signal characteristics without diagnostic labels. PTB-XL provides two label sets: a coarse-grained label set for disease detection (four classes) and a fine-grained label set for specific disease classification (five classes). (2) TUSZ v1.5.2 2 [Shah et al., 2018]: The Temple University Seizure Corpus (TUSZ) is a large-scale corpus of EEG (electroencephalogram) signals that record the electrical activity of the brain. It includes 19-channel EEG recordings and the clinical history for each patient session. Similar to PTB-XL, TUSZ offers two label sets: a coarse-grained label set for distinguishing seizure and non-seizure EEG signals (two classes), and a fine-grained label set for seizure type classification (four classes). More details about the datasets, including the label sets, data splits, and preprocessing steps, are provided in Appendix 1.1.

1 https://physionet.org/content/ptb-xl/1.0.3/

Baselines Representative baselines are selected to ensure sufficient experiments. (1) Unimodal LM-free methods: MLP-based models (LightTS [Zhang et al., 2022], DLinear [Zeng et al., 2023]); RNN-based models (LSTM [Hochreiter and Schmidhuber, 1997]); CNN-based models (TimesNet [Wu et al., 2022], TS2Vec [Yue et al., 2022], TS-CoT [Zhang et al., 2023b]); Transformer-based models (Pyraformer [Liu et al., 2021], ETSformer [Woo et al., 2022], Autoformer [Wu et al., 2021], Crossformer [Zhang and Yan, 2022], FEDformer [Zhou et al., 2022], Informer [Zhou et al., 2021], Reformer [Kitaev et al., 2020], iTransformer [Liu et al., 2023], PatchTST [Nie et al., 2022], TS-TCC [Eldele et al., 2021]). (2) Unimodal LM-based methods: BERT [Devlin et al., 2018], GPT-2 [Radford et al., 2019], Llama 3, ClinicalBERT [Wang et al., 2023], GPT4TS [Zhou et al., 2024]. (3) Multimodal LM-based methods: TimeLLM [Jin et al., 2023], UniTime [Liu et al., 2024], GPT4MTS [Jia et al., 2024], MedTsLLM [Chan et al., 2024] for supervised learning; METS [Li et al., 2024], MERL [Liu et al., ] for unsupervised learning. (4) MedualTime variants: MedualTime (Time) for temporal-primary multimodal adapter, MedualTime (Text) for textual-primary multimodal adapter. Note that for GPT-2, BERT, Llama 3, and ClinicalBERT, we use textual embeddings generated by them and then train a linear classifier from scratch for the downstream task. More introductions about baselines are in Appendix 1.4.

2 https://isip.piconepress.com/projects/nedc/html/tuh eeg/

Implementations Inspired by [Liu et al., 2024; Jia et al., 2024; Zhou et al., 2024], which have validated GPT-2's capability in handling time-series data, MedualTime adopts a frozen GPT-2 as the backbone. In the textual-primary multimodal adapter, the tokenizer is derived fromGPT-2. To avoid heavy computational costs, we use a lightweight CNN-based model as the temporal encoder, consisting of three convolutional blocks, each with three CNN layers. This encoder is trained from scratch to adapt to the specific tasks. In the temporal-primary multimodal adapter, following METS [Li et al., 2024], a frozen ClinicalBERT [Wang et al., 2023] pre-trained on medical data as the textual encoder. All hidden dimensions are set to 768 to match the backbone (i.e., GPT-2). The time series patch size and stride are both set to 25. Adam is adopted as the optimizer [Kingma, 2014]. All experiments are implemented by PyTorch Framework with NVIDIA A6000 (48G) GPU.

4.2 Supervised Learning

A linear classifier is added as the output layer of MedualTime under supervised learning. As shown in Table 1, (1) Timeonly models perform better than text-only models, achieving second best in most experiments. (2) Among multimodal approaches based on LMs, UniTime and GPT4MTS exhibit similar performance, outperforming TimeLLM by a 2% accuracy improvement. This performance gap may be due to the differences in their fine-tuning strategies. While TimeLLM relies on a frozen LLM, UniTime and GPT4MTS employ parameter-efficient fine-tuning techniques. MedTsLLM outperforms GPT4MTS, perhaps due to its tailored design for medical data. (3) MedualTime significantly outperforms these LM-based multimodal methods by a 9% accuracy improvement. This discrepancy likely arises from their temporal-primary paradigm, which overlooks critical information embedded in the text modality. In contrast, MedualTime utilizes a textual-temporal perspective, allowing for a more comprehensive understanding of the multimodal interactions. (4) MedualTime achieves the best performance, improving accuracy by 8% and F1 by 12% on average.

4.3 Few-shot Learning for Label Transfer

To evaluate the transferability of representations in a supervised setting, we propose a Few-shot Label Transfer setting for in-dataset transfer between coarse-grained and finegrained label sets (Figure 3). It is common that in real-world applications coarse-grained labels, such as disease presence, are easier to obtain, while fine-grained labels, like specific disease types, are expensive to obtain. In this setting, we pretrain the model on coarse-grained labels and fine-tune it on limited fine-grained labels by freezing the pre-trained parameters and training an additional classifier for few-shot learning. Specifically, we train the model on PTB-XL with four classes and fine-tune on five classes, and train on TUSZ with two classes and fine-tune on four classes. We conduct {5, 10, 15, 20, 50, 100}-shot experiments. Figure 4 shows the results on accuracy and more results are in Appendix 1.3. Generally speaking, all models' accuracy generally shows a continuous growth trend as the few-shot (K) increases. MedualTime consistently outperforms baselines on both datasets and achieves strong classification performance even with limited training samples, demonstrating its effectiveness and robust transferability powered by language models and multimodal inputs.

4.4 Unsupervised Learning

We follow the experimental setting in previous works [Zhang et al., 2023b] to assess our model's ability to generate general representations without ground truth supervision. First, we conduct unsupervised experiments. Then, using the unsupervised embeddings obtained, we train a linear classifier with varying proportions (from 10% to 100%) of labeled training data. Table 2 shows the results for the 100% labeled training data proportion.

As shown in Table 2, text-only models generally outperform time-only models, highlighting the importance of text data in the unsupervised setting. While the multimodal model MERL outperforms the best time-only model TSCoT, METS lags behind, suggesting that multimodal does not always outperform single modality. The effectiveness of multimodal fusion is crucial. MedualTime surpasses MERL in most experiments, emphasizing the advantages of our complementary textual-temporal multimodal design. Overall, MedualTime achieves an average accuracy improvement of 3% and F1 improvement of 2%, consistently outperforming other baselines.

4.5 More Explorations

(a) Ablation Study We ablate MedualTime into MedualTime (Time) and MedualTime (Text) under both supervised and unsupervised settings, as shown in Tables 1 and 2. Generally, MedualTime (Text) achieves better performance than MedualTime (Time) in both settings, suggesting that the backbone language model demonstrates a stronger understanding of text compared to time series. Overall, MedualTime consistently outperforms the single-adapter variants, highlighting the complementary nature of textual and temporal information.

(b) Textual Encoder Analysis We investigate the impact of various textual encoders by examining the following options: BERT, RoBERTa [Liu, 2019], ClinicalBERT [Wang et al., 2023], and GPT-2. The results of supervised learning are in Table 3. More results about unsupervised learning are in Appendix 1.5. As shown in Table 3, BERT-based textual encoders consistently outperform GPT-2. This is likely due to GPT-2's primary focus on text generation, while BERT and its variants excel in comprehending the entire textual input. ClinicalBERT, pre-trained on medical texts, achieves the highest accuracy on average, perhaps attributed to its domainspecific pre-training knowledge.

(c) Visualization To better visualize the learned representations of unsupervised learning, we use UMAP [McInnes et al., 2018] to project the unsupervised representation learning results into 2D plots. (1) Figure 5 displays the embeddings of various encoders on PTB-XL, with labels assigned to different categories. Both TS2Vec (time-only) and ClinicalBERT (text-only) cannot successfully identify different categories of ECGs. (2) Compared with ClinicalBERT, MedualTime (Text) can better distinguish abnormal ECG and normal ECG, indicating the effectiveness of two modalities over one modality. (3) Compared with MedualTime (Time), MedualTime (Text) has obviously better discriminative capacity, supporting the advantage of textual-primary modeling over temporal-primary modeling. (4) Overall, MedualTime provides the most distinct representations, attributed to the benefits of complementary multimodal modeling.

(d) Efficiency Evaluation Due to limited space, we compare the computational costs in seconds on representative baselines and our model under supervised setting in Figure 6. MedualTime employs a frozen LLM backbone and introduces learnable adaptation tokens, enabling more efficient fine-tuning. It has approximately 1.0 million trainable parameters, even smaller than TimesNet with 7.1 M. Its training efficiency (163s per epoch) is close to TimesNet (124s per epoch) even though it has far larger total parameters. Compared with UniTime and TimeLLM, MedualTime has the shortest training time and highest accuracy, highlighting its efficiency and superior performance.

5 Conclusion

We propose a new textual-temporal paradigm for medical time series-text multimodal learning to explore the complementary modeling of different modalities. Our model leverages a dual-adapter design for temporal-primary and textual-primary modeling, with high-level multimodal fusion achieved through learnable token injection in the top layers of a shared frozen pre-trained LM pipeline, ensuring lightweight fine-tuning. Extensive experiments demonstrate that our model significantly enhances representation learning of medical data in both supervised and unsupervised settings. Few-shot label transfer experiments highlight its transferability. Our model leverages the capabilities of language models to equip clinicians with multimodal tools that enhance medical diagnosis.

Acknowledgments

This work is funded by National Natural Science Foundation of China Grant No. 72371217, the Guangzhou Industrial Informatic and Intelligence Key Laboratory No. 2024A03J0628, the Nansha Key Area Science and Technology Project No. 2023ZD003, and Project No. 2021JC02X191.

Contribution Statement

† Co-first Authors, ∗ Corresponding Authors.