ECG2TOK: ECG Pre-Training with Self-Distillation Semantic Tokenizers

1 Introduction

Cardiovascular diseases (CVDs), responsible for 17.9 million annual deaths [Kyu et al., 2018], are central to the UN's "Good Health and Well-being" Sustainable Development Goal. While electrocardiogram (ECG) analysis is critical for early CVD detection, its efficacy remains constrained in resource-limited settings due to inadequate healthcare infrastructure. Self-supervised learning (SSL) offers a solution by minimizing labeled data dependency, reducing costs, and improving diagnostic accessibility to advance healthcare equity [Krishnan et al., 2022].

∗Corresponding author.

Existing state-of-the-art ECG SSL methods rely on masked reconstruction in the time-frequency domain to learn semantic representations, which has notable limitations in the context of ECG. The inherent characteristics of ECG signals include the periodic repetition of specific waveforms (such as P waves, QRS complexes, and T waves), as well as significant individual differences in amplitude, duration, and morphology [Clifford, 2002; Yuan et al., 2024; Yuan et al., 2025]. However, reconstruction loss tends to emphasize lowlevel time-frequency features, leading to overfitting of repetitive waveform parts and individual-specific characteristics. Discrete label prediction may serve as a better pre-training objective compared to reconstruction, based on applications in other domains [Ramesh et al., 2021; Bao et al., 2021; Hsu et al., 2021]. This is because discrete label prediction reduces the impact of information redundancy caused by periodic waveforms by mapping ECG signals with similar physiological meanings to the same label space. Additionally, discrete label prediction avoids over-reliance on individualspecific low-level time-frequency features (such as amplitude, duration, and morphology), allowing the model to better learn global semantic features related to heart health.

However, applying discrete label prediction to ECG SSL faces great challenges. On the one hand, ECG signals are continuous and lack discrete semantic units similar to words in natural language processing, which makes label prediction difficult. On the other hand, ECG data exhibits a high degree of variability, and characteristic waves show obvious temporal persistence and morphological differences. For example, semantically meaningful waves such as QRS complexes, T waves, and P waves vary greatly in duration and shape, increasing the complexity of extracting semantic token from ECG signals. These challenges make it difficult to effectively apply commonly used tokenizers (such as Vector Quantization [Peng et al., 2022] and Clustering [Hsu et al., 2021; Bao et al., 2021]) to discrete label generation for ECG.

To address these challenges, we propose a novel framework, ECG2TOK, which integrates a self-distilled semantic tokenizer with a SSL model, employing an iterative optimization mechanism for ECG pretraining, as illustrated in Figure 1. Specifically, ECG2TOK utilizes a target encoder to extract context embeddings enriched with knowledge distillation and performs online clustering to refine the high-dimensional embedding space into a semantic codebook, generating corresponding discrete semantic labels. Discrete semantic label prediction and masking strategies are then used to guide SSL model training and enhance the ability of the context encoder to learn universal representations. The introduction of knowledge distillation strengthens the expression of semantics and improves the consistency of the discrete label generation process. ECG2TOK outperforms state-of-the-art ECG SSL methods across six downstream ECG classification tasks. Additionally, visualization experiments reveal that the discrete labels generated by the tokenizer exhibit consistent semantics closely related to clinical features. This demonstrates that ECG2TOK effectively generates discrete labels containing high-level semantics. By predicting these labels, the model gains the ability to overcome signal redundancy and individual variability, achieving efficient and robust ECG modeling and understanding.

Our contributions are as follows:

• We propose a novel ECG pretraining framework with self-distillation semantic tokenizers, which utilizes discrete label prediction loss and outperforms traditional reconstruction loss methods, opening new avenues for ECG pretraining.
• We provide an effective semantic tokenizer to quantize continuous ECG features into semantically compact discrete labels, facilitating future ECG pretraining and time series pretraining work.
• Our method achieves state-of-the-art results on six ECG classification benchmark tasks with minimal pretraining cost, and up to a 30.73% AUC improvement under low-resource conditions.

2 RelatedWork

2.1 ECG-based SSL

Recent advances in self-supervised learning for ECG have shown strong generalization capabilities, greatly enhancing downstream task performance [Lai et al., 2023]. Among these, contrastive learning has gained significant attention. CLOCS [Kiyasseh et al., 2021] promotes representation consistency across spatial, temporal, and patient dimensions, while ASTCL [Wang et al., 2023] introduces adversarial spatiotemporal contrastive learning to improve robustness and semantic invariance. Other approaches [Lan et al., 2022;

Yang and Hong, 2022] explore inter- and intra-subject representations, as well as temporal and frequency domain alignment. However, contrastive learning methods often rely on complex pair construction and prior knowledge [Zhang et al., 2022], limiting their applicability. In contrast, masked reconstruction methods explicitly encourage the model to capture global contextual information by predicting masked content. For example, CRT [Zhang et al., 2023] proposes reconstructing both temporal and frequency domain data to discover cross-domain correlations and enhance representation learning. ST-MEM [Na et al., 2024] adopts a similar masked modeling strategy. However, traditional reconstruction methods tend to focus on low-level time-frequency detail recovery, neglecting high-level semantic information, which limits their generalization capability. This highlights the need for incorporating high-level semantic modeling to further enhance the representation learning of ECG models. HeartLang [Jin et al., 2025] treats ECG as a language with words (QRS waves) and sentences (rhythms), facilitating representation learning by predicting the vocabulary index of each word.

2.2 Discrete Labels-based SSL

Discrete label-based self-supervised learning (SSL) has made significant strides in natural language processing (NLP) [Lan, 2019; Kenton and Toutanova, 2019a; Liu, 2019], computer vision (CV) [Bao et al., 2021; Peng et al., 2022], and speech processing [Hsu et al., 2021; Liu et al., 2023]. In NLP, BERT [Kenton and Toutanova, 2019b] uses masked prediction to effectively learn representations from discrete input sequences. In CV, Beit [Peng et al., 2022] introduce vector-quantized (VQ) visual tokenizers, enabling the generation of discrete tokens for masked image patches and the prediction of their original tokens. Similarly, HuBERT [Hsu et al., 2021] employs iterative hidden state clustering to generate discrete labels for speech SSL tasks, showcasing the versatility of this approach across domains.

However, applying discrete label-based self-supervised learning (SSL) methods to ECG signals presents two major challenges. First, ECG signals are continuous-valued sequences, lacking discrete semantic units like phonemes in speech or words in NLP, making it difficult to directly apply predictive loss. Second, due to significant variations in waveform and rhythm across individuals, ECG signals contain excessively larger data variations, making it challenging to effectively capture unified semantic patterns using methods like VQ tokenizers or clustering. To address this, our approach incorporates self-distillation combined with clustering as a discrete tokenizer, enhancing the semantic representation capability before discretization.

3 Methodology

3.1 Overview

ECG2TOK pre-training framework integrates a semantic tokenizer and an ECG masked SSL model, leveraging discrete label prediction to optimize the pre-training process. As shown in Figure 2, the semantic tokenizer generates discrete semantic labels from unlabeled ECG data, which are then used to train the ECG SSL model through a combination of masked prediction and discrete label prediction tasks. During iterative training, ECG2TOK uses knowledge distillation to train a new semantic tokenizer, replacing the previous tokenizer in the next training iteration.

This process enables mutual enhancement between the semantic tokenizer and the SSL model: the tokenizer learns semantically rich representations from the SSL model, while the SSL model benefits from the discrete semantic labels generated by the tokenizer. Together, they produce robust and semantically aligned ECG representations.

3.2 Masked ECG SSLModel

We propose a masked modeling task for training ECG pretraining models, as illustrated in the upper part of Figure 2. Unlike traditional methods that rely on reconstructing the time-frequency details of input ECG signals for model optimization, our approach introduces a Transformer-based label predictor to replace the conventional decoder, predicting patch-level discrete labels. These predicted labels are combined with pseudo-labels generated by the semantic tokenizer (detailed in Section 3.3) to compute the loss, providing an innovative and efficient solution for ECG model pre-training.

Input Embedding. Inspired by ST-MEM [Na et al., 2024], the raw ECG signal I ∈ RL×12 is divided into a set of nonoverlapping patches I′ ∈ RT×(12·p) , where T = L/p and p represents the length of each patch. Subsequently, a learnable linear projection is applied to each patch, resulting in the

where Ep denotes the embedding for each patch, and dmodel is the projected feature dimension.

To construct the final input, the patches embedding Ep is combined with a temporal positional embedding Et ∈ RT×1×dmodel and a spatial positional embedding Es ∈ R1×12×dmodel :

where X represents the final input patch embedding.

Masked Prediction. Given the input patch embedding X = {xt}Tt=1 , we apply random masking to 75% of the patches embedding. The masked positions in the patches are denoted byM = {1, . . . , T }0.75T . Next, the unmasked patch XU = {xt : t ∈M}Tt=1 is fed into the ViT encoder Eθ, producing the encoded representations RU = {rt : t ∈M}Tt=1 . The encoded non-masked patch features are then combined with the masked patch features, and passed to the label predictor Pθ. The predictor outputs the predicted discrete labels Z = {zt}Tt=1 . The calculation formula is as follows:

where XU is the unmasked patches embedding, RU is the unmasked patches representation and the zero vector represents the masked patch features.

3.3 Self-Distilled Semantic Tokenizer

We propose a self-distilled semantic tokenizer that combines self-distillation with online clustering to quantize continuous ECG features into discrete semantic labels. The framework leverages a target encoder to distill knowledge from the context encoder, generating high-dimensional semantic aware embeddings. Online clustering is then applied to these highdimensional embeddings from target encoder to construct a codebook. The pseudo-labels are obtained through the codebook indices and are used to guide the self-supervised training of the semantic network. Unlike traditional offline clustering or vector quantization methods, our approach integrates online clustering directly into the training process, enabling dynamic adaptation between semantic representations and discrete labels. This significantly enhances the model's optimization efficiency and semantic consistency.

Target Encoder Parameterization. As illustrated in Figure 2, our method is inspired by data2vec [Baevski et al., 2023] and adopts the same overall framework. The objective is to train the context encoder Eθ under the guidance of the target encoder E∆, where both models share the same architecture. In our work, this architecture is a 12-layer Transformer encoder [Vaswani, 2017].

The weights of the context encoder, denoted as θ, are updated using backpropagation based on the gradient of the loss function. The target encoder weights, represented by ∆, are initialized to be identical to the context encoder weights at the beginning of training. During training, the target encoder weights are updated through an Exponentially Moving Average (EMA) of the context encoder weights, expressed as:

where τ is a hyperparameter that governs the update frequency of the target encoder weights. The value of τ increases linearly throughout training, starting from an initial value τ0 and gradually approaching 1. This scheduling allows the target encoder to adapt quickly during the early stages of training while stabilizing in later stages.

Codebook Assignment. After distilling high-dimensional semantic aware embeddings from the target encoder, we leverage online clustering to construct a codebook (set of centroids) E = {e1, . . . , eV }, where there are V codewords (centroids) ei ∈ RD . To align the target encoder outputs with the codebook, each codebook entry v is associated with a set ˜Zv, which contains the representations closest to the current centroid ev. Formally, this is defined as:

where the set index v will be used as a pseudo label to train the semantic network.

Codebook Update. To ensure the codebook dynamically adapts to the target encoder outputs, each codeword ev is updated using an EMA. The update rules are given by:

where sv represents the accumulated sum of all neighboring target representations (i.e., ˜Zv as defined in Eq. 6), while nv denotes the count of neighbors. Both terms are updated using an EMA mechanism with a decay rate σ, enabling the computation of the codeword ev as the weighted average of its neighboring representations. During initialization, sv, nv, and ev are all set to 1.

By defining codewords based on their corresponding neighboring representations, these codewords can be interpreted as semantic units derived from the target model in an unsupervised manner. Subsequently, these units are used to train the SSL model. The clustering approach produces discrete labels for ECG based on their context, which have consistent semantics. As shown in the section 5.4, the semantics of these codewords are closely consistent with those of the clinical patterns.

where ψ is the softmax activation over the codebook, v is the codeword index of the corresponding patch from the target network (i.e., ˜zt ∈ ˜Zv), and zt is the corresponding output feature of the context network.

4 Experiments

4.1 Pre-training Configuration

MIMIC-IV-ECG. We use the MIMIC-ECG dataset [Gow et al., 2023] to pre-train the proposed ECG2TOK framework. This dataset comprises 800,035 ECG recordings collected from 161,352 unique subjects, with each recording lasting 10 seconds and sampled at 500 Hz. To handle missing or invalid values (e.g., "NaN" and "Inf") in the ECG data, we replaced them with the average of six neighboring points. After this preprocessing step, we obtained a final pre-training dataset containing 771,693 samples.

Implementation. MIMIC-IV-ECG dataset is split into training and validation sets in a 9:1 ratio, with the validation set used for semantic token analysis. During pre-training, a randomly initialized transformer [Vaswani, 2017] is employed as the target encoder, and input data is processed with a 75% random masking rate. The number of discrete semantic tokens generated through online clustering is set to 128, with a cluster center dimension of 768. The learning rate is configured to 1.5 × 10-4 , and the model is trained for a total of 7 epochs using the AdamW optimizer with momentum parameters [0.9, 0.95]. All experiments are conducted on a single NVIDIA A800 GPU with a batch size of 256. To ensure reproducibility, the random seed is fixed at 0.

4.2 DownstreamTasks Configuration

We evaluate our method on six tasks across three publicly available ECG datasets, covering over 100 cardiac conditions. Data splitting details are provided in the appendix.

PTB-XL. The PTB-XL dataset [Wagner et al., 2020] contains 21,837 12-lead ECG recordings from 18,885 patients, each lasting 10 seconds and sampled at 500 Hz. The dataset supports classification into Superclass (5 classes), Subclass

(23 classes), Form (19 classes), and Rhythm (12 classes). We adopted the official split [Wagner et al., 2020] for training, validation, and testing.

CPSC2018. The CPSC2018 dataset [Liu et al., 2018] includes 6,877 12-lead ECG recordings with durations between 6 and 60 seconds. It contains 9 distinct labels and is divided into 70% for training, 10% for validation, and 20% for testing.

Chapman-Shaoxing-Ningbo (CSN). The CSN dataset [Zheng et al., 2020; Zheng et al., 2022] contains 45,152 12lead ECG recordings, each lasting 10 seconds and sampled at 500 Hz. After excluding samples with "unknown" annotations, the refined dataset includes 23,026 recordings across 38 labels. We used a 70%:10%:20% split for training, validation, and testing.

Implementation. For linear probing, the target encoder remains frozen, and only the linear classifier with randomly initialized parameters is trained. To assess the method's performance under low-resource conditions, linear probing is conducted on each task using 1%, 10%, and 100% of the training dataset. The training process spans 100 epochs, including a 5-epoch warmup phase, with a base learning rate of 5.0 × 10-3 , a weight decay of 0.05, and a batch size of 16. The AdamW optimizer is utilized, with gradient clipping and distributed evaluation omitted. Binary cross-entropy (BCE) is employed as the loss function, suitable for multi-label classification. Macro AUC is used as the evaluation metric for all downstream tasks. To ensure reproducibility, the random seed is set to 0, and torch.manual seed is fixed at 42. Test results are derived from the best-performing validation model.

5 Results and Discussion

5.1 Main Results

Table 1 presents the linear probing performance of our proposed ECG2TOK model across six tasks and three resource configurations, compared to the current state-of-the-art SSL methods. For each dataset, the bold numbers represent the best performance, while the underlined numbers indicate the second-best performance (this formatting is applied consistently throughout all tables in this paper). The results demonstrate that ECG2TOK achieves significant performance improvements across multiple tasks, obtaining the highest AUC in almost all resource settings (highlighted in bold in the table). Moreover, the AUC performance of ECG2TOK is particularly outstanding on the PTBXL-Rhythm, CPSC2018, and CSN datasets, with average AUC improvements of 16.4%, 9.43%, and 15.6%, respectively, over the next-best models. These results indicate that, compared to other SOTA methods, ECG2TOK exhibits superior representation capability, effectively extracting high-level semantic features and adapting to various downstream tasks.

Notably, under low-resource conditions (using only 1% and 10% of the training samples), ECG2TOK continues to exhibit superior performance compared to other models, highlighting its strong generalization ability in resourceconstrained environments. For instance, with only 1% of the training data, ECG2TOK achieves an AUC of 81.16 on the PTBXL-Rhythm dataset, representing an improvement of approximately 30.73% over the next-best model, HeartLang. Similarly, on the CSN dataset, ECG2TOK achieves an improvement of around 19.66% over the next-best model, BarlowTwins. In some classification tasks, such as CPSC2018 and CSN, ECG2TOK outperforms other SSL methods trained on 100% of the data, even when using only 10% of the training data. These significant improvements suggest that ECG2TOK learns more advanced, semantically rich, and general representations, outperforming other self-supervised learning methods and further validating its robustness and effectiveness under low-resource conditions.

In summary, ECG2TOK consistently achieves superior performance across various downstream tasks, demonstrating its strong representation ability and broad adaptability in self-supervised learning. Its remarkable performance in low-resource scenarios underscores its robustness and effectiveness, making it highly suitable not only for largescale datasets but also for practical applications in resourceconstrained fields such as healthcare and health monitoring.

5.2 Pre-training Efficiency

The experimental results demonstrate that ECG2TOK significantly outperforms existing ECG self-supervised learning methods in both pre-training efficiency and performance. As shown in Table 2 and Figure 3, ECG2TOK achieves exceptional results with only 7 pre-training epochs, reducing pre-training time by 28.5 times compared to HeartLang and 9.12 times compared to MERL. Notably, ECG2TOK surpasses HeartLang (pre-trained for 200 epochs) and MERL (pre-trained for 50 epochs) after just 4 epochs, highlighting its ability to quickly learn high-quality representations. After 7 total pre-training epochs, ECG2TOK achieves an average AUC of 81.18 across six downstream tasks, significantly outperforming both models despite their longer pre-training periods.

The efficiency improvement of ECG2TOK can be attributed to three key factors. First, ECG2TOK employs a 75% high masking rate during pre-training. This large masking means that a significant portion of the ECG data is excluded before being input to the context encoder, enhancing batch processing capabilities and leveraging parallel computing advantages to improve efficiency. Second, the reconstruction objective in the proposed self-supervised framework differs from traditional ECG input space reconstruction objectives. It provides semantically rich discretized tokens (cluster center) as pre-training targets and encourages the model to discard redundant details, allowing the model to capture key features faster, thereby accelerating the convergence of the pretraining process. Lastly, the incorporation of self-distillation further speeds up the learning by refining the model's feature representations, allowing for faster and more efficient training.

5.3 Ablation Study

Training Target. Table 3 illustrates the importance of each component in the ECG2TOK framework. Removing the semantic tokenizer (w/o semantic tokenizer), where the model reverts to a traditional masked reconstruction of raw signals, results in the most significant performance drop of 10.1%. Excluding self-distillation (w/o self-distillation), where ECG's continuous input space is directly discretized into labels without knowledge compression from a target model, leads to a performance decrease of 4.13%. Finally, removing the clustering module (w/o clustering), which shifts the training objective to reconstructing the abstract feature rather than clustering-based discretization, results in a drop of 3.79%. Overall, the full ECG2TOK model achieves the best performance (average AUC: 81.18), demonstrating the complementary contributions of these components in effectively capturing and leveraging ECG signal representations.

Masking Ratio. We investigate the impact of random masking rates during pre-training. The masking rate largely determines the number of unmasked patches accessible to the predictor during training, which directly affects the complexity of the pre-training task. Figure 4 illustrates how finetuning performance on downstream tasks changes with different masking rates. Notably, when the masking rate is set to 75%, our proposed ECG2TOK model achieves the best finetuning performance, indicating that at this moderately high masking rate, the model is able to effectively transfer and optimize downstream tasks while maintaining a higher pretraining task complexity.

Number of Clusters. We investigate the impact of different codebook sizes (Cluster size) on the performance of the ECG2TOK model. The results in Table 4 show that codebook size significantly affects model performance. A small codebook (e.g., 32) leads to lower performance, likely due to insufficient feature representation. Performance improves with larger codebooks, peaking at a size of 128, which balances feature representation and computational complexity. However, further increasing the codebook size to 512 or 1024 results in a slight decline, possibly due to dispersed features and reduced generalization. In conclusion, a codebook size of 128 is found to be the most optimal, as it effectively captures data features and provides the best performance under varying resource conditions.

5.4 Visualization

To validate whether the semantic tokenizer in ECG2TOK captures clinically valuable information, we perform inference on the MIMIC-IV test set, which includes annotated clinical text reports and ECG signals from diverse patients. We cluster the ECG patches and visualize the 10 patches closest to each cluster center from different subjects, displaying their corresponding clinical reports for comparison. Each label is presented as "P xx(ID xx):...xxx...", where "P" denotes the ECG patch index, "ID" is the subject ID, and "...xxx..." is the clinical report. By comparing the ECG waveforms with the associated clinical reports, we assess whether the discrete labels accurately reflect clinical features, thus confirming the effectiveness of the semantic tokenizer.

As shown in Figure 5, ECG patches sharing the same label or cluster center exhibit consistent semantic patterns aligned with clinical phenomena. For example, patch "37" shows ST segment changes, indicative of myocardial ischemia, while "47" demonstrates QT interval prolongation, a sign of arrhythmias or drug effects. Patch "67" presents normal waveforms, indicating a healthy heart. These results suggest that the generated labels capture clinically relevant ECG features and contain meaningful, consistent semantic information.

The consistent semantic patterns across different subjects further indicate that the pseudo-labeling process is robust and generalizes well across clinical scenarios. This validates that the semantic tokenizer effectively captures high-level clinical patterns, not just memorizing patient-specific data, confirming the generated discrete labels reflect the clinical semantics of the ECG signals and are useful for ECG analysis.

In conclusion, these findings demonstrate that the semantic tokenizer in ECG2TOK effectively capture consistent and clinically relevant patterns in ECG signals, validating the practicality of ECG2TOK for ECG analysis.

6 Conclusion

In this paper, we present ECG2TOK, a novel ECG pretraining framework that incorporates a self-distillation semantic tokenizer. Unlike conventional approaches that focus on timefrequency signal reconstruction, ECG2TOK leverages discrete label prediction to drive the model towards higher-level semantic abstraction. The key innovation of ECG2TOK lies in the self-distillation semantic tokenizer, which transforms continuous ECG signals into a compact semantic space and generates discrete labels for self-supervised learning. Our experimental results demonstrate that ECG2TOK surpasses state-of-the-art methods across six downstream classification tasks, achieving up to a 30.73% improvement in AUC under low-resource conditions. Furthermore, the self-distillation framework contributes to enhanced pretraining efficiency, accelerating the learning process without sacrificing performance. Visualization analysis further highlights that the semantic tokenizer produces discrete labels with consistent and meaningful semantics. These findings underscore the potential of self-supervised learning with discrete label prediction in overcoming challenges such as individual variability and redundancy in ECG analysis, providing innovative solutions to improve healthcare access and efficiency, particularly in resource-limited settings.

Acknowledgments

This work was supported in part by the Sustainable Development Science and Technology Project of Shenzhen Science and Technology Innovation Commission under Grant KCXFZ20201221173411032.