Contrastive Learning Is Not Optimal for Quasiperiodic Time Series

1 Introduction

With the rise of Self-Supervised Learning (SSL), there have been many efforts to adapt different works from other fields to time series processing, particularly physiological time series ([Diamant et al., 2022], [Wickstrøm et al., 2022]). In this field, being able to optimize a model without the need for labels is highly valuable given that: (i) the model learns generic representations that are useful across many tasks, (ii) the labeling is costly, or the specific task is not known a priori, and (iii) the model learns representations that are more robust across different perturbations occurred during the recording of the data. Existing methods drive the model to learn representations that are refined throughout the optimization process. The learned representations are capable of performing a downstream task better and better throughout the training process, as shown in Figure 1a, when the Machine Learning (ML) model considers a high number (8500) of recordings when is fitted.

Figure 1: Evolution of model performance across training procedures for best performing Contrastive Learning Method (blue) and proposed method (orange). While the downstream task displayed in Figure 1a considers multiple annotated recordings, the one shown in Figure 1b only considers a few of them. For an easier track of the evolution across the iterations, a polynomial has been fitted.

In certain other fields of Artificial Intelligence, existing SSL methods have shown promising results for a variety of applications by merely fitting a simple ML model on top of the representation using a limited amount of data [Caron et al., 2021], [Grill et al., 2020]. Achieving this level of performance is anticipated in a comparable time series analysis experiment, focusing on just a few records that encompass both normal and abnormal states. However, Figure 1b shows that the results of this experiment are far from what was expected. It can also be seen how the performance of the model decreases throughout the optimization process.

While each method has its own uniqueness, they all share the use of Contrastive Learning [Chen et al., 2020] as part of their optimization objective. This contrastive goal leads to the representations to be distant from the negative pairs which are drawn from distinct records. Quasiperiodic signals exhibit a combination of regular patterns with subtle variations. These quasiperiodic signals, such as electrocardiogram (ECG), exhibit much more visible differences between recordings from different subjects than between different classes within the same recording (See Figure 2).

Just by seeking similarities between positive pairs belonging to the same record and dissimilarities between different ones, Contrastive learning drives the model to capture unique record-based representations. However, in parallel, this contrastive objective leads the model to neglect the less visible, dynamic changes that occur across time. This clarifies why these methods struggle to guide the model in generalizing across various classes present in a recording when provided with a limited set of recordings.

In this paper, we present Distilled Embedding for AlmostPeriodic Time Series (DEAPS), a new SSL method designed for physiological time series. The underlying concept of this approach is based on the categorization of signal patterns into two types: (i) Static patterns that account for individual characteristics such as gender and age and, (ii) dynamic patterns that can reveal transitional states or events experienced by the subjects during the recording. DEAPS extends existing noncontrastive methods by not requiring the use of negative pairs during model optimization. This alleviates the model's blindness to dynamic patterns, while effectively continuing to encode the static ones.

In addition, avoiding the use of negative pairs allows DEAPS to introduce a novel loss function, "Gradual Loss (Lgra)", that encourages the model to put more emphasis on representing the dynamic patterns. This loss function operates by following these steps: (i) Three time strips belonging to the same recording are given as inputs, (ii) Assuming that the dynamic patterns exhibit a smooth and linear behavior through the recording, Lgra ensures that the representation of the middle input can be interpolated from the other two representations, and (iii) A selective optimization is incorporated for only considering the most important features (i.e., the features for which their value changes the most over the recording interval).

We hypothesize that (i) the use of Contrastive Learning techniques, by introducing an objective that prioritizes that the representations of different recordings are far apart, causes the model to neglect the dynamic patterns, (ii) Dynamic patterns are much less evident than static ones, so avoiding the use of negative pairs is not enough. An additional objective function is needed for the model to capture them, and (iii) By effectively capturing these dynamic patterns, the model will acquire a better generalization capability by just considering a few labeled records.

As a result, DEAPS is able to not only encode static patterns but also to distinguish different classes in the same recording by encoding the dynamic patterns too. Therefore, the model generalizes these classes to perform up to 10% better when evaluated on another dataset, compared to SOTA contrastive methods (Figure 1b). It also performs better when the ML model is fitted using multiple recordings (Figure 1a). In addition to these two experiments, DEAPS is evaluated against SOTA methods in two other experiments. Overall, the evaluation involves up to 3 different tasks and 3 different datasets different from the one used during the optimization process.

In summary, the contributions of this paper are:

1. It is demonstrated, both logically and empirically, that Contrastive Learning objectives lead the model to neglect subtle changes within quasiperiodic time series data.
2. We introduce DEAPS, a novel SSL method that stands apart from previous SOTA Contrastive Learning methods. Dispensing with the negative pairs not only alleviates this model blindness to shifts within the record but also allows us to incorporate the novel Lgra function. This function, which emphasizes understanding changes within these quasiperiodic data, leads the model to encode dynamic patterns.
3. We show that by understanding and encoding these dynamic patterns, the model is able to generalize different classes given a small sample of recordings where these two appear. This is exemplified by the up to 10% improvement when compared with existing methods.

2 RelatedWork

Unsupervised Learning in Time Series

Interpreting physiological signals demands a level of expertise, making the labeling of such data costly. It represents a bottleneck within the deep learning context, particularly given the substantial volume of data required for training deep learning models. Consequently, various studies have been directed towards optimizing the model using a limited amount of labels [Fan et al., 2021] or directly without relying on these labels ([Tonekaboni et al., 2021], [Franceschi et al., 2020]).

Contrastive Learning in Time Series

With the rise of SSL, different methods have recently been designed in the field of time series analysis, exploiting the characteristics of the time series data. Contrastive Learning of Cardiac Signals Across Space (CLOCS) [Kiyasseh et al., 2021] utilizes two consecutive ECG time strips as positive pairs. The mixing-up method [Wickstrøm et al., 2022] utilizes the temporal characteristics of the data for a more tailored data augmentation by creating a time series, which is a product of two time series from the same subject. An alternative approach is proposed in the Time-Frequency Consistency (TF-C) [Zhang et al., 2022] method, which produces two variations of the time-domain and frequency-domain pairs associated with the input signal. SSL method is developed to identify similarities inherent in these paired representations. The last example is Patient Contrastive Learning (PCLR) [Diamant et al., 2022] which considers two inputs from the same subject but different recordings as positive pairs.

Although each of these methods has its particularity in defining the positive pairs, they all share the use of the Contrastive Learning objective. PCLR obtains the best results in the benchmark used in this work. Accordingly, DEAPS considers the same selection of positive pairs criteria, i.e., between pairs from different recordings.

Non-Contrastive Learning

In other fields of artificial intelligence, such as Computer Vision, non-Contrastive Methods have emerged as an alternative to Contrastive Methods. The major challenge of these methods is to avoid the collapse mode in the representations. This collapse occurs when the model computes the same representations independently of the given input. Given that noncontrastive methods do not force representations of negative pairs to be distinct, other strategies have to be considered to avoid this issue.

Methods such as Bootstrap Your Own Latent (BYOL) [Grill et al., 2020] or Self-Distillation with no Labels (DINO) [Caron et al., 2021] rely on a teacher-student architecture. The cost functions are computed between the corresponding teacher-student. While the student network is optimized with Stochastic Gradient Descent (SGD), the weights of the teacher network are updated via exponential moving average (EMA) of the student weights. Other methods such as Variance-Invariance-Covariance Regularization (VIC-REG) [Bardes et al., 2022] maximize the batch variance of each feature within the representations. The method presented in this work utilizes BYOL framework as the baseline.

3 Distilled Embedding for Almost-Periodic Time Series (DEAPS)

DEAPS is designed to capture both the static and the dynamic patterns of the time series. It is illustrated in Figure 3. The model takes up to four different time strips belonging to two different records from the same subject. The encoder computes the representations of these distinct inputs. The static projector (labeled GS) takes a pair of representations from the two different recordings which are displayed in a static space. The similarity loss function (Lsim) is computed between these pairs of projections, ensuring that the unique patterns of each subject are represented consistently even between different records. The dynamic projector (labeled GD) processes the three representations from the same recording. DEAPS only considers the most important features for each triplet of inputs, i.e., the ones that are encoding dynamic patterns that are evolving between this time interval. The rest of the dynamic projection is masked. The unmasked projections are used for computing the gradual loss function (Lgra). The overall loss function proposed by DEAPS can be expressed as the following:

This section provides detailed explanations for each of its different components.

3.1 Non-Contrastive Learning. Revisiting BYOL

BYOL [Grill et al., 2020] was the first SSL non-contrastive learning method that dispensed with negative pairs. DEAPS utilizes it as baseline.

In other to avoid the mode collapse when the use of negative pairs is removed, BYOL features both a teacher network and a student network, each with an encoder and a projector initialized with identical parameters. The student network is additionally equipped with a predictor, which acts on the views computed by the projector. The optimization involves SGD for the student network, projector, and predictor. Both the teacher network and the teacher projector serve as an EMA of the student network. The loss function only measures the similarities between the student/teacher output pairs and it is described as the following:

where zs2 and q(ζ1)s are the teacher similarity projection and student similarity prediction, respectively, for inputs belonging to the first and second records.

Both the projector and the predictor are discarded after the optimization process. Therefore, the representations used for the downstream tasks are obtained directly from the student network. The student-teacher framework and the characteristic predictor are excluded from Figure 3 for clarity. In practice, the different loss functions are computed between projection/prediction pairs obtained by the teacher and student networks, denoted as z and q(ζ), respectively, where ζ represents the student projection.

3.2 Gradual Loss

Avoiding the use of a Contrastive objective alleviates the model's blindness to changes in the record. However, it is not sufficient to lead the model to capture dynamic patterns. Non-contrastive methods do not compare each representation with the rest of the representations in the batch. It allows us to incorporate a metric that compares just the representations within the same record.

Capturing the Dynamic Patterns

We introduce the "Gradual Loss (Lgra)" as a part of the training objective. This loss function just considers Xt-i, Xt, and Xt+j, three that belong to the same record and have been drawn over a fixed window size. DEAPS enforces the representation of the dynamic patterns to exhibit a linear behavior by mandating an accurate interpolation of the representation of an inter- mediate input from the representations of the other two. In other words, if a subject's state evolves fromXt-i to Xt+j, the representation of Xt, i.e., zt, should approximate an intermediate point between zt-i to zt+j . This objective ensures that the temporal evolution is properly captured within the representations. Lgra is described as:

where Pondered Average Representation (PAR) is the approximation of zt, drawn from zt-i and zt+j . We do not force zt to be equally distant from both zt-i and zt+j, therefore, we calculate PAR as,

Note that the incorporation of Lgra into the method is only possible since DEAPS is a non-contrastive method. Lgra only makes sense when taking into account only the zt and the interpolation of this representation given the zt-i and zt+j pairs, and not the rest of the representations in the batch considered as negative-pairs.

Intuitions of the Use of Two Projectors

Lsim and Lgra have conflicting goals. While the first one prioritizes that representations between positive pairs are invariant, the purpose of Lgra is encouraging changes within the recording to be captured in the representations. To facilitate the disentanglement between static and dynamic features, two projectors (represented as Gs and Gd in Figure 3) are incorporated into both the student and teacher network instead of the single projector of the original BYOL framework. In this way, only static features will be projected by Gs and thus taken into account by Lsim. Analogously, only dynamic features will be considered when computing Lgra.

Intuitions of the Windows Size

An essential consideration in implementing the method is determining the appropriate spacing window size between Xt-i and Xt+j, i.e., how much these inputs may be separated in time. This spacing window size must be large enough to accommodate signal changes, yet narrow enough just allow different patterns that encode distinct changes to move in one single direction each. If successive changes occur within this window, it can lead to conflicting directions in which these changes are reflected in the representations, thereby PAR(ztt-i , ztt+j ) may not be aligned with qs(zst ). The optimal value of this window size may vary depending on the quasiperiodic data modality. The effect of this size is studied in Section 5.

3.3 Selective Optimization

It is assumed that a set of features contained in the representation encodes the dynamic patterns of the time series. However, it is not reasonable to think that all these patterns will evolve in every fixed time window. To address this issue, DEAPS introduces a Selective Optimization. This ensures that only the features that encode changing patterns within a particular time window are considered when computing the loss function.

Masking the Output The process of determining which features of each dynamic projection to include in each loss function follows these steps: (i) It calculates the absolute difference between the values dynamic projections of each input pair belonging to the Xt-i and Xt+j batches, (ii) the N values that exhibit a larger difference are considered when computing Lgra, and (iii) any remaining features in each projection are masked and their gradients are not considered. This process is represented in Figure 3. The effect of this selective optimization is discussed in section 5.

Covariance Term To enhance the efficacy of our selective optimization approach, we strive to ensure that each feature within the representations encapsulates a distinctive and individual pattern. To achieve this objective, we incorporate the Covariance Loss function (Lc) as a regularization factor, effectively penalizing redundancy within the representations. This particular cost function, which has also found application in prior studies like Barlow Twins [Zbontar et al., 2021] and VIC-REG [Bardes et al., 2022], operates as follows:

where C(ζ) is the covariance matrix computed on the student projection, ζ. Its effect is studied in Section 5.

3.4 Implementation Details

In order to ensure the reproducibility of the method, this subsection details the configuration of the hyperparameters.

Model Architecture We use an adaptation of the Vision Transformer (ViT) [Dosovitskiy et al., 2021] model for processing physiological signals. The input data is a time series of 1000 samples, which correspond to 10 seconds-length signal sampled at 100Hz. This input is split into segments of a length of 20 samples. The model counts with 6 regular transformer blocks with 4 heads each. The model dimension is set to 128, for a total of 1,192,616 trainable parameters.

DEAPS Implementation The projectors and predictors in our approach are implemented as a two-layer Multilayer Perceptron (MLP) with a dimensionality of 512 and 256, respectively. Batch normalization and rectified linear unit (ReLU) operations are incorporated between the two layers of each structure. The EMA updating factor (τ) is set to 0.995. The window size is set to 2 minutes. We weigh the covariance loss with a factor of 0.1. We optimize the most important 32 features during the selective optimization. The effect of both the window size and the number of features to be considered during the selective optimization is discussed in Section 5.

Optimization The model is trained with 10 second-length signals belonging to the Sleep Heart Health Study (SHHS) dataset [Zhang et al., 2018], [Quan et al., 1998]. The training procedure consists of 30,000 iterations. We use a batch size of 256, and Adam [Kingma and Ba, 2017] with a learning rate of 3e-4 and a weight decay of 1.5e-6 as the optimizer. The training procedure and the evaluations are performed on a local computer, with a Nvidia GeForce RTX 3070 GPU.

4 Evaluation

The proposed model has undergone extensive simulation studies to evaluate its performance. Initially DEAPS was assessed by comparing it to four SOTA methods across three distinct downstream tasks, each using four different databases. These four databases are: MIT-BIH Atrial Fibrillation Database (MIT-AFIB)[Moody and Mark, 1983], MIT-BIH Arrhythmia Database (MIT-ARR), [Moody and Mark, 1983], Physionet Challenge 2017 (Cinc2017) [Clifford et al., 2017] and SHHS [Zhang et al., 2018]. All used databases are publicly available in Physionet [Goldberger et al., 2000] and National Sleep Research Resource (NSRR).

Furthermore, we conducted a Principal Component Analysis (PCA) analysis to substantiate the core concept of our approach, which revolves around disentangling static and dynamic patterns within time-series data with the aim of demonstrating DEAPS' capability to encode both of them.

4.1 Comparison against SOTAMethods

In order to evaluate the performance of DEAPS, three experiments of different nature have been carried out: (i) Atrial Fibrillation (AFib) identification to assess the effectiveness of generalising different classes within the same recording (ii) Gender Identification, to assess the effectiveness of static patterns and (iii) Physionet Challenge 2017 to assess the scalability of the representations when multiple recordings are given. DEAPS' performance has been compared against the four most relevant SOTA methods, namely; (i) PCLR [Diamant et al., 2022], (ii) CLOCS [Kiyasseh et al., 2021], (iii) Mixing-Up [Wickstrøm et al., 2022], and (iv) TF-C [Zhang et al., 2022]. Moreover, we have included the standard BYOL [Grill et al., 2020] framework as a baseline.

To ensure fairness in evaluation, we have optimized the same model used in this work, under the same configuration (optimizer, data, batch size, and number of iterations), except for the TF-C method, where their proposed model has been used, since it requires the use of two encoders instead of one. The TF-C model contains approximately 32 million parameters, which is 30x more parameters than the proposed DEAPS model. Therefore, it has been optimized over 75K iterations, instead of the 30K iterations proposed in this work.

Atrial Fibrillation (AFib) Identification

In order to assess the ability of the method to generalise different classes within the same record, we have conducted two experiments where there is no subject overlap between the training and validation sets. This overlapping would significantly simplify AFib identification. In the first experiment, we employed 10-second ECG time series strips originating from the eight subjects affected by AFib, belonging to the MIT-ARR database for fitting a Support Vector Classificatier (SVC) [Platt, 2000] on top of the representation. We subsequently evaluated this model on the complete MIT-AFIB database. In the second, we have conducted a Leave-OneOut (LOO) validation across the 23 MIT-AFIB subjects. The outcomes of this experiment are tabulated in Table 1. DEAPS demonstrates a significantly superior performance.

Gender Classification

We randomly selected 1549 ECG time series strips of length 10 seconds, each associated with a distinct subject, from the SHHS database. We conducted a five-fold cross-validation to evaluate the performance of the downstream tasks. The ML model used is the SVC and the outcomes of these experiments are presented in Table 2, where it can be seen how DEAPS obtains comparable results with the best performing method.

Physionet Challenge 2017

In this final experiment, we utilized the Cinc2017 database, which categorizes ECG time series strips as either Sinus Rhythm (SR), AFib, or Others. We used the dataset's predefined partitioning of "train" and "validation" sets for evaluating the SVC model fitted on top of the representations. The findings of this experiment are summarized in Table 3, wherein DEAPS achieves superior performance compared with the rest of the methods.

Figure 4: A PCA has been fitted on top of the MIT-AFIB representations. The values of the different components have been studied for disentangling the static and dynamic features.

4.2 PCAAnalysis on Representations

The fundamental concept underpinning this approach revolves around the integration of dissimilarities to facilitate the encoding of both static and dynamic features within the model. To ascertain the validity of this premise, a Principal Component Analysis (PCA) [Tipping and Bishop, 1999] is executed on the representations from MIT-AFIB database.

As depicted in Figure 4a where different patients are displayed in different colors, temporal segments originating from the same individual exhibit analogous positions in Principal Components 1 and 2, irrespective of the individual's status. Conversely, Principal Component 5 yields distinct values between AFib and normal rhythm (NR) classifications, regardless of the individual, as shown in Figure 4b. This second behavior is not seen in Contrastive Learning methods (See Appendix).

4.3 Discussion of the Results

This study has demonstrated that by following a noncontrastive methodology in addition to introducing a training objective (Lgra) to focus on the dynamic patterns, both static and dynamic natures of time series data are encoded by the model (Figure 4). It leads to a performing model, which can be used in a wide spectrum of downstream tasks, using different types of datasets (Tables 1, 2 and 3). Compared to related methods, DEAPS shows a superior performance.

These results show substantial support for the hypothesis: (i) the use of Contrastive Learning techniques, by introducing an objective that prioritizes that the representations of different recordings are far apart, causes the model to neglect the dynamic patterns, (ii) Dynamic patterns are much less evident than static ones, so avoiding the use of negative pairs is not enough. An additional objective function is needed to lead the model to capture them. (iii) By effectively capturing these dynamic patterns, the model will acquire a better generalization capability by just considering a few labeled records.

(a) MIT-ARR → MIT-AFIB

(b) LOO MIT-AFIB

Figure 5: Effect of incorporating different elements to the training procedure. Components inherited from other existing methods are labeled in red. The impact of adding each component is displayed by the difference in the performance of each downstream task.

5 Ablation Study

This ablation study evaluates the effect of incorporating the different parts of the proposed method. In addition, the effect of the effect of the configuration of the two hyperparameters of the method i.e. window size and number of features that are considered in the Selective Optimization.

Role of the Different Components

Starting with contrastive methods, DEAPS changes this contrastive goal to the student-teacher architecture proposed by BYOL work. In addition, it incorporates the covariance term from methods such as VIC-REG. In Figure 5 it can be seen how these two components, which are not part of the contributions of this study, slightly improve these Contrastive Methods. In addition, it shows how the incorporation of the gradual loss, together with the selective optimization, plays a decisive role in obtaining the results with which DEAPS significantly improves existing SOTA methods.

Role of the Different Hyperparameters

As it is described in Section 3.2 The size of the spacing window must be sufficiently broad to capture signal variations but sufficiently limited to ensure that the direction of these variations is constant. Table 4 shows the impact of this window size. Although the proposed configurations improve existing methods, the best performance level is achieved by considering inputs belonging to a two-minute window size.

The last aspect to take into consideration is the number of features that are taken into account in each iteration, as described in Section 3.3. Empirically we have determined that they should be 32. Table 5b shows the results of different configurations, where it can be seen that the proposal obtains the best performance. It can be seen that all configurations achieve better performance than existing methods.

6 Contributions

This study presents compelling evidence elucidating the tendency of Contrastive Learning objectives to induce neglect of subtle changes within quasiperiodic time series data. Addressing this limitation, we introduce DEAPS as a novel SSL method, distinct from conventional SOTA Contrastive Learning approaches. By dispensing with negative pairs, our method mitigates the model's susceptibility to capture shifts and integrates the novel Lgra function. This function guides the model to effectively encode dynamic patterns. Consequently, the model's capacity to generalize across different classes is demonstrated. The practical impact of this advancement is highlighted through a notable improvement of up to 10% compared to existing methods.

We believe that this characterization of physiological signals in static and dynamic patterns as well as the incorporation of a specific training objective for dynamic ones will provide inspiration for future Time Series SSL methods.

Limitations

Only one database (SHHS) has been used to pre-train the model. Using time strips from different recordings for the computation of Lsim is found to be the best strategy. SHHS is the only database we have found with multiple recordings from the same subject. However, we believe we have mitigated this lack of distinct datasets for pre-training by involving multiple datasets in the evaluation.