Time-Series Representation Learning via Temporal and Contextual Contrasting

1 Introduction

Time-series data are being incrementally collected on daily basis from IoT and wearable devices for various applications in healthcare, manufacturing, etc. However, they generally do not have human recognizable patterns and require specialists for annotation/labeling. Therefore, it is much harder to label time-series data than images, and little time-series data have been labeled in real-world applications [Ching et al., 2018]. Given that deep learning methods usually require a massive amount of labeled data for training, it is thus very challenging to apply them on time-series data with these labeling limitations.

∗Corresponding Author

Self-supervised learning gained more attention recently to extract effective representations from unlabeled data for downstream tasks. Compared with models trained on full labeled data (i.e., supervised models), self-supervised pretrained models can achieve comparable performance with limited labeled data [Chen et al., 2020]. Various selfsupervised approaches relied on different pretext tasks to train the models and learn representations from unlabeled data, such as solving puzzles [Noroozi and Favaro, 2016] and predicting image rotation [Gidaris et al., 2018]. However, the pretext tasks can limit the generality of the learned representations. For example, classifying the different rotation angles of an image may deviate the model from learning features about the color or orientation of objects [Oord et al., 2018].

Contrastive learning has recently shown its strong ability for self-supervised representation learning in computer vision domain because of its ability to learn invariant representation from augmented data [Hjelm et al., 2019; He et al., 2020; Chen et al., 2020]. It explores different views of the input images by first applying data augmentation techniques and then learns the representations by maximizing the similarity of different views from the same sample and minimizing the similarity with the views from different samples. However, these image-based contrastive learning methods are not able to work well on time-series data for the following reasons. First, they may not be able to address the temporal dependencies of data, which are key characteristics of time-series [Franceschi et al., 2019]. Second, some augmentation techniques used for images such as color distortion, generally cannot fit well with time-series data. So far, few works on contrastive learning have been proposed for time-series data. For example, [Mohsenvand et al., 2020; Cheng et al., 2020] developed contrastive learning methods for bio-signals such as EEG and ECG. However, the above two methods are proposed for specific applications and they are not generalizable to other time-series data.

To address the above issues, we propose a Time-Series representation learning framework via Temporal and Contextual Contrasting (TS-TCC). Our framework employs simple yet efficient data augmentations that can fit any time-series data to create two different, but correlated views of the input data. Next, we propose a novel temporal contrasting module to learn robust representations by designing a tough cross-view prediction task, which for a certain timestep, it utilizes the past latent features of one augmentation to predict the future of another augmentation. This novel operation will force the model to learn robust representation by a harder prediction task against any perturbations introduced by different timesteps and augmentations. Furthermore, we propose a contextual contrasting module in TS-TCC to further learn discriminative representations upon the robust representations learned by the temporal contrasting module. In this contextual contrasting module, we aim to maximize the similarity among different contexts of the same sample while minimizing similarity among contexts of different samples.

In summary, the main contributions of this work are as follows.

• A novel contrastive learning framework is proposed for unsupervised time-series representation learning.
• Simple yet efficient augmentations are designed for time-series data in the contrastive learning framework.
• We propose a novel temporal contrasting module to learn robust representations from time series data by designing a tough cross-view prediction task. In addition, we propose a contextual contrasting module to further learn discriminative representations upon the robust representations.
• We perform extensive experiments on our proposed TSTCC framework using three datasets. Experimental results show that the learned representations are effective for downstream tasks under supervised learning, semisupervised learning and transfer learning settings.

2 RelatedWorks

2.1 Self-supervised Learning

The recent advances in self-supervised learning started with applying pretext tasks on images to learn useful representations, such as solving jigsaw puzzles [Noroozi and Favaro, 2016], image colorization [Zhang et al., 2016] and predicting image rotation [Gidaris et al., 2018]. Despite the good results achieved by these pretext tasks, they relied on heuristics that might limit the generality of the learned representations. On the other hand, contrastive methods started to shine via learning invariant representations from augmented data. For instance, MoCo [He et al., 2020] utilized a momentum encoder to learn representations of negative pairs obtained from a memory bank. SimCLR [Chen et al., 2020] replaced the momentum encoder by using a larger batch of negative pairs. Also, BYOL [Grill et al., 2020] learned representations by bootstrapping representations even without using negative samples. Last, SimSiam [Chen and He, 2020] supported the idea of neglecting the negative samples, and relied only on a Siamese network and stop-gradient operation to achieve the state-of-the-art performance. While all these approaches have successfully improved representation learning for visual data, they may not work well on time series data that have different properties, such as temporal dependency.

2.2 Self-supervised Learning for Time-Series

Representation learning for time series is becoming increasingly popular. Some approaches employed pretext tasks for time series data. For example, [Saeed et al., 2019] designed a binary classification pretext task for human activity recognition by applying several transformations on the data, and trained the model to classify between the original and the transformed versions. Similarly, SSL-ECG approach [P. Sarkar, 2020] learned ECG representations by applying six transformations to the dataset, and assigned pseudo labels according to the transformation type. Additionally, [Aggarwal et al., 2019] learned subject-invariant representations by modeling local and global activity patterns. Inspired by the success of contrastive learning, few works have recently leveraged contrastive learning for time series data. For example, CPC [Oord et al., 2018] learned representations by predicting the future in the latent space and showed great advances in various speech recognition tasks. Also, [Mohsenvand et al., 2020] designed EEG related augmentations and extended SimCLR model [Chen et al., 2020] to EEG data. Existing approaches used either temporal or global features. Differently, we first construct different views for input data by designing time-series specific augmentations. Additionally, we propose a novel cross-view temporal and contextual contrasting modules to improve the learned representations for time-series data.

3 Methods

This section describes our proposed TS-TCC in details. As shown in Figure 1, we first generate two different yet correlated views of the input data based on strong and weak augmentations. Then, a temporal contrasting module is proposed to explore the temporal features of the data with an autoregressive model. These models perform a tough cross-view prediction task by predicting the future of one view using the past of the other. We further maximize the agreement between the contexts of the autoregressive models by a contextual contrasting module. Next, we will introduce each component in the following subsections.

3.1 Time-Series Data Augmentation

Data augmentation is a key part in the success of the contrastive learning methods [Chen et al., 2020; Grill et al., 2020]. Contrastive methods try to maximize the similarity among different views of the same sample, while minimizing its similarity with other samples. It is thus important to design proper data augmentations for contrastive learning [Chen et al., 2020; Mohsenvand et al., 2020]. Usually, contrastive learning methods use two (random) variants of the same augmentation. Given a sample x, they produce two views x1 and x2 sampled from the same augmentations family T , i.e., x1 ∼ T and x2 ∼ T . However, we argue that using different augmentations can improve the robustness of the learned representations. Consequently, we propose applying two separate augmentations, such that one augmentation is weak and the other is strong. In this paper, weak augmentation is a jitter-and-scale strategy. Specifically, we add random variations to the signal and scale up its magnitude. For strong augmentation, we apply permutation-and-jitter strategy, where permutation includes splitting the signal into a random number of segments with a maximum of M and randomly shuffling them. Next, a random jittering is added to the permuted signal. Notably, the augmentation hyperparameters should be chosen carefully according to the nature of the time-series data. For example, the value of M in a time-series data with longer sequences should be greater than its value in those with shorter sequences when applying permutation. Similarly, the jittering ratio for normalized time-series data should be much less than the ratio for unnormalized data. Encoder Features

For each input sample x, we denote its strongly augmented view as xs , and its weakly augmented view as xw , where xs ∼Ts and xw ∼Tw. These views are then passed to the encoder to extract their high dimensional latent representations. In particular, the encoder has a 3-block convolutional architecture as proposed in [Wang et al., 2017]. For an input x, the encoder maps x into a high-dimensional latent representation z = fenc(x). We define z = [z1, z2, . . . zT ], where T is the total timesteps, zi ∈ Rd , where d is the feature length. Thus, we get zs for the strong augmented views, and zw for the weak augmented views, which are then fed into the temporal contrasting module.

3.2 Temporal Contrasting

The Temporal Contrasting module deploys a contrastive loss to extract temporal features in the latent space with an autoregressive model. Given the latent representations z, the autoregressive model far summarizes all z≤t into a context vector ct = far(z≤t), ct ∈ Rh , where h is the hidden dimension of far. The context vector ct is then used to predict the timesteps from zt+1 until zt+k (1 < k ≤ K). To predict future timesteps, we use log-bilinear model that would preserve the mutual information between the input xt+k and ct, such that fk(xt+k, ct) = exp((Wk(ct))T zt+k), where Wk is a linear function that maps ct back into the same dimension as z, i.e. Wk : Rh→d .

In our approach, the strong augmentation generates cst and the weak augmentation generates cwt . We propose a tough cross-view prediction task by using the context of the strong augmentation cst to predict the future timesteps of the weak augmentation zw t+k and vice versa. The contrastive loss tries to minimize the dot product between the predicted representation and the true one of the same sample, while maximizing the dot product with the other samples Nt,k within the minibatch. Accordingly, we calculate the two losses LsTC and LwTC as follows:

We use Transformer as the autoregressive model because of its efficiency and speed [Vaswani et al., 2017]. The architecture of the Transformer model is shown in Figure 2. It mainly consists of successive blocks of multi-headed attention (MHA) followed by an MLP block. The MLP block is composed of two fully-connected layers with a non-linearity ReLU function and dropout in between. Pre-norm residual connections, which can produce more stable gradients [Wang et al., 2019], are adopted in our Transformer. We stack L identical layers to generate the final features. Inspired by BERT model [Devlin et al., 2019], we add a token c ∈ Rh to the input whose state acts as a representative context vector in the output. The operation of the Transformer starts by applying the features z≤t to a linear projection WTran layer that maps the features into the hidden dimension, i.e. WTran : Rd→h . The output of this linear projection is then sent to the Transformer i.e. ˜ z = WTran(z≤t), ˜z ∈ Rh . Next, we attach the context vector into the features vector ˜z such that the input features become ψ0 = [c; ˜z], where the subscript 0 denotes being the input to the first layer. Next, we pass ψ0 through Transformer layers as in the following equations:

Finally, we re-attach the context vector from the final output such that ct = ψ0L. This context vector will be the input of the following contextual contrasting module.

3.3 Contextual Contrasting

We further propose a contextual contrasting module that aims to learn more discriminative representations. It starts with applying a non-linear transformation to the contexts using a non-linear projection head as in [Chen et al., 2020]. The projection head maps the contexts into the space of where the contextual contrasting is applied.

Given a batch of N input samples, we will have two contexts for each sample from its two augmented views, and thus have 2N contexts. For a context cit , we denote ci+ t as the positive sample of cit that comes from the other augmented view of the same input, and hence, (cit , ci+ t ) are considered to be a positive pair. Meanwhile, the remaining (2N - 2) contexts from other inputs within the same batch are considered as the negative samples of cit , i.e., cit can form (2N - 2) negative pairs with its negative samples. Therefore, we can derive a contextual contrasting loss to maximize the similarity between the positive pair and minimizing the similarity between negative pairs. As such, the final representations can be discriminative.

Eq. 5 defines the contextual contrasting loss function LCC. Given a context cit , we divide its similarity with its positive sample ci+ t by its similarity with all the other (2N - 1) samples, including the positive pair and (2N - 2) negative pairs, to normalize the loss.

where sim(u, v) = uT v/‖u‖‖v‖ denotes the dot product between ℓ2 normalized u and v (i.e., cosine similarity), 1[m̸=i] ∈ {0, 1} is an indicator function, evaluating to 1 iff m ̸= i, and τ is a temperature parameter.

The overall self-supervised loss is the combination of the two temporal contrasting losses and the contextual contrasting loss as follows.

where λ1 and λ2 are fixed scalar hyperparameters denoting the relative weight of each loss.

4 Experimental Setup

4.1 Datasets

To evaluate our model, we adopted three publicly available datasets for human activity recognition, sleep stage classification and epileptic seizure prediction, respectively. Additionally, we investigated the transferability of our learned features on a fault diagnosis dataset.

Human Activity Recognition (HAR)

We use UCI HAR dataset [Anguita et al., 2013] which contains sensor readings for 30 subjects performing 6 activities (i.e. walking, walking upstairs, downstairs, standing, sitting, and lying down). They collected the data using a mounted Samsung Galaxy S2 device on their waist, with a sampling rate of 50 Hz.

Sleep Stage Classification

In this problem, we aim to classify the input EEG signal into one of five classes: Wake (W), Non-rapid eye movement (N1, N2, N3) and Rapid Eye Movement (REM). We downloaded Sleep-EDF dataset from the PhysioBank [Goldberger et al., 2000]. Sleep-EDF includes whole-night PSG sleep recordings, where we used a single EEG channel (i.e., Fpz-Cz) with a sampling rate of 100 Hz, following previous studies [Eldele et al., 2021].

Epilepsy Seizure Prediction

The Epileptic Seizure Recognition dataset [Andrzejak et al., 2001] consists of EEG recordings from 500 subjects, where the brain activity was recorded for each subject for 23.6 seconds. Note that the original dataset is labeled with five classes. As four of them do not include epileptic seizure, so we merged them into one class and treat it as a binary classification problem.

Fault Diagnosis (FD)

We conducted the transferability experiment on a real-world fault diagnosis dataset [Lessmeier et al., 2016]. This dataset was collected under four different working conditions. Each working condition can be considered as a separate domain as it has different characteristics from the other working conditions [Ragab et al., 2020]. Each domain has three classes, namely, two fault classes (i.e., inner fault and outer fault) and one healthy class.

Table 1 summarizes the details of each dataset, e.g., the number of training samples (# Train) and testing samples (# Test), the length of the sample, the number of sensor channels (# Channel) and the number of classes (# Class).

4.2 Implementation Details

We split the data into 60%, 20%, 20% for training, validation and testing, with considering subject-wise split for Sleep-EDF dataset to avoid overfitting. Experiments were repeated for 5 times with 5 different seeds, and we reported the mean and standard deviation. The pretraining and downstream tasks were done for 40 epochs, as we noticed that the performance does not improve with further training. We applied a batch size of 128 (which was reduced to 32 in few-labeled data experiments as data size may be less than 128). We used Adam optimizer with a learning rate of 3e-4, weight decay of 3e-4, β1 = 0.9, and β2 = 0.99. For the strong augmentation, we set MHAR = 10, MEp = 12 and MEDF = 20, while for the weak augmentation, we set the scaling ratio to 2 for all the datasets. We set λ1 = 1, while we achieved good performance when λ2 ≈ 1. Particularly, we set it as 0.7 in our experiments on the four datasets. In the Transformer, we set the L = 4, and the number of heads as 4. We tuned h ∈ {32, 50, 64, 100, 128, 200, 256} and set hHAR,Ep = 100, hEDF = 64. We also set its dropout to 0.1. In contextual contrasting, we set τ = 0.2. Lastly, we built our model using PyTorch 1.7 and trained it on a NVIDIA GeForce RTX 2080 Ti GPU.

5 Results

To show the efficacy of our proposed TS-TCC, we test it on three different training settings, including linear evaluation, semi-supervised training and transfer learning. We evaluate the performance using two metrics namely the accuracy and the macro-averaged F1-score (MF1) to better evaluate the imbalanced datasets.

5.1 Comparison with Baseline Approaches

We compare our proposed approach against the following baselines. (1) Random Initialization: training a linear classifier on top of randomly initialized encoder; (2) Supervised: supervised training of both encoder and classifier model; (3) SSL-ECG [P. Sarkar, 2020]; (4) CPC [Oord et al., 2018]; (5) SimCLR [Chen et al., 2020]. It is worth noting that, we use time-series specific augmentations to adapt SimCLR to our application as it was originally designed for images.

To evaluate the performance of our TS-TCC model, we follow the standard linear benchmarking evaluation scheme [Oord et al., 2018; Chen et al., 2020]. Particularly, we train a linear classifier (single MLP layer) on top of a frozen self-supervised pretrained encoder model. Table 2 shows the linear evaluation results of our approach against the baseline methods. Overall, our proposed TS-TCC outperforms all the three state-of-the-art methods. Furthermore, TS-TCC, with only linear classifier, performs best on two out of three datasets while achieving comparable performance to the supervised approach on the third dataset. This demonstrates the powerful representation learning capability of our TS-TCC model. Notably, contrastive methods (e.g., CPC, SimCLR and our TS-TCC) generally achieve better results than the pretext-based method (i.e., SSL-ECG), which reflects the power of invariant features learned by contrastive methods. Additionally, CPC method shows better results than SimCLR, indicating that temporal features are more important than general features in time-series data.

5.2 Semi-supervised Training

We investigate the effectiveness of our TS-TCC under the semi-supervised settings, by training the model with 1%, 5%, 10%, 50%, and 75% of randomly selected instances of the training data. Figure 3 shows the results of our TS-TCC along with the supervised training under the aforementioned settings. In particular, TS-TCC fine-tuning (i.e., red curves in Figure 3) means that we fine-tuned the pretrained encoder with few labeled samples.

We observe that supervised training performs poorly with limited labeled data, while our TS-TCC fine-tuning achieves significantly better performance than supervised training with only 1% of labeled data. For example, TS-TCC fine-tuning can still achieve around 70% and 90% for HAR and Epilepsy datasets respectively. Furthermore, our TS-TCC fine-tuning with only 10% of labeled data can achieve comparable performance with the supervised training with 100% of labeled data in the three datasets, demonstrating the effectiveness of our TS-TCC method under the semi-supervised setting.

5.3 Transfer Learning Experiment

We further examine the transferability of the learned features by designing a transfer learning experiment. We use Fault Diagnosis (FD) dataset introduced in Table 1 for the evaluation under the transfer learning setting. Here, we train the model on one condition (i.e., source domain) and test it on another condition (i.e., target domain). In particular, we adopt two training schemes on the source domain, namely, (1) supervised training and (2) TS-TCC fine-tuning where we fine-tuned our pretrained encoder using the labeled data in the source domain.

Table 3 shows the performance of the two training schemes under 12 cross-domain scenarios. Clearly, our pretrained TSTCC model with fine-tuning (FT) consistently outperforms the supervised pretraining in 8 out of 12 cross-domain scenarios. TS-TCC model can achieve at least 7% improvement in 7 out of 8 winning scenarios (except for D→B scenario). Overall, our proposed approach can improve the transferability of learned representations over the supervised training by about 4% in terms of accuracy.

5.4 Ablation Study

We study the effectiveness of each component in our proposed TS-TCC model. Specifically, we derive different model variants for comparison as follows. First, we train the Temporal Contrasting module (TC) without the cross-view prediction task, where each branch predicts the future timesteps of the same augmented view. This variant is denoted as 'TC only'. Second, we train the TC module with adding the cross-view prediction task, which is denoted as 'TC + XAug'. Third, we train the whole proposed TS-TCC model, which is denoted as 'TC + X-Aug + CC'. We also study the effect of using a single augmentation in TS-TCC. In particular, for an input x, we generate two different views x1 and x2 from the same augmentation type, i.e., x1 ∼Tw and x2 ∼Tw when using the weak augmentation.

Table 4 shows this ablation study on the three datasets. Clearly, the proposed cross-view prediction task generates robust features and thus improves the performance by more than 5% on HAR datasets, and ∼1% on Sleep-EDF and Epilepsy datasets. Additionally, the contextual contrasting module further improves the performance, as it helps the features to be more discriminative. Studying the augmentations effect, we find that generating different views from the same augmentation type is not helpful with HAR and Sleep-EDF datasets.

On the other hand, Epilepsy dataset can achieve comparable performance with only one augmentation. Overall, our proposed TS-TCC method using both types of augmentations achieves the best performance.

5.5 Sensitivity Analysis

We perform sensitivity analysis on HAR dataset to study three parameters namely, the number of predicted future timesteps K in the temporal contrasting module, besides λ1 and λ2 in Eq. 5.

Figure 4a shows the effect of K on the overall performance, where x-axis is the percentage K/d, d is the length of the features. Clearly, increasing the percentage of the predicted future timesteps improves the performance. However, larger percentages can harm the performance as it reduces the amount of past data used for training the autoregressive model. We observe that predicting 40% of the total feature length performs the best, and thus we set K as d×40% in our experiments. Figures 4b and 4c show the results of varying λ1 and λ2 in a range between 0.001 and 1000 respectively. We fix λ1 = 1 and change the values of λ2 in Figure 4c. We observe that our model achieves good performance when λ2 ≈ 1, where the model performs best with λ2 = 0.7. Consequently, we fix λ2 = 0.7 and tune the value of λ1 as in Figure 4b, where we find that our model achieves the best performance when λ1 = 1. We also find that as λ1 < 10, our model is less sensitive to its value, while it is more sensitive to different values of λ2.

6 Conclusions

We propose a novel framework called TS-TCC for unsupervised representation learning from time-series data. The proposed TS-TCC framework first creates two views for each sample by applying strong and weak augmentations. Then the temporal contrasting module learns robust temporal features by applying a tough cross-view prediction task. We further propose a contextual contrasting module to learn discriminative features upon the learned robust representations. The experiments show that a linear classifier trained on top the features learned by our TS-TCC performs comparably with supervised training. In addition, our proposed TS-TCC shows high efficiency on few-labeled data and transfer learning scenarios, e.g., our TS-TCC by using only 10% of the labeled data can achieve close performance to the supervised training with full labeled data.

(b)

(c)

Figure 4: Three sensitivity analysis experiments on HAR dataset.

and Cuntai Guan. An attention-based deep learning approach for sleep stage classification with single-channel eeg. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021.

[Franceschi et al., 2019] Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In NeurIPS, 2019.

[Gidaris et al., 2018] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICML, 2018.

[Goldberger et al., 2000] Ary L Goldberger, Luis AN Amaral, Leon Glass, and et al. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation, 101(23), 2000.

[Grill et al., 2020] Jean-Bastien Grill, Florian Strub, and Florent Altch´e et al. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020.

[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.

[Hjelm et al., 2019] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICML, 2019.

[Lessmeier et al., 2016] Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In European conference of the prognostics and health management society, 2016.

[Mohsenvand et al., 2020] Mostafa Neo Mohsenvand, Mohammad Rasool Izadi, and Pattie Maes. Contrastive representation learning for electroencephalogram classification. In Machine Learning for Health NeurIPS Workshop, 2020.

[Noroozi and Favaro, 2016] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.

[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.

Acknowledgements

This research is supported by the Agency for Science, Technology and Research (A*STAR) under its AME Programmatic Funds (Grant No. A20H6b0151) and Career Development Award (Grant No. C210112046).