Adversarial Spectral Kernel Matching for Unsupervised Time Series Domain Adaptation

1 Introduction

Unsupervised domain adaptation (UDA) is a machine learning strategy that transfers knowledge from one domain to another [Pan and Yang, 2009]. The unsupervised adaptation progress utilizes labeled data from the source domain but unlabeled data from the target domain to train the model. Many successful cases have been achieved by methods in traditional research fields as computer vision [Long et al., 2015] and text categorization [Dai et al., 2009]. These methods usually learn a domain-invariant space by minimizing discrepancy distance computed by a certain metric between source and target domains. Maximum Mean Discrepancy (MMD) metric [Gretton et al., 2007] is the which utilizes a kernel function to extract statistics from samples to the Reproducing Kernel Hilbert Space (RKHS) and measures the domain discrepancy on RKHS. For example, DDC [Tzeng et al., 2014] applies MMD with a linear kernel which captures first-order statistics (Mean). DAN [Long et al., 2015] enhances MMD metric with multiple Gaussian kernels. Besides, HoMM [Chao et al., 2020] proposed an innovate high-order metric, which approximates to MMD with third or fourth order polynomial kernel. UDA methods with MMDs embedded by distinct kernels have several advantages over methods with classical metrics, namely easy convergence and low bias of finite sample estimates. However, when met with the unsupervised time series domain adaptation (UTSDA) task, a challenging and emerging area of UDA, these methods cannot be robustly applied to.

∗Contact Author

For the real-world time series, the data usually change through time (i.e., non-stationary) and continuously reverse derivatives (i.e., non-monotonic). Based on Fourier spectral theory, these non-stationary and non-monotonic statistics are non-local in the energy-frequency-time distributions [Clements and Hendry, 2000]. However, existing methods based on MMD metrics only capture the low-order or local statistics of distributions on structured objects such as images, instead of appropriate statistics for time series, leading to rough-grained domain matching. The linear-kernel MMD in DDC capturing low-order statistics degrades to Euclidean distance metric. The MMD with multiple Gaussian kernel in DAN reveals local statistics, i.e., only capturing the identical similarity of structured data [Xue et al., 2019]. Even the high-order MMD in HoMM is only suitable for traditional images with translation invariance and stills suffers from locality. Visual proof of the shortcomings of these local statistics are shown in Fig.1, where two time series distributions with same mean are given in Fig.1(a). In Fig.1(b), the unsuitable statistics maps two distributions into the same point in the measure space and leads to incorrect distance metric. But the appropriate statistics in Fig.1(c) can capture sufficient information of distributions and make method effectively distinguish between domains.

To solve the above problems, we propose a Adversarial Spectral Kernel Matching (AdvSKM) method for UTSDA. Our method perform time series domain matching by minimizing a reinforced MMD metric embedded by a welldesigned hybrid spectral kernel network. The hybrid spectral kernel network interpretably integrates spectral kernel and arc-cosine kernel into the deep architecture via spectral theorems and kernel approximation, which exposes nonstationary and non-monotonic properties through compact representation of complex time series and boosts computational efficiency. Adopting the hybrid spectral kernel network as inner kernel for MMD not only facilitates time series feature mapping to RKHS, but also reveals non-local properties in time series distributions, and therefore ensures injective embedding to RKHS and guarantees fine-grained domain matching. Besides, the differentiable architecture of the spectral kernel network enables adversarial kernel learning, thus bringing more discriminatory expression for discrepancy matching. Extensive experimental results demonstrate the superiority of our AdvSKM to several state-of-the-art adaptation methods in UTSDA task.

2 RelatedWorks

Several researches have been devoted to time series classification (TSC) problems. Wang et al. [2017] designed a Fully Convolutional Neural (FCN) Network for the TSC problem. The FCN consists of three convolutional layers without any subsampling, followed by a global average pooling layer, and the last layer is a softmax classifier. Although recurrent neural networks (RNN) propagating through time are intuitively more suitable for TSC problems, experimental results in [Bai et al., 2018] show that the simple FCN outperforms canonical RNNs such as LSTMs across a diverse range of tasks. However, the performance of backbone FCN trained on specific domain decreases abruptly when applied to the other domain. This problem can be solved with transfer learning strategy.

Unsupervised domain adaptation (UDA) is a rising field in transfer learning since it does not require labels in target domains. Most UDA methods generally learn a domaininvariant space by minimizing domain discrepancy between domains. These discrepancy-based methods can be categorized into two groups: (1) Metric-based: Learning domaininvariant features between domains by reducing the discrepancy distance computed by a certain metric. (2) Domaindiscrimination-based: Blending domains with a domain adversarial loss, specifically by training discriminators to distinguish between source and target domains.

Most of the metric-based methods are based on Maximum Mean Discrepancy (MMD) metric. MMD has several advantages over classical metrics, since it compares two distributions without density estimation. Specifically, Tzeng et al. [2014] proposed DDC which matches one-order statistic at last layer of the model with linear-kernel MMD. Further, Long et al. [2015] presented DAN which performs domain matching at multiple layers via multiple Gaussian kernel MMD. Sun and Saenko [2016] proposed Correlation alignment (CORAL) which is equivalent to MMD with second-order polynomial kernel. Besides, Central Moment Discrepancy (CMD) [Zellinger et al., 2017] and Higherorder Moment Matching (HoMM) [Chao et al., 2020] further considered higher-order metrics, which both approximate to the MMD using third or fourth order polynomial kernel. However, these methods capture low-order or local statistics that are only suitable for structured data instead of non-local statistics for time series, leading to rough-grained domain matching in unsupervised time series domain adaptation (UTSDA) task.

The second category of domain-discrimination-based approaches such as Domain-adversarial Neural Network (DANN) [Ganin et al., 2016] attempt to minimize domain discrepancy by distinguishing domains. However, the backbone architectures in these methods are designed for image data and is not adapted to the time series data. Therefore, they cannot be applied robustly to UTSDA.

Although less numerous, a few methods have been introduced to UTSDA. These methods combine different TSC models as backbones with an domain adversary following DANN. Specifically, variational recurrent adversarial deep domain adaptation (VRADA) [Purushotham et al., 2017] and recurrent domain adversarial neural network (R-DANN) explore long short-term memory (LSTM) network and variational RNN as backbone, respectively. While Convolutional Domain Adaptation model for Time Series data (CoDATS) [Wilson et al., 2020] utilizes FCN as backbone network. However, in terms of domain matching methodology, these UTSDA methods all follow domain-discrimination rule, where domain and category classification losses are equally important and a slight imbalance between them can cause inter-class confusion and leads to accuracy degradation.

Unlike the UTSDA methods above, our approach minimize the domain divergence directly on a special MMD metric for time series data. The special MMD metric is embedded by a well-designed hybrid spectral kernel network as inner kernel. The hybrid spectral kernel network can precisely characterize non-stationary and non-monotonic statistics instead of local statistics in time series distributions. Embedding the hybrid spectral kernel network to MMD guarantees precise discrepancy metric and benefits domain matching in UTSDA.

3 Our Method

Our method aim to solve the unsupervised domain adaptation task where a significant amount of labeled source instances and several unlabeled target instances are given. We first denote the instances and labels from the source and target domains with XS = [xs1 , . . . , xsns ]⊤ ∈ Rds×ns ,YS = [ys1 , . . . , ysns ] ∈ Rns×1 and XT = [xt1 , . . . , xtnt ]⊤ ∈ Rdt×nt ,YT = [yt1 , . . . , ytnt ] ∈ Rnt×1 , respectively. The source domain provides NS labeled instances and the target domain data contains NT unlabeled instances. In this paper, we focus on the multivariate time series classification task. Time series include both univariate and multivariate data. If univariate, the sample x = [x1, x2, . . . , xH] consists of H real values through time. If multivariate, the sample x = [x1 , x2 , . . . , xK ] contains K univariate time series, each of which consists of H real values through time.

Given source and target samples, we aim to train a crossdomain FCN backbone network φ(·) and a classifier f(·) which together map cross-domains samples to a unified space and identify unlabeled target samples on the new space. φ(x) denotes the outputs of the deep FCN and f(φ(x)) denotes the predicted labels from the classifier. Similar to the experimental framework in [Long et al., 2015], a two-stream FCN framework is adopted as in Fig.2. We aim to minimize domain discrepancy at the last global pooling layer. The framework is trained under a basic domain adaptation loss including source domain classification loss and domain discrepancy loss:

where Lc is the classification loss for source domain, Ld is the domain discrepancy loss, and λ is the factor. Specifically, the classification loss function is defined as (2) and J is the cross-entropy function.

3.1 MaximumMean DiscrepancyMatching

The two-stream FCN with shared parameters φ(·) transform instances from two distinct domains into the unified space. The new representations of the two domains on the new space often obey distinct distributions. We utilized kernel-based MMD metric as a loss to minimize this divergence.

And to go deeper, MMD compares two distributions by embedding each distribution in to RKHS H with a kernel function k. When given two distributions PX,QX and the MMD can be expressed as:

The kernel function serves to extract statistics from samples to RKHS. However, when met with time series data, the commonly used shallow kernels cannot effectively capture appropriate statistics for MMD. For instance, MMD with a linear kernel captures only one-order statistics. MMD with Gaussian kernel cannot reveal non-local statistics except for the identical similarity of structured data [Xue et al., 2019]. Besides, these shallow kernels are also restricted by the computational inefficiency as the learning problems become more complex.

3.2 Hybrid Spectral Kernel Network

To overcome the shortcoming of these kernels, we design a deep hybrid spectral kernel network specifically for time series data. We first interpretably integrate non-stationary and non-monotonic spectral kernel into the deep architectures via several spectral theorems [Xue et al., 2019]. Based on the formula of deep spectral kernel network, the spectroscopic kernel function is represented by the spectral kernel mapping:

where (4) is the spectral kernel mapping. Through this spectral kernel mapping, the spectroscopic kernel function can be represented as (5) and Ω = [ω1, · · · ,ωM] ,Ω′ = [ω′1 , · · · ,ω′M ] denotes the frequency matrices in inverse Fourier transform. In terms of structural form, the kernel function (5) represented in spectral way is similar to a onelayer neural network. Except for non-stationary and nonmonotonic property, this deep kernel with cosine activation function is more likely to depict the periodic and long-term characteristics of a distribution.

In order to preserve locality and general information in a distribution, we further introduce arc-cos kernel [Cho and Saul, 2009]:

where J(θ) = (sinθ + (π - θ) cos θ) and θ = cos-1 ( x·x′ ‖x‖‖x′ ‖ ). θ is the angle between x and x′ . Through kernel approximation, the feature map of arc-cos kernel is related to ReLUs in deep neural networks as:

Based on the above two deep kernel mappings, we design a hybrid spectral kernel mapping as in Fig. 3. In the hybrid spectral kernel mapping, spectral and arc-cos kernels are parallel and intertwined as ΦH(x) = [Φ1s (x) ,Φ1a (x)]. Then, we use the unified spectral transform technique [Xue et al., 2019] to stack an extra layer-wise kernel mapping in a hierarchical composite way to construct the hybrid spectral kernel network. The notation of the final hybrid kernel network is:

where the composite mapping is Ξ(2) (x) = Φ2H (Φ1H (x)). Combining deep spectral and arc-cos kernels provides a complete picture of the distribution and therefore circumvents drawbacks of the classical kernels. Besides, the deep architecture can also avoid the tricky kernel training with specific predefined kernel.

With the well-designed hybrid kernel network K = K(2) , we reinforce the biased empirical estimation of MMD metric and design the distribution matching loss function as:

3.3 Adversarial Kernel Training

The differentiable architecture of hybrid spectral kernel network makes adversarial kernel learning possible and therefore results in a more precise distribution alignment. Unlike domain-discrimination-based methods, our method directly applies adversarial learning to the metric computation, thus bringing more discriminatory expression for discrepancy metric. The adversarial training procedure in our AdvSKM can be represented by reformulating (9) as:

The procedure in (10) is associated with Generative Adversarial Network (GAN), where a generative model G and a discriminative model D are constructed to generate and discriminate instances, respectively. In our paper, the backbone network φ(·) is similar to G and strive to narrow the gap between domains, while K is similar to D and enlarges the gap. These two competitive components are trained separately. K is trained by maximizing the distribution matching loss (9), while φ(·) and f are trained when minimizing the overall loss function (11) with fixed K as input. Through adversarial training, the MMD with K can better discriminate between two distributions, and backbone network φ and classifier f can achieve more accurate results and better generalization capability.

3.4 Optimization

The components above construct the final objective function:

Based on the designed objective, the main steps of our proposed method are illustrated in Algorithm 1 where the back propagation optimization is based on the adaptive moment estimation (ADAM) algorithm. At first, the networks f, φ,K are initialized by truncated Gaussian distribution. Secondly, source and target instances are transformed into the invariant space with the FCN φ. Then our model is trained in a maxmini adversarial manner. At the maximization stage, the hybrid spectral kernel network K is trained with iteration L = 3. Correspondingly at the minimization phase, the projection networks and the classifier are jointly optimized with kernel network K as input. With tighter distribution distance, models can be fitted before the max iterations M = 30, 000.

Algorithm 1 AdvSKM

Input: Instances and labels XS, XT and YS; Initialized φ, f and K; Hyperparameters λ, L and M.

Output: Labels of target test data, YT .

while not convergence or max iteration do

Project XS and XT to the common space with φ.

for i=1:L do

Compute and gradient of (9) w.r.t K.

Update K to maximize (9) with gradient.

end for

Compute the loss (9) with updated K.

Compute gradient of (11) w.r.t φ and f .

Update φ and f to minimize (11) with gradient.

end while

Predict YT with the optimized model.

4 Experiments

4.1 Datasets

In the experimental settings, four multivariate time series datasets collected by sensors are selected following [Wilson et al., 2020]. The first dataset is the Human Activity Recognition (HAR) dataset [Anguita et al., 2013] which contains accelerometer, gyroscope, and estimated body acceleration data from 30 participants. The second is the Heterogeneity Human Activity Recognition (HHAR) dataset [Stisen et al., 2015], including acceleration sensor data from 31 smartphones. Next is WISDM activity recognition (WS AR) dataset [Kwapisz et al., 2011] including 33 participants' accelerometer data. Finally, hand gesture accelerometer data from 8 participants is provided from the gesture recognition dataset (uWave) [Liu et al., 2009]. In each dataset, certain sensor device is used to capture different activity behaviors of the participants. Each dataset includes a participant identifier, and we use this property to delineate the domains.

4.2 ComparingMethods

We compare our method with the following methods: Deep Domain Confusion (DDC) [Tzeng et al., 2014], Deep Adaptation Network (DAN) [Long et al., 2015], Deep Correlation Alignment (CORAL) [Sun and Saenko, 2016], Domainadversarial Neural Network (DANN) [Ganin et al., 2016], Central Moment Discrepancy (CMD) [Zellinger et al., 2017], Higher-order Moment Matching (HoMM) [Chao et al., 2020], Convolutional Domain Adaptation for Time Series (CoDATS). Specially, DDC, DAN, CORAL and CMD are the representative metric-based methods, while DANN and CoDATS are the representative domain-discrimination-based methods. Among them, CoDATS is the latest state-of-the-art adaptation method for time series data.

4.3 Experimental Settings

All the comparison methods are based on the same FCN backbone network for a fair comparison. We implemented our experimental framework by following the official codes of CoDATS. Following CoDATS's released codes, the difference between CoDATS and DANN is the domain/task classifier with different layers. We split the data in each dataset into training, validation and test. The training-test were split at 80% and 20%, respectively, and the training data were further split into training-validation with the same proportions. We randomly selected 5 adaptation problems between two domains. For each data point, we average over three different randomly initialized network weights on test set. During the training, we performed model selection by selecting the model that performed best on the validation set. We evaluate the models every 4,000 iterations in addition to the end of the training. The framework was trained for 30,000 iterations, which means that the best model was selected from nine models. Our codes 1 are published online.

4.4 Results

Table 1 tabulate the experimental results on four datasets. Boldface is used to highlight the best result of each case. "FCN" in the table represents the no adaptation model. From the results, we can observe that our method achieves the highest accuracy in all cases, which demonstrates the superiority of our method in various cross-domain scenarios. The poor performance of the no adaptation model indicates the significant difference between the source and target domains and the necessity for domain adaptation. The DDC model is much worse than our method on the HAR and WISDM AR datasets, indicating that the first-order statistic cannot characterize the samples between domains. Our method surpasses the second-order method CORAL by a large margin (20%) on the uWave and HHAR. Compared to the third and fourth order methods CMD and HoMM, our method outperforms them by 3-10% on the HAR dataset and far exceeds them on WS AR dataset. Our approach surpasses above metric-based approaches, which shows the importance of non-stationary, non-monotonic properties captured by our hybrid spectral kernel network. Besides, our method outperforms the domain-discrimination-based methods (DANN, CoDATS). Besides, from the feature visualization result in Fig. 4, we can see that these methods can confuse domains by domain-oriented discrimination. But they may eventually lead to the failure of the classification task due to the interclass confusion during domain matching.

Table 1: classification accuracy(source→target, mean±std%) on 5 randomly chosen problems for 5 datasets.

4.5 Feature Visualization

We also used the visualization tool t-SNE to help prove the validity of the AdvSKM. We visualize the final features obtained by different methods on the HAR 14→19 task. T-SNE embedding results are in Fig. 4(a)-4(j). We can distill the following observations: The shift between source and target domains is significant. The third and fourth order metric-based methods CMD and HoMM cannot effectively align domain despite the relatively higher classification accuracy. The latest domain-discrimination-based method CoDATS neglects category aggregation while conducting relatively effective domain confusion. With our proposed method, the source and target samples are matched better and categories are discriminated better as well.

4.6 Sensitivity Analysis

Two hyperparameters including λ and L are prepared for sensitivity analysis on four tasks. The influence of parameter variations on accuracy is shown in Fig. 5(a)-5(b). As shown in Fig. 5(a), since the accuracy rate does not fluctuate much, our model is not sensitive to the change of λ. Fig. 5(b) shows that our model is relatively insensitive within the appropriate range of adversarial iteration L. Specifically, L = 0 in Fig. 5(b) indicates that the distance metric is calculated using the initialized kernel mapping without updating the weights in the maximization phase.

5 Conclusion

In this paper, a novel metric-based domain matching method AdvSKM is proposed for UTSDA task by reform MMD metric with a well-designed hybrid spectral kernel network. Specially, the hybrid spectral kernel network maps the time series samples to RKHS and preserves the non-stationary and nonmonotonic statistics in the distributions, and thus bringing a more accurate discrepancy metric and ensuring the effective domain matching for UTSDA. Experimental results on distinct datasets demonstrate the superiority of our method to several state-of-the-art adaptation methods.

Acknowledgments

This work was supported by National Natural Science Foundation of China (Grant No. 62076062) and National Key RD Program of China (Grant No. 2017YFB1002801). Furthermore, the work was also supported by Collaborative Innovation Center of Wireless Communications Technology.