Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data

1 Introduction

Deep learning (DL) models, particularly convolutional neural networks (CNNs), have achieved remarkable successes in various time series tasks, such as human activity recognition (HAR) [Wang et al., 2019], sleep stages classification [Eldele et al., 2021] and fault diagnosis [Zhao et al., 2019]. These advanced DL models are often over-parameterized for better generalization on unseen data [Chen et al., 2017]. However, deploying those models on a resource-limited environment (e.g., smartphones and robots) is a common requirement for many real-world applications. The contradiction between model performance and complexity leads to the exploration of various model compression techniques, such as network pruning and quantization [Liang et al., 2021], network architecture search (NAS) [Elsken et al., 2019] and knowledge distillation (KD) [Hinton et al., 2015]. Among them, KD has demonstrated its superior effectiveness and flexibility on enhancing the performance of a compact model (i.e., Student) via transferring the knowledge from a cumbersome model (i.e., Teacher). Another well-known problem in many time series tasks is the considerable domain shift between model development and deployment stages. For instance, due to the difference between subject's genders, ages or data collection sensors, a model trained on one subject (i.e., source domain) might perform poorly on another subject (i.e., target domain). Such domain disparity makes cross-domain model compression even more challenging.

∗Corresponding author

Some recent works have already attempted to explore the benefits of applying unsupervised domain adaption (UDA) techniques during compressing cumbersome DL models by knowledge distillation. However, there are some drawbacks in these approaches. For instance, joint training of a teacher with UDA and student with KD would result in unstable loss convergence [Granger et al., 2020], while the knowledge from teachers trained on source domain only [Yang et al., 2020; Ryu et al., 2022] is biased and limited. For crossdomain knowledge distillation, a proper teacher should possess the knowledge of both domains. In particular, the generalized knowledge (namely Universal Knowledge) across both domains is more critical in improving student's generalization capability on target domain. However, the aforementioned methods coarsely align teacher's and student's predictions, but neglect to disentangle the domain-shared knowledge (namely Joint Knowledge). Due to the existence of domain shift, introducing source-specific knowledge would result in poor adaptation performance.

Fig. 1 presents an example of our proposed universal and joint knowledge under cross-domain scenario. On the one hand, the universal knowledge across source and target domains as shown in Fig. 1(a) is important to improve the generalization capability for the student. On the other hand, the inevitable domain shift makes the distributions of source and target domains overlapped. Suppose that there exists a data-domain discriminator to correctly classify the samples into source or target domain. As depicted in Fig. 1(b), if some samples lie around its decision boundary, then these samples most likely possess some domain-shared information (i.e., joint knowledge) which makes the discriminator incapable of correctly identifying their data domain (i.e., source or target). Meanwhile, those samples which can be very confidently identified by the data-domain discriminator tend to possess domain-specific knowledge. Equally treating all samples like conventional KD approaches would be adverse to diminishing domain disparity, leading to poor generalization on target data. It is thus highly motivated to pay more attentions on samples with joint knowledge than samples with domainspecific knowledge for cross-domain knowledge distillation.

In this paper, we propose an innovative end-to-end model compression framework to improve student's generalization capability under the cross-domain scenarios. Specifically, we design a feature-domain discriminator to align teacher's and student's feature representations for effectively distilling the universal knowledge. Meanwhile, a data-domain discriminator is developed to prioritize the samples with joint knowledge across two domains. It assists to disentangle teacher's logits by paying more attentions on the samples with joint knowledge. Via an adversarial learning scheme, teacher's universal and joint knowledge can be effectively transferred to the compact student. Our main contributions are summarized as follows.

• A novel approach named universal and joint knowledge distillation (UNI-KD) approach is proposed to transfer teacher's universal and joint knowledge, which is an end-to-end framework for cross-domain model compression. Two discriminators (i.e., feature-domain and data-domain discriminators) with an adversarial learning paradigm are designed to distill above two knowledge on feature-level and logit-level, respectively.
• We propose to disentangle teacher's logits with a datadomain discriminator by prioritizing the samples with joint knowledge across source and target domains. The joint knowledge could further boost the generalization ability of the compact student on target domain.
• Extensive experiments are conducted on four real-world datasets across three different time series classification tasks and the results demonstrate the superiority of our approach over other SOTA benchmarks.

2 RelatedWork

Knowledge distillation, as one of the most popular model compression techniques, has been widely explored in many applications. Originally, the knowledge from a complex teacher model is formulated as the logits soften by a temperature factor in [Hinton et al., 2015]. Then, researchers extend the knowledge to the feature maps as they contain more lowlevel information than logits. Several works try to minimize the discrepancy between teacher's and student's feature representations via explicitly defining distance metrics, such as L2 [Romero et al., 2014], attention maps [Zagoruyko and Komodakis, 2016], probability distributions [Passalis and Tefas, 2018] and inter-channel correlation matrices [Liu et al., 2021]. On the contrary, other researchers exploit the adversarial learning scheme which implicitly forces the student to generate similar feature maps as the teacher [Gao et al., 2019; Xu et al., 2021; Xu et al., 2022]. However, these approaches cannot be directly applied to cross-domain scenarios as they do not consider the domain shift during the compression.

To tackle the domain shift issue, various UDA approaches have been proposed. Generally, these techniques can be categorized into two types, namely discrepancy-based and adversarial learning-based. The former ones intend to minimize some statistical distribution measurements between source and target domains, e.g., maximum mean discrepancy (MMD) [Tzeng et al., 2014], the second-order statistics [Rahman et al., 2020; Sun and Saenko, 2016] or higher-order moment matching (HoMM) [Chen et al., 2020]. Whereas the adversarial learning-based ones attempt to learn domaininvariant representations via a domain discriminator [Ganin et al., 2016; Long et al., 2018; Shu et al., 2018; Wilson et al., 2020]. Although above mentioned UDA approaches have been successfully applied to many research areas, they seldom consider model complexity issue during domain adaptation, which is more practical for many time series tasks.

Recently, there are some attempts to jointly address model complexity and domain shift problems by integrating UDA techniques with KD for cross-domain model compression. In [Granger et al., 2020], a framework was proposed to employ the MMD to learn domain-invariant representations for teacher and progressively distill the knowledge to the student on both source and target data. However, their approach would lead to difficulty on student's convergence. MobileDA [Yang et al., 2020] performed the distillation on target domain with the knowledge from a source-only teacher. It leveraged the correlation alignment (CORAL) loss to learn domaininvariant representations for student. Similarly, a framework was proposed to perform adversarial learning and distillation on target domain with the knowledge from a source-only teacher in [Ryu et al., 2022]. The teachers in [Yang et al., 2020] and [Ryu et al., 2022] are trained on source data only and the knowledge from such teachers is very biased and limited. Unlike them, our method employs a teacher trained on labeled source domain and unlabeled target domain, and we distill not only the universal feature-level knowledge across both domains but also the joint logit-level knowledge shared by both domains via an adversarial learning scheme.

3 Methods

3.1 ProblemDefinition

For the cross-domain model compression scenario, we first assume that a proper teacher Ψ is pre-trained on source and target domain data with SOTA UDA methods (e.g., DANN [Ganin et al., 2016]). Our objective is to improve the generalization capability of the compact student ψ on target domain data. Same as other UDA works, we assume that data come from two domains: source and target. Data from source domain are labeled, DL src = {xisrc , yisrc }Nsrc i=1 , and data from target domain are collected from a new environment without labels, DU tgt = {xitgt }Ntgt i=1 . Here, Nsrc and Ntgt refer to the number of samples in source and target domains, respectively. Let P(Xsrc) and Q(Xtgt) be the marginal distributions of two domains. UDA problems assume that P(Xsrc) ̸= Q(Xtgt) but P(Ysrc|Xsrc) = Q(Ytgt|Xtgt), indicating that source and target domains have different data distributions but share a same label space.

Our proposed UNI-KD is depicted as Fig. 2. In order to effectively compress the model for the cross-domain scenario, we formulate teacher's knowledge into two categories: feature-level universal knowledge across two domains and logit-level joint knowledge shared by two domains. These two types of knowledge are complementary to each other. We introduce two discriminators with the adversarial learning scheme to efficiently transfer these two types of knowledge.

3.2 Universal Knowledge Distillation

For cross-domain KD, we define the generalized knowledge across both domains as the universal knowledge. Such universal knowledge contains the fundamental characteristics existing in both source and target domains. In order to align teacher's and student's feature representations, adversarial learning scheme is utilized as it is capable of enhancing student's robustness [Maroto et al., 2022]. Previous works also demonstrate that it could improve student's generalization capability on unseen data [Xu et al., 2022; Chung et al., 2020]. Motivated by this, we first design a feature-domain discriminator Df for transferring the universal feature-level knowledge. Df is a binary classification network to identify the source of input feature, i.e., whether the input features [fsrc, ftgt] come from Ψ or ψ.

We train Df and student ψ in an adversarial manner. To be specific, in the first step, we fix student ψ and train Df via loss LDIS as Eq. (1) shows. A batch of 'real' samples (feature maps from teacher with source and target domain samples as input) is forwarded through Df to calculate the loss log(Df (f Ψ x )). The gradients are calculated with back propagation. Then a batch of 'fake' samples (feature maps from student with same inputs) is forwarded through Df to calculate loss log(1-Df(f ψ x )). The gradients are accumulated to previous gradients from 'real' samples. At last, minimizing LDIS maximizes the probability of correctly classifying the input features as 'real' (from teacher) or 'fake' (from student). The second step is to fix the Df and train the student to generate similar feature maps as teacher. By minimizing LGEN in Eq. (2), the discriminator Df is expected to be incapable of telling whether the features are from Ψ or ψ. Alternately applying above two steps over all the training samples forces the student to learn similar feature maps as the teacher.

However, there are some challenges for the above adversarial learning scheme. Firstly, it can only transfer the universal knowledge but neglect the domain disparity between source and target domains, resulting in poor generalization on target domain. Secondly, the optimization of student ψ heavily relies on the accuracy of Df and the student would be difficult to converge especially in the early training stage. Thus, we introduce three additional losses in the second step of adversarial learning scheme to cope with the above issues in the following section.

3.3 Joint Knowledge Distillation

Logits from teacher contain more information compared to one-hot labels and thus could be utilized as 'dark' knowledge for distillation [Hinton et al., 2015]. However, we empirically found that simply combining conventional logits knowledge with feature distillation might lead to performance degradation. Due to the existence of domain shift, knowledge from teacher can be divided into domain-joint knowledge shared by two domains and domain-specific knowledge only existing in a particular domain. Since we feed both source and target domain samples to teacher and student, roughly minimizing their logits distributions would transfer both knowledge, leading to poor transferring performance.

Therefore, we intend to transfer the domain-joint knowledge but not domain-specific knowledge to the student. To achieve this, we utilize a data-domain discriminator Dd whose output is a binary probability vector ˆld = [pc=0, pc=1]. The element in above vector represents the probability of the inputs belonging to source domain (c = 0) or target domain (c = 1). We argue that the samples lying around the distribution boundary of source and target domains in the feature space are more generic than those samples which can be classified with high confidence. In other words, if the data-domain discriminator Dd cannot distinguish certain sample in the feature space, this sample most likely belongs to P(Xsrc) ⋂ Q(Xtgt) and possesses more domainjoint knowledge than others. Mathematically, pc=0 and pc=1 should be close to each other for these samples. Thus, we can utilize ˆld to disentangle teacher's logits and let the student pay more attentions on those low-confidence samples during logits distillation. Specifically, for each sample i, we assign a different weight wi to adjust its contribution for logits-level knowledge distillation in Eq. (3).

Then, the loss LJKD for joint knowledge distillation can be formulated as Eq. (4) where KL represents the KullbackLerbler divergence and qs , qt ∈ RC are the predictions of student and teacher, respectively. C is the number of classes. Each element qj in qs or qt is the probability of input sample belonging to the j th class and j ∈ {1, ..., C}. qj is a function of temperature factor τ used for smoothing the distribution and can be calculated via Eq. (5), and zj represents model outputs (i.e., logits) before the softmax layer.

It is foreseeable that the efficacy of disentangling domainjoint and domain-specific knowledge to a large extent depends on the accuracy of Dd. Therefore, we introduce a domain confusion loss LDC to assist the training of Dd in Eq. (6), where ld and ˆld are the data domain labels and predictions of data-domain discriminator Dd, respectively.

Furthermore, overfitting might occur if we only utilize target domain samples to train the student. To avoid this, we also optimize the student on the source domain via a cross-entropy loss as Eq. (7) shows.

Then, the final loss for training the student ψ in the second step of adversarial learning paradigm is formulated as below:

Here, we introduce two hyperparameters: α to balance the importance between domain confusion loss and logits KD loss, and β to adjust the contribution of LCE. For α, intuitively we intend to place more weights on UDA to achieve a good data-domain discriminator first and then gradually increase the importance of JKD as the training process goes on. This strategy can assist to stabilize student's training progress at the early stage. At each training epoch m, the corresponding value of α can be calculated by Eq. (9), where M is the total number of epochs, a and b are the starting and end values of α. In our experimental setting, we set α ∈ [0.1, 0.9]. The value of α is exponentially increased with training epochs. Meanwhile, we utilize the grid search approach to identify the optimal value of parameter β.

Algorithm 1 illustrates the details of our proposed UNI-KD for cross-domain model compression.

4 Experiments

4.1 Datasets

We evaluate our method with four commonly-used time series classification datasets across three different real-world applications: human activity recognition, sleep stage classification and fault diagnosis.

Algorithm 1 UNI-KD for cross-domain model compression

Input: A pre-trained teacher Ψ, a student model ψ, a featuredomain discriminator Df , a data-domain discriminator Dd, source dataset DL

• src and target dataset DU tgt . 1: for epoch m ∈ [1,M] do 2: Randomly shuffle DL src and DU tgt ; 3: Update α for current epoch via Eq. (9); 4: for each mini-batch in Xb ∈ (DL src , DU tgt ) do 5: Fix the parameters of ψ and Dd; 6: Forward Xb through Ψ as 'real' samples; 7: Forward Xb through ψ as 'fake' samples; 8: Calculate LDIS as Eq. (1); 9: Optimize Df via minimizing LDIS; 10: Fix the parameters of Df ; 11: Calculate LGEN as Eq. (2) and LCE as Eq. (7); 12: Calculate LDC as Eq. (6) and LJKD as Eq. (4); 13: Update ψ and Dd by minimizing L via Eq. (8); 14: end for 15: end for

UCI HAR [Anguita et al., 2013]: A smartphone is fixed on the waist of 30 experimental subjects and each subject is requested to perform six activities, i.e., walking, walking upstairs, walking downstairs, standing, laying and sitting. The measurements from accelerometer and gyroscope are recorded for identifying each activity. Due to the variability between different subjects, we consider each subject as an independent domain and randomly select five cross-domain scenarios same as [Ragab et al., 2023] for evaluation.

HHAR [Stisen et al., 2015]: In this dataset, each subject conducts six activities, i.e., biking, sitting, standing, walking, walking upstairs and downstairs. Since different brands of smartphones and smart watches are leveraged for data collection, this dataset is considered more challenging than UCI HAR in terms of domain shift. We follow [Liu and Xue, 2021] and select five cross-domain scenarios for evaluation.

FD [Lessmeier et al., 2016]: Total 32 bearings are tested under four different operation conditions for rolling bearing fault diagnosis. The motor current signals are recorded for classifying bearing health status, i.e., healthy, artificial damages (D1) and damages from accelerated lifetime tests (D2). We consider each operation condition as an independent domain and select five cross-domain scenarios for evaluation.

SSC [Goldberger et al., 2000]: Sleep stage classification (SSC) dataset aims to utilize electroencephalography (EEG) signals to identify subject's sleep stage, i.e., wake (W), nonrapid eye movement stage (N1, N2 and N3) and rapid eye movement (REM) stage. Each subject is considered as an independent domain and we select five scenarios for evaluation as previous studies [Eldele et al., 2021].

4.2 Experiments Setup

For our method, a well-trained teacher is a pre-requisite to perform cross-domain knowledge distillation. We adopt 1DCNN as the backbone of our teacher and student models since it consistently outperforms other advanced backbones such as

1D residual network (1D-Resnet) and temporal convolutional neural network (TCN) as indicated in [Ragab et al., 2023]. We leverage domain-adversarial training of neural networks (DANN) [Ganin et al., 2016] approach to train the teacher. The student is a shallow version of teacher which has less filters. See Supplementary for network details of teacher and student. Table 1 summarizes the model complexity of teacher and student in terms of the number of trainable parameters and the number of floating-point operations (FLOPs). We can see that our compact student is about 15× smaller than its teacher in the aspect of parameters and requires less operations during inference. Furthermore, regarding the evaluation metric, considering the fact that accuracy metric might not be representative for imbalanced dataset, we adopt macro F1score for all experiments as suggested in [Ragab et al., 2023]. For all experiments, we repeat 3 times with different random seeds and report the averaged values.

4.3 Effectiveness of Adversarial Distillation

Feature-level knowledge from teacher's intermediate layers has already been known as a good extension of logit-based knowledge as DL models are able to learn multiple levels of feature representations [Gou et al., 2021]. Various featurebased knowledge distillation approaches have been proposed in existing works. However, for cross-domain KD scenarios, we argue that adversarial learning could more effectively transfer the universal knowledge from teacher to student. To prove it, we first compare the adversarial feature KD with some commonly-used feature distillation approaches: Fitnet [Romero et al., 2014], PKT [Passalis and Tefas, 2018], AT [Zagoruyko and Komodakis, 2016], IEKD [Huang et al., 2021] and ICKD [Liu et al., 2021]. Please refer to Supplementary for details of each approach.

It is worth noting that we adapt above methods to our joint logit-level knowledge distillation. The only difference between these methods and ours is the feature distillation part. We utilize the t-SNE to visualize the learnt feature maps of above feature distillation approaches on HHAR dataset. As depicted in Fig. 3, the features learned from our proposed UNI-KD are more concentrated and all classes are well separated without overlapping. These observations demonstrate that the adversarial feature KD scheme could efficiently transfer the universal knowledge to the student for the crossdomain scenario. More t-SNE visualization results on other three datasets can be found in Supplementary.

4.4 Benchmark Results and Discussions

We compare our method with SOTA UDA algorithms, including some discrepancy-based and adversarial learningbased approaches as follows: deep domain confusion (DDC) [Tzeng et al., 2014], minimum discrepancy for domain adaptation (MDDA) [Rahman et al., 2020], higher-order moment matching (HoMM) [Chen et al., 2020], convolutional deep domain adaptation for time series data (CoDATS) [Wilson et al., 2020], conditional adversarial domain adaptation (CDAN) [Long et al., 2018] and decision-boundary iterative refinement training with a teacher (DIRT-T) [Shu et al., 2018]. Note that these UDA methods are directly applied to compact student. Meanwhile, we also include the results of some advanced works which integrate UDA with KD as follows: joint knowledge distillation and unsupervised domain adaptation (JKU) [Granger et al., 2020], adversarial adaptation with distillation (AAD) [Ryu et al., 2022] and MobileDA [Yang et al., 2020]. See Supplementary for the details of benchmark approaches. Besides, the performance of teacher and student trained on source domain ("Student src-only") are also reported as they can be considered as the upper and lower limits of the compact student. Tables 2 and 3 summarize the evaluation results over different domain adaptation scenarios on four datasets. More experimental results on additional transfer scenarios can be found in Supplementary.

From Tables 2 and 3, some observations can be found. Firstly, directly applying UDA on a compact student, either the discrepancy-based or adversarial-based UDA approaches, could somehow boost the student's performance on target domain in most of cross-domain scenarios as expected. However, in certain transfer scenarios, negative transfer might also occur. For instance, students trained with the DDC method on 2→7 for HHAR and DIRT-T on 0→11 for SSC even achieve lower performance than the student trained on source only, indicating that those UDA methods might suffer from the inconsistency problem on different domain adaptation tasks. Secondly, the JKU method performs worse than AAD and MobileDA in most of transfer scenarios. The possible reason is that the teacher in JKU is trained together with the student, and it would lead to convergence problem for the student when progressively distilling the knowledge. Moreover, since the teachers used in AAD and MobileDA are only trained on source domain data, the knowledge from these teachers is very limited and biased to source domain, resulting performance degradation. Thirdly, those jointly optimizing KD and UDA methods even under-perform UDA methods like DIRTT in most transfer scenarios, indicating that improperly transferring teacher's knowledge might decrease student's generalization capability on target data.

Lastly, our method consistently performs the best in terms of averaged macro F1-score over all the four datasets and outperforms other benchmarks on most of transfer scenarios. Moreover, compared with other joint KD and UDA methods, our UNI-KD can significantly reduce the performance gaps between teacher and student with the proposed universal and joint knowledge. This observation demonstrates the effectiveness of our method on the cross-domain model compression scenario. Via the evaluation on various time series domain adaptation tasks, our method can robustly compress the DL models with competitive performance as the complex teacher.

4.5 Ablation Study

There are three key components in our proposed approach: feature-domain discriminator Df , data-domain discriminator Dd and joint knowledge distillation (JKD). To analyze the contribution of each component, we conduct the ablation study as Table 4 shows. Moreover, to validate the effectiveness of proposed JKD, we also include the standard KD (SKD) in Table 4. Some conclusions can be observed from Table 4 . First, applying universal feature-level KD via integrating Df upon Dd could consistently improve student's performance over all datasets. However, integrating JKD upon Dd unexpectedly causes performance degradation in HHAR and SSC compared with only employing Dd. The possible reason is that logits contain less information than feature maps. Aligning teacher's and student's features could assist the student to learn more general representations than logits. Moreover, our UNI-KD suggests that these two types of knowledge are complementary to each other, and combining them can yield better performance as the last row shows. Furthermore, from the last two rows in Table 4, we can conclude that compared with standard KD, our proposed JKD is more effective in the cross-domain scenario.

4.6 Sensitivity Analysis

There are two hyperparameters (i.e., α and β) in our proposed approach as shown in Eq. (8). For α, we propose to gradually increase the importance of JKD loss during the training process as our method relies on an accurate data-domain discriminator. To validate its effectiveness, we compared our adaptive α method with fixed α values as illustrated in Fig. 4. We can see that the proposed adaptive α could consistently achieve better results than fixed α.

For the hyperparameter β, we utilize the grid search approach to identify the optimal values for different datasets. Fig. 5 illustrates the performance under different values of β. We can see that higher value of β will result in over-fitting to source data and would decrease the performance as expected. The optimal values for β is around [0.5, 1.0]. In all our experiments, we set β = 0.5 for dataset UCI HAR and HHAR and β = 1.0 for dataset FD and SSC.

5 Conclusion

In this paper, we propose an end-to-end framework for crossdomain knowledge distillation. Our method utilizes an adversarial learning paradigm with a feature-domain discriminator and a data-domain discriminator to improve student's generalization capability on target domain. With our proposed approach, the universal knowledge across both domains and the joint knowledge shared by both domains from a pre-trained teacher can be effectively transferred to a compact student. The experimental results show that the proposed UNI-KD can not only reduce the model complexity but also address domain shift issue.

Acknowledgments

This work is supported by the Agency for Science, Technology and Research (A*STAR) Singapore under its NRF AME Young Individual Research Grant (Grant No. A2084c1067) and A*STAR AME Programmatic Funds (Grant No. A20H6b0151).