A Unified Meta-Learning Framework for Dynamic Transfer Learning

1 Introduction

Transfer learning [Pan and Yang, 2009] aims to leverage the knowledge of a source task to improve the generalization performance of a learning algorithm on a target task. The knowledge transferability across tasks can be theoretically guaranteed under mild assumptions, even when no labeled training examples are available in any target task [Ben-David et al., 2010; Ghifary et al., 2016; Acuna et al., 2021]. One key assumption is that source and target tasks are sampled from a stationary task distribution. The resulting relatedness between tasks allows transferring knowledge from a source task with adequate labeled data to a target task with little or no labeled data. However, the learning task might be evolving over time [Mohri and Medina, 2012] in real scenarios. For example, the data distribution of clothes images in Amazon is changing over the years due to the varying fashion trend [Al-Halah et al., 2017], thus resulting in a time-evolving image recognition task. Another example is the IMDb's film rating system where the rating scores of a film change in different time periods due to the dynamic user preference [Rafailidis and Nanopoulos, 2015], thus leading to a time-evolving film recommendation task. Such application scenarios would challenge the conventional static transfer learning algorithms [Pan and Yang, 2009] due to the dynamic task relatedness.

Recent works [Hoffman et al., 2014; Liu et al., 2020; Wang et al., 2020; Kumar et al., 2020] have studied continuous transfer learning with a static source task and a time evolving target task. They revealed that the prediction function on the newest target task can be learned by aggregating the knowledge from the labeled source data and historical unlabeled target data. Nevertheless, in real scenarios, both source and target tasks could be changing over time. In this case, those works will lead to the sub-optimal solution due to the under-explored source knowledge. To the best of our knowledge, very little effort has been devoted to modeling the knowledge transferability from a labeled dynamic source task to an unlabeled dynamic target task.

To bridge this gap, in this paper, we study the dynamic transfer learning problem with dynamic source and target tasks sampled from a non-stationary task distribution. As shown in Figure 1, conventional knowledge transfer problems focus on either static tasks (e.g., multi-task learning [Zhou et al., 2017; Zhou et al., 2019], transfer learning [Pan and Yang, 2009] and meta-learning [Finn et al., 2017]) or one dynamic task (e.g., continual learning [Li and Hoiem, 2017]), whereas we aim to transfer the knowledge from a dynamic source task to a dynamic target task. More specifically, we focus on the learning scenario where both source and target tasks always share the same class-label space (a.k.a. domain adaptation [Zhang et al., 2019]) at any time stamp. We are able to show that the generalization error bounds of dynamic transfer learning can be derived under the following assumptions. First, the class labels of the source task are available at any time stamp. Second, the source and target tasks are related at the initial time stamp. Third, the data distributions of both source and target tasks are continuously changing over time. The theoretical results indicate that the target error is bounded in terms of flexible domain discrepancy measures (e.g., H-divergence [Ben-David et al., 2010], discrepancy distance [Mansour et al., 2009], f-divergence [Acuna et al., 2021], etc.) across tasks and across time stamps. This motivates us to propose a generic meta-learning framework L2E for dynamic transfer learning. It reformulates the dynamic source and target tasks into a set of meta-pairs of consecutive tasks, and then learns the prior model initialization for fast adaptation on the newest target task. The effectiveness of L2E is empirically verified on various image data sets. The main contributions of this paper are summarized as follows:

• We derive the error bounds for dynamic transfer learning with time-evolving source and target tasks.
• We propose a generic meta-learning framework (L2E) for transfer learning on dynamic tasks by minimizing the error upper bounds with flexible divergence measures.
• Extensive experiments on public data sets confirm the effectiveness of our proposed L2E framework.

The rest of the paper is organized as follows. We review the related work in Section 2, followed by our problem setting in Section 3. In Section 4, we derive the error bounds of continuous transfer learning and then present the L2E framework. The extensive experiments and discussion are provided in Section 5. Finally, we conclude the paper in Section 6.

2 RelatedWork

2.1 Transfer Learning

Transfer learning [Pan and Yang, 2009] improves the generalization performance of a learning algorithm under distribution shift. Most existing algorithms [Zhang et al., 2019; Acuna et al., 2021; Wu and He, 2021] assume the relatedness of static source and target tasks in order to guarantee the success of knowledge transfer. The most related problem to our work is the meta-transfer learning [Sun et al., 2019], which learns to adapt to a set of few shot learning tasks sampled from a stationary task distribution. However, our work focuses on the non-stationary task distribution where a sequence of meta-pairs of consecutive tasks could be formulated for dynamic transfer learning.

2.2 Continual Learning

Continual learning aims to learn a model on a new task using knowledge from experienced tasks. Conventional continual learning algorithms [Li and Hoiem, 2017] focused on mitigating the catastrophic forgetting when learning the prediction function on one time-evolving task. In contrast, continuous transfer learning [Hoffman et al., 2014; Bobu et al., 2018; Wu and He, 2020; Liu et al., 2020; Wang et al., 2020; Kumar et al., 2020] transferred the knowledge from a labeled static source task to an unlabeled time-evolving target task. Our work would further extend it to the transfer learning setting with dynamic source and target tasks.

2.3 Meta-Learning

Meta-learning [Finn et al., 2017; Fallah et al., 2020], or learning to learn, leverages the knowledge from a set of prior tasks for fast adaptation to unseen tasks. It assumes that all the tasks follow a stationary task distribution. More recently, it has been extended into the online learning setting [Finn et al., 2019] where a sequence of tasks are sampled from nonstationary task distributions. However, those works focused on improving the prediction performance with the accumulated data, whereas we aim to explore the knowledge transferability between dynamic source and target tasks.

3 Problem Setting

Let X and Y be the input feature space and output label space respectively. We consider the dynamic transfer learning problem 1 with dynamic source task {Ds j }Nj=1 and target task {Dtj }Nj=1 with time stamp j. In this case, we assume that there are msj labeled training examples Ds j = {(xsij , ys ij )}msj i=1 in the j th source task and no labeled training examples in the target task. Let mtj be the number of unlabeled training examples Dtj = {xtij }mtj i=1 in the j th target task. Furthermore, each task Ds j (Dtj ) is associated with a task-specific labeling function f s j (f t j ). Let H be the hypothesis class on X where a hypothesis is a function h : X → Y. L(·, ·) is the loss function such that L : Y × Y → R. The expected classification error on the task Dj (either source or target) is defined as ϵj (h) = E(x,y)∼Dj [L(h(x), y)] for any h ∈ H, and its empirical estimate is given by ˆ ϵj(h) = 1 mj ∑mj i=1 L(h(xij ), yij ).

Formally, our dynamic transfer learning problem can be defined as follows.

Definition 1. (Dynamic Transfer Learning) Given labeled dynamic source tasks {Ds j }Nj=1 and unlabeled dynamic target tasks {Dtj }Nj=1 , dynamic transfer learning aims to learn the

1 In this paper, we assume that all the tasks share the same output label space Y for simplicity. Besides, we use Ds j to represent both the time-specific task (i.e., source task at the j th time stamp) and its data distribution (i.e., probability distribution of the source task at the j th time stamp over X ×Y) for notation simplicity.

prediction function for the newest target task DtN+1 by leveraging the knowledge from historical source and target tasks.

In dynamic transfer learning, we have the following mild assumptions. (1) The class labels of the source task are available at any time stamp. Specially, when the source task is static, it is naturally degenerated into a continuous transfer learning problem [Hoffman et al., 2014; Bobu et al., 2018; Wu and He, 2020; Liu et al., 2020]. (2) The source and target tasks are related at the initial time stamp j = 1. That is, those tasks might not be related in the following time stamps j > 1, as their data distributions can be evolving towards different patterns. (3) The data distributions of both source and target tasks are continuously changing over time.

4 The Proposed Framework

In this section, we first derive the error bounds for dynamic transfer learning, and then propose a generic meta-learning framework (L2E).

4.1 Error Bounds

Following [Ben-David et al., 2010], we consider a binary classification problem with Y ∈ {0, 1} for simplicity. Before deriving the generalization error bound for a dynamic target task, we first introduce some basic concepts below.

Definition 2. (L1 -divergence [Ben-David et al., 2010]) The L1 -divergence between two distributions D and D′ over X is defined as follows.

where Q is the set of measurable subsets under D and D′ .

Definition 3. (µ-admissibility) A loss function L(·, ·) is µadmissible if there exists µ > 0 such that for all x ∈ X, y, y′ ∈ Y and h, h′ ∈ H, the following inequalities hold.

For dynamic transfer learning, the following theorem states that the expected target error of the newest target task can be bounded in terms of historical source and target knowledge.

Theorem 1. Assume that the loss function L(·, ·) is µadmissible and obeys the triangle inequality. Given a class of functions HL = {(x, y) ↦→ L(h(x), y) : h ∈ H}, for any δ > 0 and h ∈ H, with probability at least 1-δ, the expected error ϵtN+1 for the newest target task DtN+1 is bounded by

where ˜ d = µ·max { max1≤j≤N-1 d1(Ds j , Ds j+1 ), d1(Ds 1 , Dt1 ), max1≤j≤N d1(Dtj ,Dtj+1 )}, ˜λ = µ · max { max1≤j≤N-1 λ∗(Ds j , Ds j+1 ), λ∗(Ds 1 , Dt1 ), max1≤j≤N λ∗(Dtj , Dtj+1 )}

and λ∗ measures the difference of the labeling functions across task and across time stamps, i.e., λ∗(Ds 1 ,Dt1 ) = min{EDs 1 [|f s 1 (x) - f t 1 (x)|], EDt1 [|f s 1 (x) - f t 1 (x)|]}. ˜ℜ(HL) is a Rademacher complexity term and ˜m = ∑Nj=1 (msj +mtj ) is the total number of training examples from source and historical target tasks.

This theorem reveals that the expected error on the newest target task is bounded in terms of (i) the empirical errors of historical source and target tasks; (ii) the maximum of the distribution discrepancy across tasks and across time stamps; (iii) the maximum of the labeling difference across tasks and across time stamps; and (iv) the average Rademacher complexity [Mansour et al., 2009] of the class of functions HL = {(x, y) ↦→ L(h(x), y) : h ∈ H} over all the tasks.

However, it has been observed that (1) L1 -divergence cannot be accurately estimated from finite samples of arbitrary distributions [Ben-David et al., 2010]; and (2) the generalization error bound with L1 -divergence is not very tight because L1 -divergence involves all the measurable subsets over X. Therefore, we would like to derive much tighter error bounds with existing domain divergence measures over either marginal feature space (e.g., H-divergence [Ben-David et al., 2010], f-divergence [Acuna et al., 2021] and Maximum Mean Discrepancy (MMD) [Gretton et al., 2012]) or joint feature and label space (e.g., C-divergence [Wu and He, 2020]). It is notable [Acuna et al., 2021] that the generic f-divergence subsumes many popular divergences, including Margin Disparity Discrepancy [Zhang et al., 2019], JensenShannon (JS) divergence, etc.

Corollary 1. Assume that the loss function L(·, ·) is µadmissible and symmetric (i.e., L(y1, y2) = L(y2, y1) for y1, y2 ∈ Y), and obeys the triangle inequality. Then (a) when using f-divergence [Acuna et al., 2021] 2 , denoted

by df , the error bound of Theorem 1 holds with

where λ∗(Ds 1 , Dt1 ) = minh∈H ϵs1 (h) + ϵt1 (h).

(b) when using C-divergence [Wu and He, 2020] (measuring the distribution discrepancy over joint data distribution on X × Y), denoted by dC, the error bound of Theorem 1 holds with

2 Note that we have similar error bounds when using other marginal domain discrepancy measures (e.g., H-divergence [BenDavid et al., 2010], discrepancy distance [Mansour et al., 2009], or MMD [Gretton et al., 2012]), so we omit the details here (see Appendix for more illustration)

These theoretical results motivate us to develop a dynamic transfer learning framework by empirically minimizing the error bounds with flexible domain discrepancy measures.

4.2 L2E Framework

Following [Ben-David et al., 2010], a typical transfer learning paradigm on static source and target tasks aims to minimize the static error bound involving empirical source classification error and domain discrepancy across tasks as follows.

where θ is the trainable parameters and γ is a trade-off parameter between the empirical source error and the domain discrepancy. The second term ˆ d(Ds , Dt ; θ) aims to match the data distribution of source and target tasks by learning the domain-invariant latent representation for every input example. Then the predictive function learned by the first term ˆ ϵs(θ) on the source task could be applied to the target task directly. In this case, existing works [Ganin et al., 2016; Acuna et al., 2021] have attempted to instantiate different domain discrepancy measures. Nevertheless, when the source and target tasks are evolving over time, it might be suboptimal when directly transferring the labeled source task to the newest unlabeled target task. That is because the success of knowledge transfer cannot be guaranteed if the task relatedness becomes weaker over time [Rosenstein et al., 2005].

In this paper, we propose a generic meta-learning framework L2E for transferring the knowledge from a dynamic source task to a dynamic target task. It leverages the knowledge from both historical source and target tasks to improve the predictive performance on the newest target task. Following our generalization error bounds in Corollary 1, we have the following objective function for learning the predictive function of Dt t+1 on the (N + 1)th time stamp.

where ˆ d(·, ·; θ) is the empirical distribution discrepancy estimated from finite samples. It can be instantiated with any existing domain discrepancy measures discussed in Corollary 1. We would like to point out that the error bound of Corollary 1 is derived in terms of the maximum distribution discrepancy across tasks and across time stamps. But there is no prior knowledge regarding the time stamp with the maximum distribution discrepancy. Therefore, in our framework, we propose to minimize all the distribution discrepancies across tasks and across time stamps.

However, the learned model on the newest target task DtN+1 might have the issue of catastrophic forgetting such that it performs badly on historical target tasks when updating the new target task. To solve this problem, we would like to learn optimal prior model initialization shared across all the target tasks such that this model can be efficiently fine-tuned on both new and historical target tasks with just a few updates. Figure 2 illustrates our proposed L2E framework with three crucial components: meta-pairs of tasks, meta-training, and meta-testing.

Meta-Pairs of Tasks

With the assumption that the source and target tasks are continuously evolving over time, the framework of Eq. (3) is equivalent to sequentially optimizing with respect to the adjacent tasks using standard transfer learning of Eq. (2).

Thus we would like to simply split all the tasks into a set of meta-pairs consisting of two consecutive tasks as shown in Figure 2, and learn the prior model initialization with those meta-pairs of tasks. Different from previous works [Hoffman et al., 2014; Liu et al., 2020] with only static source task, we argue that the evolution pattern of the source task can also help improve the performance of L2E on the newest target task (see more empirical analysis in our experiments).

Meta-Training

Let ζj(θ) = ˆ ϵtj (θ) + γ · ˆ d(Dtj , Dtj+1 ; θ), ζ0(θ) = ˆ ϵs1 (θ) + γ · ˆ d(Ds 1 ,Dt1 ; θ) and ζ-j(θ) = ˆ ϵsj+1 (θ) + γ · ˆ d(Ds j , Ds j+1 ; θ). Then Eq. (4) has a simplified expression minθ J(θ) = ∑Nk=1-N ζk(θ), where ζk denotes the objective function of standard transfer learning across tasks and across time stamps. Moreover, it can be formulated as a meta-learning problem [Finn et al., 2017]. That is, the optimal model initialization is learned from historical source and target tasks such that it can be adapted to the newest target task with a few updates. To be more specific, we randomly split the training data from every historical source or target task into one training set Dtr k and one validation set Dval k . Let ζ tr k (ζ val k ) be the loss function of ζk on the training (validation) set. The model initialization ˜ θ∗N can be learned as follows.

where Mk : θ → θk is a mapping function which learns the optimal task-specific parameter θk frommodel initialization θ. Following the model-agnostic meta-learning (MAML) [Finn et al., 2017], Mk(θ) can be instantiated by one or a few gradient descent updates. Here α ≥ 0 is the learning rate of the inner loop when training on a specific task. In addition, the training examples of historical target tasks are not labeled. Thus, we propose to sequentially learn the predictive function for every historical target task and generate the pseudo-labels of unlabeled examples as follows.

where ˆ yt ij is the predicted pseudo-label of input example xtij from the historical target taskDtj (j = 1, · · · ,N). Notice that the training examples with incorrect pseudo-labels might lead to the accumulation of misclassification errors on the new target tasks over time. To mitigate this issue, we propose to select those examples with high prediction confidence. Specifically, we estimate the entropy of the predicted class probability of target examples, and then choose the top p% with the lowest entropy values. We empirically evaluate this sampling strategy in the experiments.

Meta-Testing

The optimal parameters θN+1 on the newest target task DtN+1 could be obtained as follows.

where ˜ θ∗N is the optimized model initialization learned in the meta-training phase.

The intuition of this meta-learning framework L2E can be illustrated as follows. The evolution of dynamic source and target tasks can be represented as a sequential knowledge transfer process across time stamps. But from the perspective of transfer learning [Pan and Yang, 2009], it would be an asymmetric knowledge transfer process for every time stamp, with the goal of maximizing the prediction performance on the new task. That explains why continuous knowledge transfer [Bobu et al., 2018; Liu et al., 2020; Kumar et al., 2020] might have the issue of catastrophic forgetting. However, the continuous evolution of source and target tasks indicates that there might exist some common knowledge transferred across all time stamps. For instance, no matter how the fashion of clothes images changes over time, it follows the basic designs (e.g., shape) for different types of clothes. This common knowledge can be captured by the prior model initialization in our meta-learning framework. It then enables the fast adaptation on the newest target task with only a few model updates.

5 Experiments

5.1 Experiment Setup

Data Sets

We used three publicly available image data sets: Office-31 (with 3 tasks: Amazon, Webcam and DSLR), Image-CLEF (with 4 tasks: B, C, I and P) and Caltran. For Office-31 and Image-CLEF, there are 5 time stamps in the source task and 6 time stamp in the target task (see Appendix for more details). Caltran contains the real-time images captured by a camera at an intersection for two weeks.

Baseline Methods

The comparison baselines are given below: (1) static adaptation: SourceOnly that trains only on the source task, DANN [Ganin et al., 2016], and MDD [Zhang et al., 2019]; (2) multi-source adaptation: MDAN [Zhao et al., 2018], M3SDA [Peng et al., 2019], and DARN [Wen et al., 2020]; (3) continuous adaptation: CUA [Bobu et al., 2018], TransLATE [Wu and He, 2020], GST [Kumar et al., 2020], and our L2E with JS-divergence. Here, we merge all the labeled source data into a large one, and then transfer its knowledge to the newest target task for static adaptation methods. For fair comparison, all the multi-source and continuous adaptation methods used both historical source and target data for knowledge transfer, and the target selection strategy for choosing high-quality target examples with pseudo-labels.

Configuration

We adopted the ResNet-18 [He et al., 2016] pretrained on ImageNet as the base network for feature extraction, and set γ = 0.1 and p = 80 for all the experiments 3 .

5.2 Results

Figure 3 shows the results of domain discrepancy via MMD, including (i) the evolution of source task, i.e., ˆ d(Ds j , Ds j+1 ), (ii) the evolution of target task, i.e., ˆ d(Dtj ,Dtj+1 ), and (iii) the evolution of task relatedness, i.e., ˆ d(Ds j ,Dtj ). It indicates that in both Office-31 and Image-CLEF, the source and target tasks are changing smoothly, whereas the relatedness between source and target tasks is decreasing over time. Table 1, Table 2 and Figure 4 provide the transfer learning results on the dynamic tasks where the classification accuracies on the newest target task (Acc) and all the historical target tasks (H-Acc) are reported. We run all the experiments five times and report the mean and standard deviation of classification accuracies (the best results are indicated in bold). It can be observed that: (1) compared to static methods, the multi-source and continuous adaptation methods can achieve much better classification performance by leveraging the historical knowledge; (2) our proposed framework L2E outperforms state-of-the-art baselines in both the newest target task and all the historical target tasks. This confirms that L2Emitigates the issue of catastrophic forgetting on historical tasks when learning the new task.

3 https://github.com/jwu4sml/L2E

5.3 Case Studies

Table 3 reports the results of several variants on Image-CLEF (B → P): (i) L2E w/o source evolution: using only source data at the initial time stamp; (ii) L2E w merged source: merging all source data into a large one; (iii) L2E w/o historical target: transferring the dynamic source tasks to the newest target task directly without historical target knowledge. It is observed that the evolution knowledge from historical source and target tasks could indeed improve the performance of L2E in dynamic transfer learning. Besides, we also consider to generate the meta-pairs from any two historical tasks (indicated in "L2E w all pairs" in Table 3). It could not outperform L2E with only the meta-pairs from consecutive tasks. One explanation is that it might generate the unrelated meta-pairs of tasks. Figure 5 shows the results of L2E instantiated with JS-divergence [Ganin et al., 2016; Acuna et al., 2021], MMD [Long et al., 2015] and Cdivergence [Wu and He, 2020] on Image-CLEF (B → P). It indicates that our L2E framework is flexible to incorporate any domain discrepancy measures. Figure 6 shows the impact of sampling selection ratio p% on L2E where the classification accuracies on the newest target task (Acc) are reported on Image-CLEF. It confirms that selecting the historical target examples with high confidence positively affects the performance of L2E. Thus we choose p = 80 for our experiments.

6 Conclusion

In this paper, we study the transfer learning problem with dynamic source and target tasks. We show the error bounds of dynamic transfer learning on the newest target task in terms of historical source and target task knowledge. Then we propose a generic meta-learning framework L2E by minimizing the error upper bounds. Empirical results demonstrate the effectiveness of the proposed L2E framework.

Acknowledgements

This work is supported by National Science Foundation under Award No. IIS-1947203, IIS-2117902, IIS-2137468, and Agriculture and Food Research Initiative (AFRI) grant no. 2020-67021-32799/project accession no.1024178 from the USDA National Institute of Food and Agriculture. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government.