Transfer Learning with Active Queries from Source Domain

1 Introduction

In many applications, we have plenty of unlabeled data but very limited labeled data, making the learning task rather difficult. Transfer learning and active learning are two important approaches to overcoming this challenge. The former tries to utilize data from a related source domain [Pan and Yang, 2010]; while the latter tries to query labels for the most valuable unlabeled data from an oracle [Settles, 2009]. In transfer learning, information is transferred from source domain to target domain at feature level [Gong et al., 2013; Tan et al., 2015] or instance level [Sugiyama et al., 2008; Xiao and Guo, 2015]. In active learning, unlabeled instances are actively queried based on informativeness or representativeness [Huang et al., 2014].

In recent years, there are some studies try to combine the transfer learning and active learning to learn with limited labeled data, either in separating stages [Li et al., 2013; Saha et al., 2011] or in one unified framework [Wang et al., 2014; Kale et al., 2015]. A common assumption of existing methods is that there are plenty of labeled data in the source domain and labels can be further queried in the target domain. However, such an assumption does not always hold. In many real tasks, label acquisition is expensive in both source domain and target domain, and thus the labeled data is usually insufficient in both domains. Furthermore, we even cannot get any additional labeled data in the target domain because the oracle is available only in the source domain. For example, in the influenza prediction task, we want to make prediction for a new strain of flu (target domain) by transferring knowledge from a known flu strain (source domain). At the beginning stage, we may not be able to precisely diagnose patients infected by the new flu, i.e., cannot acquire labels from the target domain. Although there are experts for the known flu strain, the diagnosing process could be time consuming and expensive, we thus need to actively select and diagnose a small number of patients which are most helpful for predicting the new flu. Another case is that the data from target domain contains sensitive private information and thus cannot be posted to annotators for labeling. Instead, we can actively query informative labels from a related yet non-private domain. In summary, we consider the setting where labeled data is insufficient in both source and target domains, and no oracle is available in the target domain. The problem setting is summarized in Figure 1. To the best of our knowledge, this problem has not been studied before.

⇤This research was supported by NSFC (61503182, 61472186, 61572252), JiangsuSF (BK20150754, BK20150755), CCFTencent Open Research Fund and Jiangsu QingLan Project.

In this paper, we try to address this problem by jointly performing transfer learning and active learning with queries from source domain. On one hand, to compute the importance weight for domain adaptation, we minimize the distance between the distributions of the target domain and adapted source domain. On the other hand, to select the most valuable instances from source domain for label querying, we minimize the distance between distributions of labeled and unlabeled data. These two objectives are integrated into one unified framework, where the distribution distance is estimated with Maximum Mean Discrepancy (MMD). To utilize the supervised information for better active selection, we further incorporate an uncertainty term based on the model prediction. At last, the framework is implemented and optimized with alternating quadratic programming. We test our approach for sentiment analysis on Amazon product reviews and text categorization on Reuters-21578. Results on 15 datasets validated the effectiveness of the proposed approach.

2 RelatedWork

In recent years, there have been increasing interests in combining transfer learning with active learning to deal with tasks with insufficient labeled data. However, they usually assume that there are plenty of labeled data in the source domain, and perform active queries only in the target domain.

Many approaches perform transfer learning and active learning separately. The approach proposed in [Shi et al., 2008] builds a classifier in the source domain to predict labels for the target domain, and queries the oracle only if the prediction is of low confidence. In [Li et al., 2013], two individual classifiers are trained with labeled data from the source and target domains respectively, and then informative samples are selected from the target domain based on the Query By Committee (QBC) strategy. The method in [Saha et al., 2011] builds a domain separator to distinguish between source and target domain data, and uses this separator to avoid querying labels for those target domain examples that are similar to examples from the source domain. Similar idea is implemented in another work [Rai et al., 2010].

There are also some studies combining the two tasks in one framework. The method in [Wang et al., 2014] relaxes the assumption to allow changes in both marginal and conditional distributions but assumes the changes are smooth between source and target domains. The authors incorporate active learning and transfer learning into a Gaussian Process based approach, and sequentially select query points from the target domain based on the predictive covariance. Kale and Liu [2013] present a principled framework to combine the agnostic active learning algorithmwith transfer learning, and utilize labeled data from source domain to improve the performance of an active learner in the target domain. Kale et al. [2015] propose a hierarchical framework to exploit cluster structure shared between different domains, which is further utilized for both imputing labels for unlabeled data and selecting active queries in the target domain.

Xiao and Guo [Xiao and Guo, 2013] study the active transfer learning problem under the online and multi-view setting, where instances are assumed to have multiple feature views, and arrive online in pairs, one from source domain and one from target domain. The method selects one of them with a fixed probability and decides whether to query its label based on multi-view disagreement or uncertainty.

The JO-TAL method proposed in [Chattopadhyay et al., 2013] is most related to our work. It also jointly performs transfer learning and active learning, and employs MMD to measure the distribution distance. However, it is significantly different from the proposed approach with the following reasons. Firstly, JO-TAL assumes that all source domain data are labeled, and tries to query more labels from the target domain, while in our approach, labeled data is insufficient in both domains and we need to query most valuable labels from source domain. Secondly, JO-TAL only minimizes the distribution distance between labeled and unlabeled data, while in our approach, we match the distributions of source and target data as well as labeled and unlabeled data, emphasizing the objectives of both transfer learning and active learning. Moreover, our approach explicitly incorporates the model prediction to further enhance the active selection with uncertainty. At last, different optimization techniques are used.

Transfer learning and active learning have been incorporated for various applications, such as cross-system recommendation [Zhao et al., 2013], natural language parsing [Attardi et al., 2013] and sentiment analysis [Luo et al., 2012]. Theoretical analysis is also presented in [Yang et al., 2013] with an upper bound on the sample complexity in sequential transfer learning settings.

3 The Method

We denote by S = SL [SU the dataset in the source domain, where SL = ¶(x1↪ y1)↪ · · · ↪ (xnSL ↪ ynSL )♦ is the labeled set consisting of nSL instances, SU = ¶x1↪ · · · ↪ xnSU ♦ is the unlabeled set consisting of nSU instances, and nS = nSL +nSU . Similarly, the dataset in the target domain is denoted by T = TL [ TU , with nTL labeled instances in TL and nTU unlabeled instances in TU, and nT = nTL +nTU . It is assumed that nSL ⌧ nSU and nTL ⌧ nTU , i.e., labeled data is insufficient in both source and target domains. We also assume that the oracle is available only in the source domain. This implies that the labeled data in the target domain is fixed in the whole learning procedure, and we need to actively query some informative labels from the source domain. This setting seems restricted, yet is common and practical as discussed in Section 1.

In this paper, we consider the covariate shift setting, where the margin distribution P(x) is different in the source and the target domains, while the conditional distribution P(y♣x) is the same. It is well known that the key issue for covariate shift adaptation is to accurately estimate the importance weight for each instance x, which is defined as β(x) = pT (x) pS(x) . Here pT and pS denote the density functions of target and source domains, respectively. To avoid the density estimation, we can directly optimize the importance weights by minimizing the distance between the distributions of the target domain and adapted source domain. Here we employ Maximum Mean Discrepancy (MMD) [Gretton et al., 2006; Borgwardt et al., 2006] as the criterion to estimate the distance between different distributions. Specifically, the empirical estimate of MMD between the target domain and the adapted source domain can be written as:

where ˆ S = ¶β(x)x ♣ x 2 S♦ is the set of adapted source domain data, and φ : X ! H is a mapping from the feature space to a Reproducing Kernel Hilbert Space (RKHS). It is easy to observe that the MMD is actually measured with the distance between the means of the two samples mapped into a RKHS. The target of domain adaptation is to optimize the importance weights β by minimizing Eq. (1).

As discussed previously, there is few labeled data in both target and source domains, and more labels should be queried from an oracle in the source domain. Because the label acquisition could be very expensive, we need to actively select as few unlabeled examples as possible from the source domain for label querying. The active selection criterion should favor instances which are most helpful on improving the classification model in the target domain. This is essentially different from traditional active learning, which selects instances to improve the model in the same domain. It has been validated by previous research that margin distribution matching is an effective approach for active selection [Chattopadhyay et al., 2012]. The basic idea is that after the label querying, the distributions of labeled data and unlabeled data should be close, such that the model trained will have good generalization ability. It is worth noticing that in our setting, the model is trained for predicting unseen instances in the target domain, while the queried instances along with existing labeled data are distributed in both domains. This implies that the importance weights for domain adaptation should be considered when performing distribution matching, consequently, making the active selection more challenging.

Formally, at each iteration of active learning, we select a small subset Q of size nQ from SU to query their labels. A vector ↵ = ¶0↪ 1♦nSU is introduced to identify which instances are selected, where ↵(x) = 1 indicates the instance x in SU is selected for query. In other words, we have Q = ¶x ♣ x 2 SU ↪ ↵(x) = 1♦. Again, MMD is used to measure the distance between two distributions, and the following measurement should be minimized:

Note that the labeled set consists of three parts: labeled data in the target domain TL, labeled data in the source domain ˆ SL and the queried data from source domain ˆQ. Here the symbol ˆ· represents a data set adapted with importance weights. For example, ˆQ = ¶β(x)x ♣ x 2 Q♦.

Noticing that MMD measures the distance between margin distributions, which means the label information is neglected during the active selection. We can thus incorporate an uncertainty term to further improve the active selection. Specifically, we first get the predictions of the current classification model g on all instances in SU, denoted by gSU . Then the certainty of an instance x is simply estimated with ♣g(x)♣, indicating that an instance with a prediction value closer to zero is more uncertain. Our target is to select a small batch of instances with larger uncertainty in the target domain. In other words, ↵ should be optimized to achieve a minimal value on ↵β♣gSU ♣.

By combining the objectives for the domain adaption, the margin distribution matching based active selection and the uncertainty based active selection all together, we have the following framework for Transfer Learning with Active queries from Source domain (TLAS):

where λ is a tradeoff parameter for balancing the contributions of distribution matching and uncertainty. This framework can be rewritten in more detail as the following optimization problem:

where nL = nSL + nQ + nTL and nU = nSU - nQ + nTU .

Note that the binary constraints on ↵ make the above problem NP-Hard. By relaxing the constraints to let ↵(x) 2 [0↪ 1], the problem in Eq. (4) is biconvex, and can be solved alternatingly with a guarantee on the convergence [Gorski et al., 2007]. To optimize ↵ with β fixed, we have the following quadratic programming problem:

where

Here K is the kernel matrix corresponding to the feature mapping φ(·), and 1 is a vector with all entries being 1. We use ◦ to denote the element-wise product of vectors or matrixes, and refer to by subscript SU the rows/columns in a vector or matrix for the unlabeled instances in SU. Other subscripts SL, TU , TL follow similar denotations.

To optimize β with ↵ fixed, Eq. (4) can be rewritten as follows, which is also a quadratic programming problem.

where

Note here we denote by 1 a vector or a matrix with all entries being 1, and assume that the unlabeled instances are ordered before the labeled data in S for simplicity of presentation.

The alternating optimization process is repeated until convergence. In our experiments, it converges in two iterations for most cases, and the algorithm is in general efficient. The pseudo-code of the proposed TLAS approach is summarized in Algorithm 1. Note that β can be simply initialized with all ones or by kernel mean matching. Because ↵(x) is relaxed from binary to a real value in [0↪ 1], it cannot be directly identified which instances should be selected, we thus instead sort the instances of SU in descending order of ↵, and select the top nQ instances to query their labels.

4 Experiments

4.1 Settings

The proposed TLAS approach is evaluated on two tasks: sentiment analysis and text categorization. The Sentiment Anal-

Algorithm 1 The TLAS Algorithm

• 1: Input:
• 2: S = SL [SU: source domain data;
• 4: nQ: batch size of active query; λ: tradeoff parameter.
• 3: T = TL [ TU: target domain data;
• 5: Calculate the kernel matrix K and initialize β;
• 6: For each active querying iteration:
• 7: Repeat until convergence
• 8: Update ↵ by solving Eq. (5);
• 9: Update β by solving Eq. (6);
• 10: Q top nQ instances of SU with largest ↵ values;
• 11: SU = SU ∩Q; SL = SL [Q;
• 12: Train the model based on TL and adapted SL with β.

ysis dataset 1 contains product reviews on Amazon from four domains: Book, DVD, Electronics and Kitchen. For each domain, 1000 positive reviews and 1000 negative reviews are collected. Each review text is represented by a 200 dimensional feature vector according to [Chattopadhyay et al., 2013]. By taking each domain as source or target domain, we have in all 12 domain pairs: B2D, B2E, B2K, D2B, D2E, D2K, E2B, E2D, E2K, K2B, K2D and K2E. For the text categorization task, we use a preprocessed subset of Reuters21578 2 as in [Dai et al., 2007]. Reuters-21578 is a collection of Reuters news articles, which are organized in a hierarchical structure. Following the method in [Dai et al., 2007], three top categories are selected: Orgs, People and Places. Each category has different sub-categories, and thus we can dived the data with different sub-categories into source and target domains. Then three binary classification tasks between top categories are constructed: Orgs vs People, Orgs vs Places and People vs Places.

For each dataset, we randomly divide the source domain data into two parts: 10% as the labeled set SL, and the rest 90% as the unlabeled set SU. Similarly, the target domain data is randomly divided into three parts: 50% for testing, 10% as the labeled set TL, and the rest 40% as the unlabeled set TU. We perform active queries iteratively. At each iteration, nQ instances are selected from SU, and are added into SL with label assignments. After each query, the classification model is trained based on TL along with adapted SL. The classification accuracy on the test data is recorded at each iteration. The data partition is repeated randomly for 30 times, and the average results are reported. We employ LibSVM [Chang and Lin, 2011] with default parameters to implement the classification model. In our experiments, we set nQ = 10 and λ = 10 as default for all datasets, and compute the kernel matrix K using RBF kernel with default parameters.

To the best of our knowledge, there is no existing study can be directly applied to our setting. The following methods are compared in our experiments:

• ÷ Random: Randomly selects instances from unlabeled source domain data SU, and performs domain adaptation with kernel mean matching (KMM) [Huang et al., 2006];

1 http://www.cs.jhu.edu/ mdredze/datasets/sentiment

2 http://www.cse.ust.hk/TL/dataset/Reuters.zip

• ÷ Uncertainty: Selects the most uncertain instances from the source domain, and performs domain adaptation with KMM;
• ÷ MPAL: Selects instances in the source domain by distribution matching according to the active learning method proposed in [Chattopadhyay et al., 2012], and performs domain adaptation with KMM;
• ÷ TLAS-b: A baseline of our method, which fixes β with KMM, and optimizes ↵ for active selection with Eq. (5);
• ÷ TLAS: The method proposed in this paper.

4.2 Performance comparison

We perform active queries iteratively, and record the classification performance in the target domain after updating the model with the queried labels. The performance curves with increasing queries are plotted in Figure 2 and Figure 3 respectively for the Sentiment and Reuters datasets. In Figure 2, we can observe that the proposed method TLAS achieves the best performance in most cases. As expected, Random sampling leads to the worst performance on most datasets. Uncertainty sampling usually achieves decent performance, but is less effective than TLAS. The performance of MPAL is not very stable. It works well on some datasets but fails on the others, suggesting that a valuable query for the source domain may be less helpful for the target domain. When comparing TLAS with TLAS-b, the proposed method is always superior to its baseline, validating that iteratively optimizing the importance weights β is useful on improving the performance. In Figure 3, we get similar results on the Reuters dataset. The superiority of TLAS is more obvious on Orgs vs Places and People vs Places, where even the baseline TLAS-b outperforms all the other compared methods. We also notice that the performance could be degenerated with more queries on Orgs vs Places and People vs Places. One possible reason is the negative transfer, which is an interesting challenge deserve to be overcome in the future.

4.3 Study with different labeled ratios

In this subsection, we examine the performance of the compared approaches with varying numbers of initially labeled data in the target domain. The experiments are performed with the ratio of initial labeled data (nTL ◁nT ) increasing from

10% to 50%. Due to space limitation, for each ratio, we report the area under the performance curve on Reuters, instead of plotting the whole performance curve. The comparison results are plotted in Figure 4. Note that the area under curve is normalized by the area of the full rectangle, such that the value is in the interval of 0 to 1. It can be observed that all the compared methods achieve better performance with more initial labeled data in the target domain. The superiority of TLAS over other methods is consistent with different ratios of initial labeled data. Surprisingly, even the baseline version TLAS-b can outperform the other methods in most cases, which suggests that our strategy of active selection is effective even with fixed importance weights for domain adaptation.

5 Conclusion

In this paper, we propose a novel and practical setting for active transfer learning, where labeled data is insufficient in both source and target domains, and further labels can be actively queried only from the source domain. We jointly perform domain adaptation and active selection in one framework, aiming to train an effective model for the target domain with least queries from the source domain. Experiments on 15 datasets validated the effectiveness of the proposed approach. In the future, we plan to extend the framework for transfer learning with multiple source domains. Also, other active selection strategies will be studied under the proposed setting.