Learning Discriminative Correlation Subspace for Heterogeneous Domain Adaptation

1 Introduction

In traditional machine learning, people usually have to collect as more labeled data as possible to achieve good classification performance in a target domain of interest. However, the collection and annotation of data can be very expensive and time-consuming. To tackle this issue, domain adaptation (DA) is proposed to utilize auxiliary data from another domain, a.k.a., source domain, to enhance the performance of the target task [Pan and Yang, 2010; Patel et al., 2015]. DA has been successfully applied in many real-world applications, including text classification [Chen and Zhang, 2013], visual recognition [Duan et al., 2010; 2012b], To build the connection between two different feature spaces, a usual way is to learn a feature transformation mapping to project the heterogeneous source and target data into the same space [Duan et al., 2012a; Li et al., 2014; Hubert Tsai et al., 2016]. Existing state-of-the-art methods mainly focus on reducing the domain distribution mismatch when learning the feature transformation and the target classifier. For example, SHFA [Li et al., 2014] employs feature augmentation strategy, while CDLS [Hubert Tsai et al., 2016] learns landmarks based on the maximum mean discrepancy.

∗The co-first authors.

†The corresponding author.

When learning the feature transformation to bridge the two heterogeneous feature spaces, one critical issue is how to preserve the useful information due to the feature transformation. However, if important information is lost after the feature transformation, it will be very difficult to boost the learning performance over the target data using the transformed source features, even the domain distribution mismatch is reduced by some efforts in the existing HDA approaches [Li et al., 2014; Hubert Tsai et al., 2016]. To this end, in this paper, we propose a new algorithm called Discriminative Correlation Analysis (DCA) to deal with the heterogeneous domain adaptation problem by learning a discriminative subspace that preserves the maximum feature correlation between the source and target domain data. On one hand, the learned subspace should maximize the feature correlation as measured in the canonical correlation analysis (CCA) method. On the other hand, it also ensures good discriminative ability such that the empirical training loss over target domain data is minimized.

We formulate a joint objective to learn the target classifier and the correlation subspace simultaneously, which, however, leads to a non-trivial optimization problem. To address it, we take advantage of the characteristic of general solutions to CCA, and represent the set of optimal projection matrices based on an orthogonal matrix and two arbitrary matrices (See Section 3.3 for more details). By optimizing a transformed objective, we seek to find the best projection matrix with discriminative power so that the feature correlation between two domains can be guaranteed to be maximized when learning the target domain classifier. Interestingly, the resultant optimization problem becomes much easier to address, since we have only a single orthogonal constraint in the reformulated objective. We then use an alternating direction method of multipliers (ADMM) algorithm to solve the resulting problem. We evaluate our proposed DCA approach on two real-world benchmark data sets.

2 RelatedWork

Homogeneous domain adaptation has been extensively studied in the last decade [Pan and Yang, 2010; Patel et al., 2015]. Recently, heterogeneous domain adaptation (HDA) has been attracting more and more attention. The principal challenge of HDA is that the feature spaces of the source and target domains are completely different, making it difficult to leverage the source data to assist the target task [Zhou et al., 2014a; 2016]. Generally, the existing HDA algorithms can be divided into two categories.

The first group of HDA algorithms utilizes auxiliary data to bridge source and target domains [Yang et al., 2015]. Textimage pairs are used to perform image classification tasks in [Zhu et al., 2011]. In [Wu et al., 2014], researchers constructed transition probabilities via co-occurrence data to enhance classification tasks. Tan et al. [Tan et al., 2015] proposed to select informative intermediate domains to connect source and target domains. However, collecting auxiliary data brings extra cost, and it might be difficult to obtain cooccurrence data in some applications like object recognition.

The second group of HDA algorithms performs feature transformation to connect two different feature spaces [Shi et al., 2010; Wang and Mahadevan, 2011; Kulis et al., 2011; Wu et al., 2013; Hoffman et al., 2014; Zhou et al., 2014b; Xiao and Guo, 2015]. Among them, most algorithms were designed for the supervised HDA scenario, where only labeled training instances are available in the target domain. The recent proposed SHFA [Li et al., 2014] and CDLS [Hubert Tsai et al., 2016] methods show that the unlabeled target domain instances are helpful for HDA tasks, and achieve state-of-the-art performance on the benchmark data sets. In particular, SHFA [Duan et al., 2012a; Li et al., 2014] augments labeled source and target data based on two projection matrices, and trained a SVM classifier on the augmented data. CDLS [Hubert Tsai et al., 2016] finds representative cross-domain landmarks to derive a domain-invariant feature subspace, and then train a classifier in the learned subspace. Nevertheless, those two methods did not consider the feature correlation between the source and target domain. In contrast, we find the optimal discriminative subspace that is ensured to maximize the feature correlation, thus our method is more effective for transferring the source domain knowledge for learning the target domain classifier.

3 Methodology

3.1 ProblemDefinition and Notation

In the heterogeneous domain adaptation problem, we have a source domain and a target domain that are represented by different features. The task is to learn a classifier for predicting the target domain instances.

Let us denote by XS = {(xS,i, yS,i)|nS i=1 } the source domain, where nS is the number of source domain instances, xS,i ∈ RdS is the i-th training instance with dS being the source feature dimension, and yS,i ∈ Y is the label of xS,i with Y = {1, . . . ,K} being the label space and K being the number of categories.

Similarly, we denote by XL = {(xL,i, yL,nL )|nL i=1 } the target domain labeled instances, where nL is the number of labeled instances, xL,i ∈ RdT with dT being the target feature dimension, and yL,i ∈ Y is the label of xL,i. Following [Li et al., 2014; Hubert Tsai et al., 2016], we also consider the semi-supervised situation where some unlabeled instances are available in the target domain, which is denoted by XU = {(xU,i)|nU i=1 }, where xU,i ∈ RdT . We also denote by nT = nL +nU the total number of instances in the target domain.

In the reminder of this paper, for any matrix A, A⊤ is the transpose of A, and the MATLAB style notation A(:, a : b) is the submatrix including the elements of all the rows and the a-th to b-th columns.

3.2 ProposedModel

The core issue of the HDA problem is how to learn a feature transformation, such that the source domain training data can be effectively utilized for the learning tasks in the heterogeneous target domain. To ensure that sufficient information is preserved during this process, inspired by the canonical correlation analysis (CCA), we aim to maximize the feature correlation when learning the feature transformation for HDA tasks.

Given two kinds of features, canonical correlation analysis (CCA) finds two orthogonal projection matrices P ∈ RdT×dC and Q ∈ RdS×dC , such that the correlations after projection are maximized, where dC is the dimension after projection. Since CCA usually considers the same number of two sets of data, we use n source instances XS = [xS,1, . . . , xS,n] ∈ RdS×n from XS and n target instances XT = [xT,1, . . . , xT,n] ∈ RdT×n from XL ∪ XU to perform CCA, where n = min{nS , nT }.

Mathematically, the optimization problem of CCA is

where I is an identity matrix. It can be equivalently written as a minimization problem as follows,

Although CCA finds the projection matrices that maximize the correlation, it does not utilize the label information of the target data. Therefore, the discriminative capacity of the projected features cannot be guaranteed, which is critical for learning a good classifier in the target domain. To this end, we propose to maximize the canonical correlation between two domains, and simultaneously minimize the regularized training loss on the labeled target data, which is given as follows.

where w is the parameters for the target classifier, ξL,i is the loss on the i-th labeled target instance, C and γ are the tradeoff parameters.

In particular, the cross-entropy loss is used in our model due to its smoothness and the ease of dealing with multi-class problems. To deal with the domain distribution mismatch, inspired by [Li et al., 2014], we augment each labeled target instance by its projected feature vector when learning the target classifier, i.e., the i-th augmented target instance is given by ˜xL,i = [x⊤L,i , (P⊤xL,i)⊤]⊤ ∈ RdC+dT , where P is the projection matrix. Then, the cross-entropy loss function is given by

where wk ∈ RdC+dT is the parameter vector for the k-th class. Also, the parameter w in Eq. (3) for target classifier can be written as

After w is learned, the label of the i-th unlabeled target instance xU,i can be predicted by

where ˜xU,i is an unlabeled augmented target instance.

To utilize the unlabeled instances in the target domain, we further use the current w to predict the unlabeled target instances in the training process. Then, we also minimize the cross-entropy loss for the unlabeled target instances with the pseudo labels, which is given as

where ξU,i is the loss on the i-th unlabeled target instance with the predicted label ˆ yU,i. We iteratively refine the pseudo labels, thus improve the learned subspace and the classifier.

3.3 Discriminative Correlation Subspace

Problem (7) is nontrivial to solve because of the non-convex objective function and the complicated orthogonal constraints. A suboptimal solution might be obtained, but good feature correlation and discriminative capacity can hardly be guaranteed.

To this end, we first find the set of subspaces that maximizes the canonical correlation term in (7), and then seek for the most discriminative subspace in the feasible set. To deal with the complicated orthogonal constraints, we take advantage of the characteristic of general solutions to CCA [Chu et al., 2013], which shows that P and Q can be represented by an orthogonal matrix, and two arbitrary matrices. Thus, we reformulate the problem (7) as a new one with only a simple constraint.

In particular, the characteristic of CCA is provided in the following theorem, Theorem 1 Let the reduced SVDs of XT and XS be

and

respectively. In addition, let

be the SVD of V⊤ T VS .

Under mild conditions, the solutions to Problem (2) can be represented by the following equations

where

Here, Θ is an orthogonal matrix, and ΦT and ΦS are arbitrary matrices.

For the proof, please refer to Section 3 in [Chu et al., 2013].

Remark 1 Theorem 1 states that given the matrices AT ,BT ,AS and BS, as long as Θ is orthogonal, the matrices P and Q obtained by Eqs. (11) and (12) are the solutions to Problem (2) for any ΦT and ΦS. In particular, these solutions have the same objective value of Problem (2).

Based on Theorem 1, instead of solving Problem (7) directly, we optimize P and Q from the feasible set of the CCA problem, such that the learned P and Q are beneficial for the classification task in the target domain. According to Eqs. (11) and (12), the problem of finding P and Q can be equivalently converted to the problem of finding Θ, ΦT and ΦS. Moreover, as all {Θ,ΦS,ΦT } give the same objective value of Problem (2), we further remove the CCA term in Problem (7), which leads to the following problem,

The new model leverages both source and target data to find a discriminative correlation subspace by exploiting CCA and label information of the target instances. With Θ and ΦT observed, we can obtain P according to Eq. (11), thus can project the target data into the found subspace. Since we only consider the training loss on the target data, we do not need to learn Q in the above problem.

3.4 Optimization

Now we discuss how to optimize Problem (13). In order to handle the orthogonal constraint on Θ, we construct a new matrix ˜Θ ∈ Ω, where the set Ω = {˜Θ| ˜Θ⊤ ˜Θ = I}. Then Problem (13) can be reformulated as an equality-constrained problem as

We apply the alternating direction method of multipliers (ADMM) to solve this problem [Boyd et al., 2011]. Specifically, the augmented Lagrangian function of Problem (14) can be written as

where ˜Θ ∈ Ω, Λ is the dual variable matrix, ρ is the penalty parameter.

The ADMM algorithm involves three main steps, which are detailed as follows.

• Update (w, Θ, ΦT ) by solving the following problem

• Update ˜Θ by solving the following problem

• Update Λ using a simple rule

The proposed Discriminative Correlation Analysis (DCA) algorithm is summarized in Algorithm 1. Since the objective is non-convex and Ω is also non-convex, Problem (14) is a non-convex optimization problem. However, as stated in Remark 1, as long as Θ is orthogonal, the matrices P and Q obtained by Eqs. (11) and (12) are the solutions to Problem (2) for any ΦT and ΦS. As a result, our method, which is inherited from CCA, is stable to the initialization of (Θ,ΦT ). Next, we are ready to describe how to solve Problems (16) and (17) efficiently.

Solution to Subproblem (16)

Subproblem (16) is still a non-convex optimization problem. To solve this, we alternatively update w and (Θ,ΦT ). Detailedly, we firstly fix w and update (Θ, ΦT ) by

Algorithm 1 Discriminative Canonical Correlation Analysis

Input: Unlabeled source instances XS = {xS,i}nS i=1 , labeled target instances XL = {xL,i, yL,i}nL i=1 , unlabeled target instances XU = {xU,i}nU i=1 .

• 1: Initialize (Θ,ΦT ) such that Θ⊤Θ= I.
• 2: Initialize w by Softmax Regression.
• 3: repeat
• 4: Obtain {ˆ yU,i}nU i=1 by Eq. (6),
• 5: Update (w,Θ,ΦT ) by solving Problem (16),
• 6: Update ˜Θ by solving Problem (17),
• 7: Update Λ according to Eq. (18),
• 8: until converge.

Next, we fix (Θ, ΦT ) and update w by solving the following problem,

which is a standard softmax regression problem on the augmented data.

We use the LBFGS algorithm to solve the above two subproblems [Schmidt, 2005]. The warm start strategy is used to solve these subproblems. Specifically, we update the parameters from the results that are obtained by the last iteration of ADMM. Note that the label information of the target instances is exploited to update (Θ,ΦT ), which assists to find the discriminative correlation subspace that is beneficial for the target classification task.

Solution to Subproblem (17)

Subproblem (17) seeks to find an orthogonal matrix that is closest to Θ+Λ. The solution can be calculated by Singular Value Decomposition. Specifically, let

be the SVD of Θ+Λ, the solution of Problem (17) is UV⊤.

4 Experiments

4.1 Data Sets

• Office Data Set. The Office data set [Saenko et al., 2010] includes 4106 images with 31 categories from three domains: amazon (A), dslr (D), webcam (W). We use the publicly available SURF [Bay et al., 2006] feature and the DeCAF6 [Donahue et al., 2014] feature in our experiments. The dimensions of SURF and DeCAF6 features are 800 and 4096, respectively.
• Office-Caltech Data Set. The Caltech-256 data set [Griffin et al., 2007] includes 256 categories, among which 10 categories are overlapped with the Office data set. The same 800-d SURF feature is publicly available for those 10 categories from the Caltech-256 data set.

We use these 10 overlapping categories between the Office data set and Caltech-256 in our experiments. To extensively evaluate our proposed method, we design experiments by exploiting all possible combinations across different domains and different types of features.

In particular, the Office data set contains both SURF and DeCAF6 features for all three domains (i.e., A, D and W), so for each pair of domains, we perform the HDA tasks by using one type of feature for the source domain, and the other one for the target domain, which leads to two groups of tasks (i.e., DeCAF6 → SURF, and SURF → DeCAF6). Each group contains 6 tasks, i.e., A→D, A→W, D→A, D→W, W→A, and W→D.

For the Office-Caltech data set, since the Caltech domain contains only SURF feature, we perform experiments by using Caltech as the source domain, and one of the three domains in the Office data set as the target domain, and vice versa. This also leads to two groups of HDA tasks. The first group takes DeCAF6 as the source feature, and SURF as the target feature, thus includes A→C, D→C andW→C three tasks. Similarly, the second group takes SURF as the source feature, and DeCAF6 as the target feature, thus includes C→ A, C → D and C → W three tasks.

Similar to the experimental settings in [Hubert Tsai et al., 2016], we randomly choose 3 target instances per category as the labeled target data for all the target domains, and the rest instances as the test data. For the source domains constructed on A, C and W, we randomly choose 20 source instances per category for training. For the source domains constructed on D, we randomly choose 5 source instances per category since the number of instances in D is much smaller. All experiments are conducted for 10 trials, and the average classification accuracy is reported for comparison.

For our DCA method, we empirically set the subspace dimension as dC = 20, the trade-off parameter as C = 1, and the penalty parameter of ADMM as ρ = 5. For the baseline method, we follow the previous work [Li et al., 2014] to report their best results on each data set. The parameter sensitivity of our DCA method is provided in Section 4.5.

4.2 Baseline Methods

We evaluate the effectiveness of our proposed method by comparing with the state-of-the-art HDA methods SHFA [Li et al., 2014] and CDLS [Hubert Tsai et al., 2016], as well as a few baselines as follows,

• SVM. We perform SVM [Chang and Lin, 2011] on labeled target data as a baseline without domain adaptation.
• Softmax. Since we adopt the cross-entropy loss in our proposed model, we also conduct softmax regression on labeled target data. This is also a baseline method without domain adaptation.
• CCA+Softmax. CCA+Softmax firstly performs CCA on source and target data, and then conducts softmax regression on the augmented labeled target data.
• SHFA. SHFA [Li et al., 2014] learns augmented features for source and target data, and trains an SVM classifier in the semi-supervised fashion simultaneously.
• CDLS. CDLS [Hubert Tsai et al., 2016] selects representative cross-domain instances to derive a proper feature subspace for domain adaptation. CDLS is also a semisupervised algorithm using unlabeled target data.

4.3 Results on Office Data Set

Table 1 presents the average classification accuracy and the standard deviation for all the methods on the Office data set. We have several observations as follows.

• SHFA and CDLS are better than the baseline SVMmethod, and our newly proposed DCA method is also better than the baseline Softmax method. Considering both SHFA and CDLS are designed based on SVM, and our DCA method adopts the same corss-entropy loss with the Softmax method, this clearly verifies that leveraging heterogeneous source data is beneficial to enhance the performance of the target classifier.
• CCA+Softmax performs comparably or slightly better than the baseline Softmax method. This indicates that the unsupervised method CCA cannot guarantee the good discriminative capacity of the learned subspace, which is critical for learning the target classifier.

• In terms of mean accuracy, our proposed DCA algorithm outperforms all other methods, which demonstrates the effectiveness of our proposed method for finding an optimal discriminative correlation subspace, and learning the robust target classifier. Specifically, comparing with the second best method SHFA, our method gets better results on 11 of those 12 tasks, which shows that it is beneficial to ensure the canonical correlation between the source and target domain when learning the target classifier.
• Compared to DeCAF6, SURF is relatively weak to represent objects, thus learning tasks based on SURF features are more challenging. As a result, the performance improvement of DCA can be marginal with auxiliary features.

4.4 Results on Office-Caltech Data Set

The results of all the methods on the Office-Caltech data set are reported in Table 2. We have similar observations as on the Office data set. Our proposed DCA method achieves the best performance in all of six tasks. This clearly demonstrates the effectiveness of our proposed DCA method for heterogeneous domain adaptation.

4.5 Sensitivity Study

We take the task A(DeCAF6) →D(SURF) as an example to show the parameter sensitivity of our proposed DCA method. In particular, our method mainly involves the penalty parameter ρ that controls the residual of the equality constraint in Eq. (14). We vary ρ ∈ {1, 5, 10, 15, 20, 25, 30}, and plot the results in Figure 1. In general, the performance of DCA is not sensitive to ρ. We plot the relative residual ϵ = ||Θ-˜Θ||F ||Θ||F w.r.t. ρ in Figure 1(b). When ρ = 1, the relative residual is large, which means that the equality constraint Θ = ˜Θ is not satisfied well. Nevertheless, when ρ ≥ 5, ϵ becomes very small, and the classification accuracy is also stable.

4.6 Performance w.r.t. Iterations

We study the performance of DCA w.r.t. the iteration number on two example tasks in Figure 2. Specifically, Figure 2(a) shows the results on the task of A(SURF) → D(DeCAF6), and Figure 2(b) shows the results on the task of W(SURF) → A(DeCAF6). From the figures, the accuracy of the proposed method improves with increasing iterations, and keeps relatively stable within 10 iterations. In other words, our method converges to a solution with good performance very quickly.

5 Conclusions

In this paper, we proposed a new heterogeneous domain adaptation method to learn a discriminative correlation subspace between source and target data. We formulated a unified objective to learn the target classifier, and simultaneously optimize the discriminative correlation subspace. An ADMM algorithm was applied for solving the proposed learning problem. Experiments on two real-world data sets clearly demonstrated the effectiveness of the proposed method.

Acknowledgments

This work is supported by the Guangzhou Key Laboratory of Robotics and Intelligent Software (15180007), NSFC (61502177 and 61602185), HKBU (12306616 and 12302715).