Deep Nonlinear Feature Coding for Unsupervised Domain Adaptation

1 Introduction

Conventional machine learning needs sufficient labeled data to achieve satisfactory prediction performance. Nonetheless, the acquiring of labeled data is an expensive and timeconsuming process. Domain adaptation [Ben-David et al., 2007; Pan and Yang, 2010; Margolis, 2011] provides an effective way to manage the label scarcity of data. The objective of domain adaptation is to learn a model that works well in a target domain where none or scarce labeled data is available, by leveraging upon the knowledge from a different but related source domain with plenty of labeled data. The main challenge of domain adaptation lies in the distribution discrepancy between source and target domains [Ben-David et al., 2007]. Therefore, a direct application of the model learnt from the source domain to the target domain often results in poor performance, and thus effective adaptation is in demand.

Depending on the availability of labeled data in the target domain, domain adaptation can be categorized into unsupervised one (with no labeled data) and semi-supervised one

(with scarce labeled data). In this paper, we focus on unsupervised domain adaptation, which is a more difficult task since no labeled data in the target domain can be used to guide the model learning.

In order to address the distribution discrepancy in unsupervised domain adaptation, various feature-based methods [Pan et al., 2011; Gong et al., 2012; Fernando et al., 2013; Long et al., 2014] are developed. They share the same underlying intuition which tries to find new feature representations well aligning the two domains. Among them, deep feature learning [Glorot et al., 2011; Chen et al., 2012; Zhou et al., 2014; Ding et al., 2015] has attracted much attention with demonstrated effectiveness. It learns deep features jointly from source and target data in an unsupervised manner. By doing so, generic concepts that exist in both domains can be extracted and captured in the new feature space. Such concepts can be further sharpened when more layers of deep features are constructed. Therefore, when operating on the deep feature spaces, the model learnt from one domain is able to adapt better to the other.

One typical unsupervised deep feature learning method is the marginalized stacked denoising autoencoder (mSDA) [Chen et al., 2012]. It extracts multi-layer deep features by reconstructing the input data from a number of randomly corrupted ones. By averaging over an infinite number of corruptions, robust features can be extracted efficiently without materializing any data corruption. The feature representation generated by mSDA has yielded very impressive performance for cross-domain sentiment analysis. It has also been employed as a building block in other domain adaptation methods [Zhou et al., 2014; Ding et al., 2015].

Despite its success, mSDA suffers from two limitations. First, it does not pay attention to the domain divergence that might arise in the new feature space. As a result, distribution discrepancy on the extracted features may still be large. Second, it injects the nonlinearity after the feature learning. Consequently, the extracted features may not capture much nonlinear relationship in the data, which compromises one of the biggest strengths of deep learning.

In this paper, we propose a Deep Nonlinear Feature Coding framework (DNFC) to address these two limitations of mSDA for unsupervised domain adaptation. In DNFC, the minimization of domain divergence and the exploitation of data nonlinearity are incorporated seamlessly into the deep feature learning. More specifically, we introduce two new elements, Maximum Mean Discrepancy (MMD) and kernelization, into 1-layer mSDA and stack the single-layer nonlinear feature coding to create higher-level deep features. As illustrated in Figure 1, at each layer, we measure the distribution discrepancy between the source and target domains in the new deep feature space by MMD and minimize it in the feature learning process. This enforces the distribution discrepancy in the new feature representation, on which domain adaptation is performed, to be minimized. Moreover, we also develop a kernelization that fuses the nonlinearity exploitation with the feature learning. Both elements play important roles in learning good features for domain adaptation and are enclosed in a unified framework.

2 RelatedWork

A number of feature-based methods have been developed for unsupervised domain adaptation. These methods aim to learn a common feature representation shared by source and target domains, on which the model built from the source domain adapts well to the target domain. One typical approach is to construct subspaces as new feature representations. Recent studies include GFK [Gong et al., 2012], DASA [Fernando et al., 2013], and TJM [Long et al., 2014]. GFK integrates an infinite number of subspaces that lie between source and target along the geodesic flow on a Grassmann manifold. DASA directly aligns the subspaces of the two domains, without considering intermediate subspaces. TJM extracts a subspace by jointly matching features and reweighting instances under the kernel PCA.

As another type of feature-based method, deep feature learning has recently emerged and demonstrated its effectiveness for unsupervised domain adaptation. Marginalized denoising autoencoder (mDA) [Chen et al., 2012] learns feature embedding by reconstructing the original data from the randomly corrupted ones. To extract multiple layers of deep features, mDA are stacked to mSDA in a layer-wise manner. Thanks to its effectiveness in capturing deep structures embedded in data, mSDA is also used as a building block for other domain adaptation methods [Zhou et al., 2014; Ding et al., 2015]. In addition, autoencoders are also applied to transfer model parameters across different domains [Deng et al., 2013; Kandaswamy et al., 2014; Zhuang et al., 2015].

In this paper, we incorporate two new elements into mSDA, which makes it more powerful for domain adaptation.

Our work adopts Maximum Mean Discrepancy (MMD) [Gretton et al., 2006] to quantify the domain divergence. MMD is widely used to measure the distance between two distributions in domain adaptation [Pan et al., 2011; Long et al., 2013; Wang et al., 2015]. It avoids density estimations and uses the mean embeddings of the two distributions in a Reproducing Kernel Hilbert Space (RKHS) H for distance calculation. Given two sets of samples X = ¶x1↪ ▷▷▷↪ xn1 ♦ and Y = ¶y1↪ ▷▷▷↪ yn2 ♦, drawn from the distributions P and Q respectively, the empirical MMD is calculated as:

where Φ : R ! H with K(xi↪ xj) = Φ(xi)TΦ(xj) as the kernel of H.

3 ADeep Nonlinear Coding Approach

In this section, we introduce the proposed Deep Nonlinear Feature Coding (DNFC) approach in detail.

3.1 Problem Statement

Denote Xsrc as a set of ns labeled samples from source domain: Xsrc = ¶xs1 ↪ ▷▷▷↪ xsns ♦ and Ysrc = ¶ys 1 ↪ ▷▷▷↪ ysns ♦, where xsi 2 Rd and ys i is the class label. Denote Xtar as a set of nt unlabeled samples from target domain: Xtar = ¶xt1 ↪ ▷▷▷↪ xtnt ♦, where xti 2 Rd . Denote X = Xsrc [ Xtar = [x1↪ ▷▷▷↪ xns ↪ ▷▷▷↪ xn] = [f1; f2; ▷▷▷; fd] 2 Rd⇥n , where n = ns + nt and fi 2 Rn is the i-th dimensional feature vector. The task is to predict the labels Ytar in the target domain, by leveraging the labelled data from the source domain.

3.2 Main Idea

The proposed DNFC aims to extract deep structures from source and target data, and use them as features in domain adaptation. DNFC is based on the marginalized stacked denoising autoencoder (mSDA) [Chen et al., 2012], which extracts multiple layers of deep features by stacking the marginalized denoising autoencoder (mDA). Though mSDA has demonstrated encouraging results for domain adaptation

[Chen et al., 2012; Zhou et al., 2014; Ding et al., 2015], it suffers from two limitations. First, the deep features are extracted by minimizing the reconstruction error solely. The divergence between source and target in the deep feature space is not taken care of. Second, mSDA learns a linear mapping, after which the nonlinearity is inserted by applying a nonlinear squashing function. As a result, the nonlinear relationship embedded in the data may not be well exploited and captured.

To address these two limitations, we propose to bring in two new elements: (1) Maximum Mean Discrepancy (MMD) for domain divergence minimization; and (2) kernelization for nonlinear coding. In the subsequent subsections, we first introduce the original mDA, followed by these two new elements. We then describe how we incorporate them seamlessly with mSDA in a unified framework for domain adaptation.

3.3 The Original mDA

mDA is a type of autoencoder that corrupts the inputs before mapping them into the deep representation [Chen et al., 2012]. It aims to learn a linear mappingW to reconstruct the original data from the corrupted ones. Its objective function is formulated as:

where X = [X↪X↪ ▷▷▷↪X], is the m-times repeated version of the input X; eX = [f X1 ↪ f X2 ↪ ▷▷▷↪g Xm] is the corrupted version of X with m different corruptions of X at a feature corruption probability p; and tr(·) is the operator to calculate the trace of a matrix. A robust mapping W in Eq. (1) can be obtained by averaging over infinitely many m corruptions, without actually constructing any corruption. This process is named marginalization and enables efficient implementation.

3.4 Domain Divergence Minimization byMMD

We propose to use MMD to quantify the domain divergence in the learnt deep feature space and incorporate it in the learning of W. This is particularly essential for unsupervised learning of deep features since no labeled data in the target domain is available to guide the feature learning.

Denote f xri as the i-th sample in the r-th corrupted version of X. Following the definition of MMD, the empirical distance between source and target domains in the r-th reconstructed space is:

Combining Eq. (1) and Eq. (2), and considering m corrupted versions, we formulate the objective function as:

where M = [Mi↪j]n⇥n with Mi↪j = 1 n2s if xi↪ xj 2 Xsrc; Mi↪j = 1 n2t if xi↪ xj 2 Xtar; otherwise Mi↪j = - 1 nsnt .

where fM is a block diagonal matrix with m copies of M as the diagonal elements, and ✓ > 0 is the balancing parameter. Eq. (3) has a closed-form solution:

Define q = [1 - p↪ 1 - p↪ ▷▷▷↪ 1 - p] 2 Rd , R = XXT, S = XMXT, and T = XNXT (N is a diagonal matrix whose diagonal elements are the same as those inM).

P =XeXT↪ Q1 = eXeXT↪ Q2 = eXfMeXT▷ The computation of W relies on the construction of m corruptions of X. The more corruptions are constructed, the more robust W is. Since the corruptions are obtained independently with the same corruption probability p for each feature, we can apply the weak law of large numbers and derive the expectations of P, Q1 and Q2 as m ! 1.

For Q1, we have:

When ↵ 6= β,

When ↵ = β,

For Q2, we have:

When ↵ 6= β,

When ↵ = β,

Similarly, we can obtain the expectation of P as E[P]↵↪β = qβR↵↪β .

After obtaining these expectation matrices, the mapping W can be computed as:

By incorporating MMD into mDA, we not only take advantage of mDA in obtaining discriminative deep features but also ensure that the new feature space brings the two domains as close as possible. As evidenced by our experimental results, this leads to much better adaptation performance. We also remark that the incorporation of MMD is seamless in the sense that the marginalization property of mDA is preserved and thus the efficiency of the proposed approach is ensured.

3.5 Nonlinear Coding by Kernelization

With the learning of W, only the linear relationship in the input data is captured. In the original mDA, the nonlinearity is inserted by applying a nonlinear squashing function on the output of mDA. Consequently, the nonlinearity in the data may not be well exploited because it is not taken care of in the learning of W. Note that with the incorporation of MMD, such a neglect also harms the minimization of domain divergence in the deep feature space.

In order to address this issue, we propose a kernelized solution. Specifically, we use a nonlinear mapping function Φ to map the original data to a RKHS. We then reconstruct the original data by randomly corrupting the mapped data in the RKHS. By doing so, we factor in the nonlinearity in the learning of W and thus in the extracted deep features.

Suppose that X is mapped to a RKHS H by a nonlinear mapping function Φ : Rd ! H. Denote Φ(X) = [Φ(x1)↪ ▷▷▷↪ Φ(xns )↪ ▷▷▷↪ Φ(xn)] as the mapped data matrix and K = Φ(X)TΦ(X) as the corresponding kernel matrix. The mDA term to reconstruct the original data from the corrupted mapped data now becomes:

where ^ Φ(X) = [^ Φ(X)1 ↪ ^ Φ(X)2 ↪ ▷▷▷↪ ^ Φ(X)m] represents the corrupted mapped data with m different corruptions of Φ(X).

With the incorporation of nonlinearity, the objective function that combines mDA and MMD terms becomes:

Φ(X)fM^ According to [Niyogi, 2004], the linear mapping W in H can be represented as a linear combination of the data points in H, that is, W = WkΦ(X)T. The objective function thus becomes:

+ ✓tr(Wk eKfMeKTWT where eK = [f K1 ↪ f K2 ↪ ▷▷▷↪ g Km] is the m corrupted kernel matrix with f Kr = [f kr1 ; f kr2 ; ▷▷▷; f krn ], f kri = [g kr i↪1 ↪ g kr i↪2 ↪ ▷▷▷↪g kr i↪n ] and g kr i↪j = Φ(xi)T^ Φ(xj)r . The closed-form solution of Eq. (6) is:

Pk =XeKT↪ Qk1 = eKeKT↪ Qk2 = eKfMeKT▷ Since corrupting Φ(X) corresponds to corrupting the kernel matrix K, we assume that each element g kr i↪j is corrupted with probability p. We now define the kernelized counterparts: q0 = [1 - p↪ 1 - p↪ ▷▷▷↪ 1 - p] 2 Rn , R0 = KKT, S0 = KMKT, and T0 = KNKT. The expected values of Qk1 and Qk2 can be obtained in a similar way to those of Q1 and Q2:

Similarly, we define U = XKT and obtain the expectation of the matrix Pk: E[Pk]↵↪β = q0βU↵↪β. Then, we obtain the mapping matrix:

Our nonlinear feature coding at a single layer is then defined as:

It encloses both the data nonlinearity and domain similarity in the deep feature space.

3.6 Deep Nonlinear Feature Coding (DNFC)

Similar to the stacking of mDA in mSDA, we stack our single-layer feature coding in Section 3.5 so as to create richer deep feature representations. The framework is illustrated in Figure 1. The stacking is performed by feeding the output Z = [Zsrc↪Ztar] of each layer into the next layer as the input. In this way, higher-level deep structures can be captured. To avoid overfitting the data nonlinearity during stacking, we conduct cross validation at each layer on the source data to select a suitable kernel function. Our proposed DNFC is summarized in Algorithm 1.

For domain adaptation, we concatenate the outputted feature coding at all L layers as the final feature representation. A classifier is then trained on the labeled source data and applied to the target domain to perform the prediction.

Algorithm 1 Deep Nonlinear Feature Coding

Input: Source data matrix Xsrc, target data matrix Xtar, and the number of layers L.

for k = 1 to L do

1. Select kernel function using cross validation on source;
2. Learn coding Zksrc and Zktar by Eq. (9);
3. Set Xk+1 src = Zksrc and Xk+1 tar = Zktar .

end for

Output: Feature coding ¶Zksrc ↪ Zktar ♦↪ (k = 1↪ ▷▷▷↪ L).

4 Experimental Results

In this section, we evaluate the performance of DNFC by comparing with state-of-the-art domain adaptation methods. We also study the properties of DNFC and analyze the effect of the two new elements in DNFC.

4.1 Datasets &Experimental Setting

We use two benchmark datasets that are widely used in domain adaptation.

Amazon product review dataset [Blitzer et al., 2007] contains sentiment reviews from four product categories: books (B), DVD (D), electronics (E) and kitchen appliance (K). Each review is characterized by unigram and bigram tfidf features and labeled as positive or negative. Each domain has about 5,000 samples. When a domain is selected as source (target), all the samples in this domain are used as training (test) data. By pairing up the domains for adaptation task, we have 12 domain pairs, denoted as 'source!target'. For example, K!E means that category K is the source domain and E is the target domain.

20-Newsgroups [Dai et al., 2007] consists of about 20,000 documents coming from four top categories: computer (C), recording (R), science (S), and talk (T). Each top category has four subcategories. Top categories are treated as labels, while subcategories are treated as related domains. Therefore, six binary prediction tasks are formed: C-R, C-S, C-T, R-S, R-T, and S-T. We take the task C-R for instance. Top category C is the positive class and R is the negative class. Two subcategories under each class are selected to constitute the source domain, while another two subcategories are selected to form the target domain. By exchanging the roles of the two domains, we have two domain pairs for the prediction task C-R, denoted as C-R1 and C-R2.

We compare DNFC with five baselines: NN, which predicts target labels using the 1-nearest neighbour classifier trained on the source domain with original features, DASA [Fernando et al., 2013], GFK [Gong et al., 2012], TJM [Long et al., 2014], and mSDA [Chen et al., 2012]. Following [Gong et al., 2012; Long et al., 2014], we use 1-NN as the base classifier since it avoids model parameter tuning. For subspace-based methods DASA, GFK, and TJM, we use the default subspace dimension d if it is specified by the authors, otherwise we set d to be the smallest one that contributes 85% of the energy. For TJM, we use the default kernel function specified by the authors. In our DNFC, we conduct the cross validation on source to automatically select between 'rbf' and linear kernels at each layer. For mSDA and DNFC, we set the default number of layers as three and do the cross validation on source to select the best corruption probability p between 0.1 and 0.9 with step size 0.1.

4.2 Comparison with Baselines

The classification accuracies of DNFC and five baselines on the two benchmarks are reported in Table 1. The best result on each domain pair is highlighted in bold, and the runner-up is highlighted in bold-italic.

As can be seen in Table 1, our DNFC achieves very promising performance and is a clear winner. Out of 24 domain pairs, DNFC performs the best in 21 and the second best in 3. For mSDA, on which DNFC is based, it is the best for 3 domain pairs and the runner-up for 15 domain pairs. The remaining 6 runner-ups mostly go to TJM, which is the most recent subspace-based method with nonlinear feature mapping. The performance improvement of DNFC over the second best method mSDA is up to 14.12% and is 5.32% on average. This demonstrates that the two new elements in DNFC are very effective in achieving better domain adaptation.

The results also show that the deep-feature-based methods, DNFC and mSDA, significantly outperform the subspacebased methods. This demonstrates the superiority of exploiting deep structures. The success of DNFC and TJM (with respect to other subspace-based methods) also reveals the necessity of capturing the data nonlinearity. Our DNFC takes advantage of both and thus is able to significantly boost the domain adaptation performance.

4.3 Property Study on DNFC

In this subsection, we study our proposed DNFC in four aspects: 1) sensitivity analysis on the balancing parameter ✓; 2) the influence of the number of layers; 3) the kernel functions selected in each layer; and 4) the effect of feature concatenation. Due to space limit, we only report the results of three domain pairs: S-T1, R-T1, and K!E.

As shown in Figure 2(a), with the increase of ✓, the classification accuracy generally increases until it gets stable. The larger ✓ is, the more sensitive the MMD term is to the objective function. When the empirical MMD gets stable, larger values of ✓ cannot further improve the performance. For all three domain pairs, this happens when ✓ reaches 103 , which is also the case for all other domain pairs. In our experiments, we set ✓ = 103 as default.

Figure 2(b) shows how the number of layers affects the performance of DNFC. The results of mSDA are also shown as a reference. In general, both DNFC and mSDA obtain better results when more layers are stacked. However, they have different trends. With the increase in the number of layers, the performance of DNFC converges very fast, while that of mSDA improves gradually. Even after 7 layers of stacking, mSDA is still not as good as DNFC. In particular, the accuracy of mSDA is about 5% worse than that of DNFC for S-T1 and R-T1 with 7 layers of stacking. The different trends may be due to the different ways of exploiting the data nonlinearity. mSDA injects the nonlinearity to the output of each layer, while DNFC incorporates the consideration of nonlinearity into the feature learning process. As a result, the data nonlinearity can be captured by DNFC more quickly with a small number of stacked layers. Considering that more stacked layers incur heavier computational overheads, the quicker convergence of DNFC also becomes a big strength, which makes it more practically useful in real applications.

Figure 2(c) shows the kernels selected by the cross validation on the source domain at different layers of DNFC. It demonstrates the flexibility of selecting different suitable kernels for different layers even for the same domain pair.

We also investigate the impact of concatenating all layers of features. We test DNFC using the final layer features only and find that the performance is just slightly compromised. On 20-Newsgroups, its average accuracy is only 0.53% lower than the one with concatenated features. This suggests that one can simply use the final layer features in DNFC if the slightly lower accuracy is not a big concern.

4.4 Effect of New Elements in DNFC

In this subsection, we assess the individual effect of MMD and kernelization in DNFC, as well as their joint effects, with respect to the performance of domain adaptation.

We develop two variants of mSDA: one with MMD but not kernelization, the other with the kernelization but not MMD. The two variants are referred to as mSDA MMD and mSDA Kernel, respectively. We compare DNFC with mSDA and the two variants on the 20-Newsgroups dataset.

The results are reported in Table 2. Overall, either MMD or kernelization alone improves the performance. This demonstrates the effectiveness of each individual element in DNFC. Moreover, as evidenced by the performance of DNFC, integrating MMD and kernelization together is able to further boost the improvement. We notice that the impacts of MMD and kernelization could vary even within the same domains.

This is because their effects are affected by the choices of p and the kernel, which are selected using cross validation on source data. We also observe occasional inferior performance of DNFC and the variants to mSDA. This may be again due to the parameter selection on source data, which might result in overfitting when generalizing to target domain.

5 Conclusion

In this paper, we propose a deep nonlinear feature coding framework for unsupervised domain adaptation. It incorporates MMD and kernelization into mSDA to extract nonlinear deep features with minimum domain divergence. Extensive experimental studies show that DNFC achieves very promising results. It consistently and significantly outperforms all baselines in terms of prediction accuracy. Compared to the best baseline mSDA, DNFC attains up to 14% better accuracy and converges much faster in terms of the number of stacked layers. The effects of MMD and kernelization on the performance of DNFC are also investigated. It is shown that DNFC benefits from both elements, as well as their joint effects. All the results demonstrate that DNFC is an effective and promising solution to unsupervised domain adaptation.

(a) Sensitivity analysis on the balancing parameter ✓

Acknowledgments

This work was done within the Rolls-Royce@NTU Corporate Lab with support from the National Research Foundation (NRF) Singapore under the Corp Lab@University Scheme.