Domain-Adaptive Text Classification with Structured Knowledge from Unlabeled Data

1 Introduction

Domain shift is common in many natural language processing (NLP) applications. For example, the word "rechargeable" is much more common in electronics product reviews than in book reviews, while the word "readable" is much more common in book reviews. Existing language models [Devlin et al., 2018; Liu et al., 2019] have exhibited outstanding performance in text classification tasks, but they fail to generalize to new domains without expensive labeling and retraining (Figure 1). To break out the data constraint, some methods with unlabeled data have been proposed as follows.

∗Equal contribution.

†Corresponding author. (E-mail: shanghang@pku.edu.cn)

Existing unsupervised domain adaptation methods for text classification can be grouped into two categories: taskagnostic methods [Long et al., 2015; Ganin et al., 2016; Li et al., 2021; Glorot et al., 2011] and pivot-based methods. Task-agnostic methods generally ignore the correlation among words across domains, which can contain rich semantic information in an NLP context. In contrast, pivot-based methods use domain-independent words (pivots) to bridge the domain gap by leveraging the correlations between pivots and non-pivots to learn domain-invariant features. Therefore, we would like to marry pivot-based method and pretrained language models to adapt them to novel domains.

The most prominent pivot-based methods are Structure Correspondence Learning (SCL) and its variants [Blitzer et al., 2006; Yu and Jiang, 2016; Ziser and Reichart, 2018]. In SCL, the pivots are defined as the words that occur frequently on both source and target domains and behave in similar ways that are discriminable for the classification task 1 . The model can effectively learn domain-invariant features for pivots, but it is more challenging for the non-pivots as they have domain-specific meanings. Therefore a self-supervised auxiliary task is applied to predict the pivots from the non-pivots. As a result, SCL implicitly captures the relationships between words by recognizing co-occurrence patterns between pivots and non-pivots and uses these relationships to infer domaininvariant features for the non-pivots.

1 In the existing literature, the definition of pivots often uses the term "behavior". To put it simply, the pivots should be highly correlated to one of the labels across domains.

However, SCL is limited in that it uses all non-pivots to predict the pivot terms, which leads to a noisy inference problem as very few non-pivots have a real relationship with the pivots. As a result, false correlations often occur for frequently used words such as pronouns (see Figure 2a). Alternatively, Knowledge Graphs (KG) is an effective way to represent complex relationships between concepts in a structured format and do not solely rely on noisy co-occurrence information. Therefore, in this paper, we present a pivot-based domain adaptation method from the KG perspective. Our method, called Domain Adaptation with Structured Knowledge (DASK), follows a 2-step approach as illustrated in Figure 2b. In contrast to SCL, DASK filters out false correlations by building a knowledge graph to explicitly capture the relationships between pivots and non-pivots on the target domain. Then during training, DASK employs a novel knowledge injection (KI) mechanism for the model to learn domain-invariant features of the non-pivots.

Another critical drawback of SCL is that the pivots are predefined only on labeled source domain texts and unlabeled target domain texts. There is little to ensure that the predefined pivots actually have consistent behavior across domains. To tackle this problem, DASK dynamically learns the pivots during training with pseudo-labels. A memory bank is maintained to keep track of the polarity scores of the candidate pivots on the source domain and the target domain, respectively. The words that have consistently high scores in the memory banks are collected as the pivots at the beginning of each training epoch. As we show, DASK using learned pivots outperforms the static, pre-defined pivots. To summarize, our major contributions are as follows:

• We propose DASK for text classification, which injects knowledge graph facts to better leverage the relationship between words for adaptation across domains.
• We construct a novel knowledge graph with attention scores from BERT to explicitly capture the relationship between pivots and non-pivots on the target domain.
• We design the pivot-induced cross-domain knowledge injection mechanism to learn domain-invariant features

for non-pivots using their relationship with pivots.

• We propose to maintain memory banks to learn domaininvariant pivots.
• We evaluate our method on the task of cross-domain sentiment classification for 20 domain pairs, where our method outperforms strong baselines by up to 2.9%.

2 RelatedWork

2.1 Cross-Domain Text Classification

Cross-domain text classification [Glorot et al., 2011; Ganin et al., 2016; Ziser and Reichart, 2018; Du et al., 2020] is a fundamental problem in domain adaptation for NLP. Before applying domain adaptation methods to complex problems, like machine translation and question answering, it was extensively researched as a simpler problem. Among text classification tasks, sentiment classification is most representative and established in terms of domain adaptation because there are standard datasets and evaluation protocols [Blitzer et al., 2007; Ziser and Reichart, 2018]. Therefore, we use sentiment classification to evaluate our method. It is important to note that our method can be easily generalized to other tasks such as topic labeling, news classification and so forth.

Methods for cross-domain text classification can be roughly categorized into two classes: task-agnostic methods, and pivot-based methods. The former includes divergence minimization [Long et al., 2015; Sun et al., 2016; He et al., 2018], stacked denoising auto-encoders [Glorot et al., 2011], discriminative adversarial training [Ganin et al., 2016; Tzeng et al., 2017], instance reweighting [Chen et al., 2021] and so forth. There are also some works that combine task-agnostic methods with NLP-specific approaches or models [Ghosal et al., 2020; Du et al., 2020]. In contrast, pivotbased methods are different from them in two aspects: 1) they make use of the relationship between pivots and non-pivots to help learn discriminative feature, 2) they focus on learning word-level features instead of instance-level features, which align with the nature of language. These two points make pivot-based methods stand out on NLP tasks.

2.2 Knowledge Graph

Knowledge Graph Construction. KG can be constructed in a supervised manner, such as DBPedia [Lehmann et al., 2015] and Wikidata [Vrandeˇci´c and Kr¨otzsch, 2014]; semisupervised manner such as Google's Knowledge Vault [Dong et al., 2014]; or unsupervised manner, such as MAMA [Wang et al., 2020a]. Among them, we are most interested in MAMA because it uses learned knowledge stored in pretrained language models without human supervision to construct a KG. With a forward pass of BERT, it collects the words or phrases in the input that have high attention scores with each other to form a candidate fact, which is represented by a triplet (head, relation, tail). Hence it is an easy fit for our knowledge-injected language model P-BERT. We can initialize P-BERT with the same weights from BERT used in MAMA, so that it becomes easier for P-BERT to interpret the knowledge graph facts that are constructed by MAMA.

Knowledge-Injected Transformers. Since the emergence of pre-trained language models like BERT [Devlin et al., 2018; Liu et al., 2019], many works have followed up to incorporate external knowledge into them [Wang et al., 2020b; Zhang et al., 2019; Wang et al., 2021; Liu et al., 2020]. Typically, these works either fuse textual features and factual features at the end of the corresponding encoders to enhance language understanding [Wang et al., 2021; Zhang et al., 2019], or use adapters for injection of multiple knowledge sources with the original model fixed [Wang et al., 2020b]. Although most of them tried to utilize open-domain knowledge graphs or knowledge bases, K-BERT [Liu et al., 2020] takes domainspecific knowledge as input along with text, and thus is particularly suitable for our task of domain adaptation. Also, by using a flattened sentence tree and the visible matrix, it is compatible to BERT and free from additional pre-training. Therefore, to promote domain transfer with pivots, we develop K-BERT into P-BERT by replacing its KI module with our Pivot-induced Cross-domain Knowledge Injection.

Beyond knowledge-injected transformers, another line of related works is tree-based transformers for processing treestructured data in general [Shiv and Quirk, 2019; Sun et al., 2020]. Despite their power in learning from arbitrary treestructured data, they are not directly compatible to BERT.

3 Method

DASK aims to facilitate domain adaptation with structured knowledge from the target domain. To structure the knowledge in the target domain, we build a knowledge graph to capture the relationship among pivots and non-pivots and inject them into source domain data. This is helpful because the model is able to learn domain-invariant feature for nonpivots (domain-specific) by inferring from their relationship with domain-shared pivots. Specifically, DASK follows a 2step approach as follows (see Figure 3):

Step 1. We construct a knowledge graph (KG) from the target domain texts to model the relationship between pivots and non-pivots. Specifically, it involves 1) extracting candidate facts from sentences, 2) filtering low-confidence facts. Section 3.1 shows details of how we construct the KG.

Step 2. After constructing the knowledge graph, we start knowledge-injected finetuning on the source and target domains jointly. At each training step, we obtain labeled texts from the source domain, unlabeled texts from the source domain and target domain. The inputs are fed into P-BERT where all of them are injected with pivot-relevant facts from the knowledge graph (section 3.2), and then forwarded to the transformer encoder and the linear classifier sequentially. Finally, the predictions of labeled data are used to compute the cross-entropy loss with the ground truth labels. Meanwhile, the predictions of unlabeled data are leveraged to generate pseudo-labels for pivot learning (section 3.3). The pseudolabels of unlabeled data and ground truth labels of labeled data give updates to the polarity scores in the memory banks. Note that during inference the memory bank is not updated.

3.1 Knowledge Graph Construction

Prior to training, we build a KG on the target domain corpus to capture the relationship between pivots and non-pivots.

Candidate Fact Extraction. We extract candidate facts on a sentence level. An input text is decomposed to a list of sentences. We feed each sentence into BERT to get the attention matrix in the last transformer encoder layer. As a pre-processing step, the multi-head attention is averaged into single-head so that one pair of words only corresponds to one scalar attention value (if a word consists of multiple tokens, the attentions of the tokens are also averaged). Denote the pre-processed attention matrix as M. For each pivot p in the sentence, we search for the words w1, w2 that have the highest and second highest attention with p. Then the fact is formed as the triplet of w1,w2, p in their original order in the sentence. In addition, each fact is assigned a confidence score M[p][w1] + M[p][w2].

Filtering. The candidate facts are filtered according to their confidence scores. Those whose confidence scores are under a threshold are removed from the knowledge graph.

Note that although we learn pivots dynamically during training, those pivots all come from a large pivot pool that is constructed before training (see section 3.3). We construct the knowledge graph according to the pivot pool so that we do not have to update the knowledge graph as we learn new pivots and eliminate old ones. In the appendix, we showcase some qualitative results of a KG that we constructed in this way and compare it with the ConceptNet subgraph.

3.2 Knowledge Injection

In order to utilize the rich semantics in the KG to learn domain-invariant features for the non-pivots, we propose Pivot-induced Cross-domain Knowledge Injection (PCKI) to inject knowledge facts into input texts. On the other hand, we would like to use a pretrained language model as the feature extractor, so we borrow ideas from K-BERT [Liu et al., 2020] to let the language model understand the structure of knowledge injected input. As a whole, our model P-BERT is composed of the knowledge injection module and the subsequent transformer encoder, as illustrated in Figure 3.

In the knowledge injection module, for each pivot in the input text, we search for the facts relevant to it and inject them into the text, forming a tree structure. Figure 2b shows an example for a knowledge injected text. In the example, "Great" is a pivot in the sentence on the right, and we have the fact triplet "(great, makes, simple)" in the KG that we extract from the sentence on the left. The triplet is then appended to "great" as a branch. By injecting facts extracted from the target domain into labeled source domain texts, we embed the target domain non-pivots in a labeled context, conditioning on the related pivots. Therefore, our transformer encoder is able to infer their features according to their relationships with pivots, under the supervision of source domain labels.

To feed the knowledge-injected text to the transformer encoder while keeping the structure information, we flatten it into a token sequence, and use position embedding to recover its structure [Liu et al., 2020]. Specifically, as illustrated in Figure 4, the injected words are inserted before or after the corresponding pivot following their order in the triplet so as to get the flattened token sequence. Meanwhile, to recover the tree structure, we assign the index for position embedding of each token as its depth in the tree. In the example, "makes" and "book" are assigned the same position embedding index 2. In this way, the token sequence is fed into the transformers, with the position embedding informing its structure. Besides, a visible matrix is applied to the self-attention modules in the transformers to control knowledge noise. We refer readers to K-BERT [Liu et al., 2020] for more details on visible matrix.

3.3 Pivot Learning

As a pivot-based method, DASK heavily relies on the domain-shared pivots. In this section, we describe how we select and learn the pivots with memory banks.

Recall that pivots are defined as the words that behave similarly in the source and target domains. Following previous works, we represent the behavior of a word with the labels of the texts in which it appears. The principle is that the label distribution of a pivot should be low-entropy (biased towards one label), and consistent across domains. While it is applicable to most classification tasks, for the binary sentiment classification task we focus on, we define a polarity score p(w, D) to measure more easily how much the label distribution of w is biased towards the labels on domain D:

(1)

where b(w,D) is the label set of w on D. And thus the behaviour on the domain pair (S, T) can be characterized as:

Taking the absolute average ensures that a word of high score must be biased towards the same label on both domains.

Prior to training, we collect a large pool of candidate pivots from the vocabulary which contains words that appear frequently on both domains. Other words are not taken into account because the precondition is that pivots should be frequent on both domains. To track the polarity scores of the candidate pivots, we build memory banks for the source domain and the target domain respectively. Both of them are initialized by scores computed with equation 1 constrained on source domain labeled data. Then we compute the absolute mean polarity score of the candidate pivots following equation 2 and take the K candidate pivots with the highest score as the initial pivots.

During training, at each training step, we acquire pseudolabels for the unlabeled texts if the prediction confidence in the softmax logits is over a threshold. The pseudo-labels of the unlabeled source domain inputs, and the ground truth labels of the labeled source domain inputs are used to update the source memory bank, while the pseudo-labels of the target domain inputs are used to update the target memory bank. The update is carried out in a temporal difference style: For each candidate pivot, if it is in the text from domain D labeled as l ∈ {1,-1}, then

where α is the update rate. In this way, we manage to estimate the behavior of words more accurately on the domains, so that domain-invariant pivots are acquired. At the beginning of each epoch, we compute the absolute mean scores of the candidate pivots following equation 2 and take the top-K candidate pivots as learnt pivots. The pivots are kept fixed in the middle of an epoch to avoid overly frequent pivot changes.

4 Experiments

To evaluate the efficacy of our proposed method, we extensively experiment on two cross-domain sentiment classification datasets. We compare our method with multiple baselines on 20 domain pairs and provide justification for the improvement of our method. Moreover, we carry out ablation study to respectively assess the effect of knowledge injection and dynamic pivot learning.

4.1 Experiment Settings

Datasets. We use the standard Amazon-product-review 2 dataset [Blitzer et al., 2007]. It contains four types of product reviews: books(B), dvd(D), electronics(E), kitchen(K), which form the four domains in the dataset. Besides, following PBLM [Ziser and Reichart, 2018], we also introduce the Airlines dataset 3 .

We evaluate our method on all 20 domain pairs involving the five domains. Each time before training, we randomly sample 400 labeled source domain data for dev set, and the rest 1600 along with all unlabeled data from source and target domain are used for training. During testing on the 2000 target labeled data, we stop the memory bank update, and use the pivots learnt at the best training epoch.

Baselines. We use BERT [Devlin et al., 2018] as the feature extractor. Apart from directly finetuning BERT on the source domain, we compare our method with multiple strong baselines incorporating methods proposed by previous works. The baselines are listed as follows:

2 Dataset can be found at http://www.cs.jhu.edu/∼mdredze/ datasets/sentiment/index2.html

3 Dataset and process procedures can be found at https://github. com/quankiquanki/skytrax-reviews-dataset

• Base: BERT trained on the source labeled data and directly test on the target labeled data.
• HATN: BERT combined with HATN proposed by [Li et al., 2018]. It is a prominent variant of Structure Correspondence Learning (SCL).
• DANN: BERT combined with the popular adversarial training method, DANN [Ganin et al., 2016].
• DAAT: BERT-DAAT proposed by [Du et al., 2020]. It combines target domain post-training along with domain adversarial training to boost domain adaptation.

To show the compatibility of DASK with existing methods, we stack SCL on top of our method. Since there was no existing work applying SCL on transformers, we manufactured a training scheme for SCL that mimics MLM in BERT [Devlin et al., 2018] pretraining, that is, to replace the pivots as [MASK] tokens and ask the model to recover the masked tokens given the context.

Hyperparameter Tuning. For all the methods in our experiments, we set the learning rate to 2e-5, warmup 0.1, batch size 32, and select weight decay in {1e-4, 2e-4, 3e-4}. For adversarial training, we select γ in the gradient reversal layer from {0.15, 0.25, 0.5, 0.75, 1.0}. For KG fact filtering, we set the confidence threshold in the interval [0.1, 0.45], according to the confidence distribution. Besides, we leverage the stopwords in the NLTK library and discard the facts that include the stopwords. For SCL, we apply a balance factor λ to the pivot-prediction loss, and choose its value from [0.1, 0.5]. To make the learning progress smoother, we also use a linear warmup to λ and set the warmup rate to 0.1. In addition, since our sentiment classifier is trained on the source labeled data, and the size of labeled data is 6-20 times smaller than that of unlabeled data, our model see each labeled data for 6-20 times in one epoch. In order to prevent overfitting, we update the sentiment classifier once in 5-11 training steps. For the dynamic memory bank, we update the pivots every 10 training steps, and the top 500 words are selected as pivots.The learning rate of word sentiment score is set to 1e-4 or 2e-4, and the pseudo-labeling confidence threshold to 0.9.

(a)

Table 2: a) Cross-domain sentiment classification accuracy on the 8 domain pairs between airlines dataset and 4 domains from Amazonproduct-review dataset. b) Ablation study on PCKI, SCL and dynamic memory bank. We did experiments on 5 domain pairs.

4.2 Experimental Results

Table 1 shows the performance of our method on the 12 domain pairs from Amazon-product-review dataset, compared with multiple baselines. Among all those methods, we observe that DASK+SCL gives the best performance on average. It outperforms the Base model by 0.89%, and beats the other strong baselines by 0.50%-2.90% on an average basis over all domain pairs. On the other hand, DASK also has competitive performance. It is better than DAAT and HATN by 0.8% and 1.43%, outperforms the Base Model by 0.22%. Although its average accuracy is slightly lower than DANN, it gives the best performance on the B→ K and K→B settings.

Moreover, we conduct experiments on 8 domain pairs involving the Airlines dataset. Table 2a shows the performance of our method on those settings. Although this setting is more difficult than domain pairs within Amazon-product-review dataset, we are still able to outperform Base on all of the domain pairs, by 1.13% on average. This shows that, by exploiting correlation among words, our method can help the model capture relationship between pivots and non-pivots even from distant domains. Besides, our method betters DANN by 0.39%. Although this is not a thick margin, it is non-trivial because 1) DANN itself is a very competitive method, 2) we do not expect to beat DANN by any larger margin, but to propose a strong baseline for domain adaption from a KG perspective, 3) our method is orthogonal to DANN and thus can be coupled with DANN to achieve even better performance.

Note that our Base model performs better than the previous state-of-the-art BERT-DAAT. This owes to hyperparameter tuning where BERT-DAAT uses 1e-2 for weight decay while we use 1e-4. It is not open-sourced so we cannot reproduce their results for better comparison. Nevertheless, it does not mean our performance gain comes from hyperparameter tuning as we analyze in the ablation studies below.

Ablation Study. As a sanity check, we performed experiments to study the effect of our choices of KI method and the knowledge graph. We compare PCKI versus normal KI method which inserts knowledge to every word possible, and our learnt KG versus the ConceptNet subgraph. From table 3, it can be observe that none of the configurations works other than PCKI plus learnt KG, our proposed method.

More importantly, to analyze how much each part of our method contribute to the performance gain, we conduct ablation study on three aspects: PCKI, SCL and pivot learning (Dynamic). We did experiments on 5 domain pairs as shown in table 2b. From the results, we have the following observations: 1) PCKI and SCL are both able to help DA independently, and PCKI generally works better than SCL, 2) Jointly applying them can further boost the performance, which implies that they complement each other in the ability of exploiting the correlation among pivots and non-pivots, 3) dynamically learning the pivots improve the performance by a large margin compared to using a set of pre-defined pivots. This supports that the learnt pivots are more beneficial to domain adaptation than the pre-defined ones.

5 Conclusion

In this paper, we proposed Domain-Adaptive text classification with Structured Knowledge (DASK) which elegantly marries pivot-based methods, knowledge graph and pretrained language models. It builds a knowledge graph from the target domain data to model the relationship between pivots and non-pivots. During training, DASK injects knowledge into the input sentence by means of pivot-induced crossdomain knowledge injection, and P-BERT is proposed to encode the knowledge-injected sentence. In this way, DASK learns domain invariant features by capturing the relationship between pivots and target domain-specific non-pivots. We conducted experiments on cross-domain sentiment classification and showed that DASK outperformed strong baselines in prediction accuracy by up to 2.9% on 20 domain pairs. Despite that our experiments are focused on sentiment classification, DASK generalizes to other cross-domain text classification tasks, as long as pivots are well-defined for the task. We reserve it as future work to extend DASK to other tasks.

(b)