Unsupervised Word and Dependency Path Embeddings for Aspect Term Extraction

1 Introduction

Aspect term extraction [Hu and Liu, 2004; Pontiki et al., 2014; 2015] aims to identify the aspect expressions which refer to the product's or service's properties (or attributes), from the review sentence. It is a fundamental step to obtain the fine-grained sentiment of specific aspects of a product, besides the coarse-grained overall sentiment. Until now, there have been two major approaches for aspect term extraction. The unsupervised (or rule based) methods [Qiu et al., 2011] rely on a set of manually defined opinion words as seeds and rules derived from syntactic parsing trees to iteratively extract aspect terms. The supervised methods [Jakob and Gurevych, 2010; Li et al., 2010; Chernyshevich, 2014; Toh and Wang, 2014; San Vicente et al., 2015] usually treat aspect term extraction as a sequence labeling problem, and conditional random field (CRF) has been the mainstream method in the aspect term extraction task of SemEval.

Representation learning has been introduced and achieved success in natural language processing (NLP) [Bengio et al.,

⇤Corresponding author: Ming Zhang

2013], such as word embeddings [Mikolov et al., 2013b] and structured embeddings of knowledge bases [Bordes et al., 2011]. It learns distributed representations for text in different granularities, such as words, phrases and sentences, and reduces data sparsity compared with the conventional one-hot representation. The distributed representations have been reported to be useful in many NLP tasks [Turian et al., 2010; Collobert et al., 2011].

In this paper, we focus on representation learning for aspect term extraction under an unsupervised framework. Besides words, dependency paths, which have been shown to be important clues in aspect term extraction [Qiu et al., 2011], are also taken into consideration. Inspired by the representation learning of knowledge bases [Bordes et al., 2011; Neelakantan et al., 2015; Lin et al., 2015] that embeds both entities and relations into a low-dimensional space, we learn distributed representations of words and dependency paths from the text corpus. Specifically, the optimization objective is formalized as w1 + r ⇡ w2. In the triple (w1↪w2↪ r), w1 and w2 are words, r is the corresponding dependency path consisting of a sequence of grammatical relations. The recurrent neural network [Mikolov et al., 2010] is used to learn the distributed representations of dependency paths. Furthermore, the word embeddings are enhanced by linear context information in a multi-task learning manner.

The learned embeddings of words and dependency paths are utilized as features in CRF for aspect term extraction. The embeddings are real values that are not necessarily in a bounded range [Turian et al., 2010]. We therefore firstly map the continuous embeddings into the discrete embeddings and make them more appropriate for the CRF model. Then, we construct the embedding features which include the target word embedding, linear context embedding and dependency context embedding for aspect term extraction. We conduct experiments on the SemEval datasets and obtain comparable performances with the top systems. To demonstrate the effectiveness of the proposed embedding method, we also compare our method with other state-of-the-art models. With the same feature settings, our approach achieves better results. Moreover, we perform a qualitative analysis to show the effectiveness of the learned word and dependency path embeddings.

The contributions of this paper are two-fold. First, we use the dependency path to link words in the embedding space for distributed representation learning of words and dependency paths. By this method, the syntactic information is encoded in word embeddings and the multi-hop dependency path embeddings are learned explicitly. Second, we construct the CRF features only based on the derived embeddings for aspect term extraction, and achieve state-of-the-art results.

2 RelatedWork

There are a number of unsupervised methods for aspect term extraction. Hu and Liu [2004] mine aspect term based on a data mining approach called association rule mining. In addition, they use opinion words to extract infrequent aspect terms. Using relationships between opinion words and aspect words to extract aspect term is employed in many follow-up studies. In [Qiu et al., 2011], the dependency relation is used as a crucial clue, and the double propagation method is proposed to iteratively extract aspect terms and opinion words.

The supervised algorithms [Li et al., 2012; Liu et al., 2015a] have been provided for aspect term extraction. Among these algorithms, the mainstream method is the CRF model. Li et al. [2010] propose a new machine learning framework on top of CRF to jointly extract positive opinion words, negative opinion words and aspect terms. In order to obtain a more domain independent extraction, Jakob and Gurevych [2010] train the CRF model on review sentences from different domains. Most of top systems in SemEval also rely on the CRF model, and hand-crafted features are constructed to boost performances.

The representation learning has been employed in NLP [Bengio et al., 2013]. Two successful applications are the word embedding [Mikolov et al., 2013b] and the knowledge embedding [Bordes et al., 2011]. The word embeddings are learned from raw textual data and are effective representations for word meanings. They have been proven useful in many NLP tasks [Turian et al., 2010; Guo et al., 2014]. The knowledge embedding method models the relation as a translation vector that connects the vectors of two entities. It optimizes an objective over all the facts in the knowledge bases, encoding the global information. The learned embeddings are useful for reasoning missing facts and facts extraction in knowledge bases.

As dependency paths contain rich linguistics information between words, recent works model it as a dense vector. Liu et al. [2015b] incorporate the vectors of grammatical relations (one-hop dependency paths) into the relation representation between two entities for relation classification task. Dependency-based word embedding model [Levy and Goldberg, 2014] encodes dependency information into word embeddings which is one similar work to ours. However, it implicitly encodes the dependency information, embedding the unit word + dependency path as the context vector. Besides, it only considers one-hop dependency paths and ignores multi-hop dependency paths. In this paper, we take the Dependency-based word embedding as a baseline method.

The approaches that learn path embeddings are investigated in knowledge bases. Neelakantan et al. [2015] provide a method that reasons about conjunctions of multi-hop relations for knowledge base completion. The implication of a path is composed by using a recurrent neural network. Lin et al. [2015] also learn the representations of relation paths. A path-constraint resource allocation algorithm is employed to measure the reliability of path. In this paper, we learn the semantic composition of dependency paths over dependency trees.

3 Method

In this section, we first present the unsupervised learning of word and dependency path embeddings. In addition, we describe how to enhance word embeddings by leveraging the linear context in a multi-task learning manner. Then, the construction of embedding features for CRF model is discussed.

3.1 Unsupervised Learning of Word and Dependency Path Embeddings

We learn the distributed representations of words and dependency paths in a unified model, as shown in Figure 1. We extract the triple (w1↪w2↪ r) from dependency trees, where w1 and w2 denote two words, the corresponding dependency path r is the shortest path from w1 to w2 and consists of a sequence of grammatical relations. For example, in Figure 1, given that service is w1 and professional is w2, we obtain the dependency path ⇤ conj - ⇤ dep -! ⇤ amod -! ⇤. We notice that considering the lexicalized dependency paths can provide more information for the embedding learning. However, we need to memorize more dependency path frequencies for the learning method (negative sampling) described in Section 3.3. Formally, the number of dependency paths is ♣Vword♣n-1 ♣Vdep♣n when we take n-hop dependency paths into consideration 1 . It is computationally expensive, even when n = 2 . Hence, in our paper, we only include the grammatical relations for modeling r which also shows the promising results in the experiment section.

The proposed learning model connects two words with the dependency path in the vector space. By incorporating the dependency path into the model, the word embeddings encode more syntactic information [Levy and Goldberg, 2014] and the distributed representations of dependency paths are explicitly learned. To be specific, this model requires that r is close to w2-w1 and we minimize the following loss function for the model learning:

where C1 represents the set of triples extracted from the dependency trees parsed from the text corpus, r is a sequence of grammatical relations that is denoted as (g1↪ g2↪ ▷▷▷↪ gn), n is the hop number of r, gi is the i-th grammatical relation in r, and p(r) is the marginal distribution for r. The given loss function encourages that the triple (w1↪w2↪ r) has a higher ranking score than the randomly chosen triple (w1↪w2↪ r0 ). The ranking score is measured by the inner product of vector r◁r0 and vector w2 -w1.

1Vword is the set of words and ♣Vword♣ is over 100,000 in our paper. Vdep is the set of grammatical relations and ♣Vdep♣ is about 50 in our paper.

We employ the recurrent neural network to learn the compositional representations for multi-hop dependency paths. The composition operation is realized with a matrix W:

where f is a hard hyperbolic tangent function (hTanh), [a; b] is the concatenation of two vectors, gi is the embedding of gi. We set h1 = g1 and recursively perform composition operation to obtain the final representation r = hn. Figure 1 (Left) illustrates the composition progress of the dependency path from service to professional. In this work, we train the triples whose hop numbers are less than or equal to 3. It is time-consuming to learn embeddings when we consider the triples with larger hop dependency paths.

3.2 Multi-Task Learning with Linear Context

We use the linear context to enhance word embeddings. The proposed method is based on the distributional hypothesis that words in similar contexts have similar meanings. Inspired by Skip-gram [Mikolov et al., 2013a], we enhance word embeddings by maximizing the prediction accuracy of context word c that occurs in the linear context of a target word w. The method is shown in Figure 1 (Right). Following the previous setting in [Mikolov et al., 2013b], we set the window size to 5. A hinge loss function, which has been proven to be effective in the cross-domain representation learning [Bollegala et al., 2015], is modified to the following

where, (c↪w) 2 C2 means that the word c appears in the linear context of the target word w, and p(w) is the marginal distribution for w. Every word has two roles, the target word and the context word of other target words. In this paper, we use different vectors to represent the two roles of a word.

2 We use dot product to calculate the similarity of words without using the softmax function that is employed in Skip-gram.

3.3 Model Training

The negative sampling method [Collobert et al., 2011; Mikolov et al., 2013b] is employed to train the embedding model. Specifically, for each pair (w↪ c), we randomly select kW words c0 that do not appear in the linear context window of the target word w. These randomly chosen words are sampled based on the marginal distribution p(w) and the p(w) is estimated from the word frequency raised to the 34 power [Mikolov et al., 2013a] in the corpus. We use the same method to optimize the Equation (1). The difference is, we sample path r0 separately according to the hop number and randomly select kRn dependency paths for the n-hop path r. The selected dependency paths r0 have the same hop number with r. We experimentally find that kW = 5 is a trade-off between the training time and performance. Similarly, kR1 , kR2 and kR3 are set to 5, 3 and 2.

The model is trained by back-propagation. We use asynchronous gradient descent for parallel training. Following the strategy for updating learning rate [Mikolov et al., 2013a], we linearly decrease it over our training instances. The initial learning rate is set to 0.001.

3.4 Aspect TermExtraction with Embeddings

We use the CRF model for aspect term extraction, and the features are constructed based on the learned embeddings. Besides the target word embedding features, we also utilize the features encoding the context information, (1) the linear context is within a fixed window of the target word in the linear sequence; (2) the dependency context is directly related with the target word in the dependency tree. These context features are used as complementary information to help label aspect terms.

Given a sentence s = (w1↪w2↪ ▷▷▷↪w♣s♣) and its dependency tree, we denote C(wi) as the function extracting the features for word wi. In our case, C(wi) = [Emb(wi)↪ Cw(wi)↪ Cd(wi)], where Emb(wi) represents the embedding of word wi, Cw(wi) and Cd(wi) refer to the linear context embedding and the dependency context embedding respectively. These extracted embeddings are concatenated as CRF features.

Based on the assumption that the label of a word mainly depends on itself and its neighboring words [Collobert et al.,

2011], the linear context is leveraged as additional information to help tag a word in NLP tasks. In our work, Cw(wi) is defined as:

where b⇤c is a floor function, len represents the window size of linear context and is set to 5 in this paper which follows [Collobert et al., 2011].

The dependency context captures syntactic information from the dependency tree. We collect the pairs (dependency path, context word) as the dependency context. They are directly linked with the word to be labeled. We limit the max hop number of dependency path in pairs to 3. For example, the dependency context of target word staff in Figure 1 is (⇤ dep -! ⇤, waiter), (⇤ conj -! ⇤, service), (⇤ cc -! ⇤, and), (⇤ dep -! ⇤ advmod -! ⇤, very) and (⇤ dep -! ⇤ amod -! ⇤, professional). We sum the vectors of dependency path and context word as the embedding for the pair, and average the embeddings of collected pairs to derive the dependency context embedding features Cd(wi).

The goal of embedding discretization is to map the realvalued embedding matrix M d⇥num R into the discrete-valued embedding matrix M d⇥num D , where d is the dimension of embeddings and num is the size of vocabulary. Formally, the operation of embedding discretization is shown as follows:

where max(M i⇤ R ) and min(M i⇤ R ) are the maximum and minimum respectively in the i-th dimension, l is the number of discrete interval for each dimension. By this transformation, we obtain CRF features for aspect term extraction, without losing much information encoded in the original embeddings.

4 Experiment

4.1 Dataset and Setting

We conduct our experiments on the SemEval 2014 and 2015 datasets. The SemEval 2014 dataset is from two domains, laptop (D1) and restaurant (D2) and the SemEval 2015 dataset only includes the restaurant domain (D3). The details of datasets are described in Table 1, and the F1 score is used as the evaluation metric.

We also introduce two in-domain corpora for learning distributed representations of words and dependency paths. The corpora contain Yelp dataset 3 and Amazon dataset 4 which are in-domain corpora for restaurant domain and laptop domain respectively. As the size of laptop reviews is small in Amazon corpus, we add the reviews of other similar products in laptop corpus. These products consist of Electronics, Kindle, Phone and Video Game. All corpora are parsed by using the Stanford corenlp 5 . The derived tokens are converted to lowercase, the words and dependency paths that appear less than 10 are removed. The details of unlabeled data are shown in Table 2.

3 https://www.yelp.com/academic dataset

4 https://snap.stanford.edu/data/web-Amazon.html

In order to choose l and d, we use 80% sentences in training data as training set, and the rest 20% as development set. The dimensions of word and dependency path embeddings are set to 100. Larger dimensions get similar results in the development set but need more training time. l is set to 15 that performs best in the development set.

We use an available CRF tool 6 for tagging aspect terms, and set the parameters with default values. The only exception is the regularization option. L1 regularization is chosen as it obtains better results than L2 regularization in the development set.

4.2 Result and Analysis

We compare our method with the following methods:

1. Naive: Token in test sentences is tagged as an aspect term, if it is in the dictionary that contains all the aspect terms of training sentences.
2. Baseline Feature Templates: Most of features used in the Baseline Feature Templates are adopted from the NER Feature Templates [Guo et al., 2014] which are described in Table 3.
3. IHS RD: IHS RD is the top system in D1 which also relies on CRF with a rich set of lexical, syntactic and statistical features. Unlike other systems in SemEval 2014 and our work, IHS RD trains the CRF model on the reviews of both the restaurant domain and laptop domain.
4. DLIREC: DLIREC is the top system in D2 which relies on CRF with a variety of lexicon, syntactic, semantic features derived from NLP resources. In terms of features based on the dependency tree, they use the head word, head word POS and one-hop dependency relation of the target word. Besides, different from our work, DLIREC utilizes the corpora of Yelp and Amazon reviews to derive cluster features for words.
5. EliXa: EliXa is the top system in D3 which addresses the problem using an averaged perceptron with a BIO tagging scheme. The features used in EliXa consist of n-grams, token shape (digits, lowercase, punctuation, etc.), previous prediction, n-gram prefixes and suffixes, and word clusters derived from additional data (Yelp for Brown and Clark clusters; Wikipedia for word2vec clusters).

5 http://nlp.stanford.edu/software/corenlp.shtml

6 https://crfsharp.codeplex.com/

The statistical significance tests are calculated by approximate randomization, as described in [Yeh, 2000] and the results are displayed in Table 4. Compared with Baseline Feature Templates, the top systems (DLIREC, IHS RD and EliXa) obtain the F1 score gains of 3%, 2% and 5% in D1, D2 and D3 respectively. It indicates that high-quality feature engineering is important for aspect term extraction. In terms of embedding-based features, the target word embedding performs worse than other feature settings. Both W+L and W+D improve performances, as they capture the additional context information. By combining embedding features of target word, linear context and dependency context, we achieve comparable performance with the best systems in SemEval 2015, and outperform the best systems in SemEval 2014. It shows that, (1) with the features based on distributed representations, we can achieve state-of-the-art results in aspect term extraction; (2) the context embedding features offer complementary information to help detect aspect terms. Additionally, adding the traditional features (Baseline Feature Templates), we get better performances. It can be explained by that the traditional features capture the surface information (e.g, stem, suffix and prefix) which is not explicitly encoded in the embedding features.

4.3 Comparison of Different EmbeddingMethods

We conduct experiments to evaluate our method of word and dependency path embeddings (WDEmb) with state-of-the-art embedding methods.

The baseline embedding algorithms include Dependency Recurrent Neural Language Model (DRNLM) [Piotr and Andreas, 2015], Skip-gram, CBOW [Mikolov et al., 2013a] and Dependency-based word embeddings (DepEmb) [Levy and Goldberg, 2014]. The DRNLM predicts the current words given the previous words, aiming at learning a probability distribution over sequences of words. Note that, besides the previous words, DRNLM also uses one-hop dependency relation as the additional features to help the model learning. Skipgram learns word embeddings by predicting context word given the target word, while CBOW learns word embeddings by predicting current word given the context words. DepEmb learns word embeddings using one-hop dependency context. It is similar to our method to encode functional information into word embeddings. But the dependency path considered is one-hop rather than multi-hop.

The implementations of baseline models are publicly released 7 , and are used to train embeddings on the same corpora as our model. We set the parameters of baselines to ensure a fair comparison 8 . As the baselines do not learn the distributed representation of multi-hop dependency path, we compare our model with them in the feature settings (W and W+L) where only word embeddings are used. The statistical significance tests are calculated by approximate randomization and the results are shown in Table 5.

Among these baselines, the DepEmb learns word embeddings by dependency context, and performs best in both domains. It indicates that the syntactic information is more desirable in aspect term extraction. WDEmb outperforms DepEmb and the reasons are that, (1) compared with DepEmb that embeds a unit word + dependency path into a vector, WDEmb learns the word and dependency path embeddings separately, and alleviates the data sparsity problem; (2) multi-hop dependency paths are taken into consideration in WDEmb which help encode more syntactic information. DRNLM performs worst among these baselines, though it utilizes the dependency relation in their model. The reason is that the DRNLM focuses on language modeling and does not include subsequent words which are used in other embedding baselines, as the context.

We also employ two methods for qualitative analysis of the learned embeddings, (1) we select the 6 most similar words (by cosine similarity) for each aspect word. Results are shown in Table 6; (2) we design some queries made up of one word and one dependency path, such as delicious +

7 DepEmb (https://levyomer.wordpress.com/software/) Skip-gram and CBOW (https://code.google.com/p/word2vec/) DRNLM (https://github.com/piotrmirowski/DependencyTreeRnn) .

8 The dimensions and interval numbers of baselines are tuned in the development set. The interval numbers of DRNLM, Skip-gram and CBOW are set to 10, and the interval number of DepEmb is set to 15. All dimensions of baselines are set to 100. The window sizes of Skip-gram and CBOW are set to 5, which is the same as our model of multi-task learning with linear context.

⇤ amod - ⇤ and delicious + ⇤ acomp - ⇤ nsubj -! ⇤ , to find similar words. Results are displayed in Table 7.

Table 6 shows that the similar words derived from both Skip-gram and CBOW, are less similar with the defined aspect words in syntactic function, such as, fast, estimated and speedy. The reason lies in that both the Skip-gram and CBOW models learn word embeddings based on linear context and capture less functional similarity between words than the DepEmb and WDEmb [Levy and Goldberg, 2014]. Table 6 also shows that the similar words induced fromWDEmb are more topical similar with the aspect words than DepEmb (e.g., racquetball, everything, smog). The reason is that WDEmb encodes more topical information into word embeddings by multi-task learning based on linear context. Besides, we can obtain sensible words by the queries made up of word + dependency path in Table 7. It reveals that our approach of word and dependency path embeddings is promising.

4.4 Effect of Parameters

In this section, we investigate the effect of interval number l and the dimension of embeddings d in the feature setting of W+L+D. All the experiments are conducted on the development set.

We vary l from 3 to 21, increased by 3. The performances in three datasets are given in Figure 2. We can see that the performances increase when l < 15, and the models achieve the best results in two domains when l = 15. However, when l > 15, the performances decrease. We also vary d from 25 to

225, increased by 25. The results are shown in Figure 2. The performances increase when d < 100. But they keep stable when d ≥ 100. We can infer that 100 is a trade-off between the performance and training time.

5 Conclusion and FutureWork

In this paper, we propose to learn the distributed representations of words and dependency path in an unsupervised way for aspect term extraction. Our method leverages the dependency path information to connect words in the embedding space. It is effective to distinguish words with similar context but different syntactic functions which are important for aspect term extraction. Furthermore, the distributed representation of multi-hop dependency paths are also derived explicitly. The embeddings are then discretized and used as features in the CRF model to extract aspect terms from review sentences. With only the distributed representation features, we obtain comparable results with the top systems in aspect term extraction on the benchmark dataset. Recently, [Wang et al., 2015a] propose to incorporate knowledge graph to represent the documents or short text [Wang et al., 2015b; 2016]. This could be interesting to combine this representation with our embedding methods to further improve the performance.

6 Acknowledgments

This paper is partially supported by the National Natural Science Foundation of China (NSFC Grant Numbers 61272343, 61472006), the Doctoral Program of Higher Education of China (Grant No. 20130001110032) as well as the National Basic Research Program (973 Program No. 2014CB340405).