Learning Topic Models by Neighborhood Aggregation

1 Introduction

Topic models are widely used in both academia and industry owing to their high interpretability and modular structure [Blei et al., 2003]. The highly interpretable nature of topic models allows one to gain important insights from a large collection of documents while the modular structure allows researchers to bias topic models to reflect additional information, such as supervisory signals [Mcauliffe and Blei, 2008], covariate information [Ramage et al., 2009], time-series information [Park et al., 2015] and pre-trained word embedding vectors [Nguyen et al., 2015].

However, inference in a highly structured graphical model is not an easy task. This hinders practitioners from extending the model to incorporate various information besides text of their own choice. Furthermore, adding a nonlinear output function makes the model even more difficult to train.

The present paper shows that topic modeling with pre-trained word embedding vectors can be viewed as implementing a neighborhood aggregation algorithm [Hamilton et al., 2017] where the messages are passed through a network defined over words. From the network view of topic models, Latent Dirichlet Allocation (LDA) [Blei et al., 2003] can be thought of as creating a network where nodes correspond to words in a document and edges corresponds to either a relationship describing co-occurring words in a document or a relationship describing the same word in the corpus. The network view makes it clear how a topic label configuration of a word in a document is affected by neighboring words defined over the network and adding supervisory signals amounts to adding new edges to the network. Furthermore, by replacing the message passing operation with a differentiable neural network, as is done for neighborhood aggregation algorithms [Hamilton et al., 2017], we can learn the influence of the pre-trained word embedding vectors to the topic label configurations as well as the effect of the same label relationship, from the supervisory signal.

Our contribution is summarized as follows.

• We show that topic modeling with pre-trained word embedding vectors can be viewed as implementing a neighborhood aggregation algorithm where the messages are passed through a network defined over words.
• By exploiting the network view of topic models, we propose a supervised topic model that has an adaptive message passing where the parameters governing the message passing and the parameters governing the influence of the pre-trained word embedding vectors to the topic label configuration is learned from the supervisory signal.
• Our model includes a nonlinear output function connecting document to their corresponding supervisory signal.
• Our approach outperforms state-of-the-art supervised LDA implementation [Katharopoulos et al., 2016; Zhu et al., 2009] on a wide variety of datasets regarding predictive performance and gives a comparative performance concerning topic coherence.

2 Notations

We briefly summarize the basic notations used throughout the paper. Let 1 ≤ d ≤ D, 1 ≤ w ≤ W, 1 ≤ k ≤ K and 1 ≤ s ≤ S respectively denote the document index, word index, topic number index and label index. We denote by xd,w the number of word counts for a particular document-word pair (d,w). The task of topic modeling is to assign the average topic label configuration z = zk d,w from the observed document word count matrix X = {xd,w} where the average topic label configuration is defined as zk d,w := Σxd,w i=1 zk d,w,i xd,w for all document-word pairs (d,w) in the corpus. The average topic label configuration sums to 1 over topic index 1 ≤ k ≤ K (i.e., ΣKk=1 zk d,w = 1) and one of the tasks in this paper is to calculate the messages (which we denote µkd,w ) that can approximate all zk d,w in the corpus. We denote by µkd,w (t) the estimated message at iteration t. Omission of the subscript k (i.e., µd,w(t)) simply implies the vectorized form of µkd,w (t)s (i.e., [µ1d,w (t) · · · µKd,w (t)]).

We use θd to denote the topic proportion distribution of document d and φw to denote the topic distribution. We also use vi to denote node attribute information attached to node i.

3 Background

3.1 Factor Graph Approach to LDA

From the perspective of topic modeling, our approach is related to the work of [Zeng et al., 2013], who reframed LDA as a factor graph and used the message passing algorithm for inference and parameter estimation.

The classical Bayesian network plate diagram representation of LDA is presented in Figure 1. The joint distribution of the model can be summarized as

where α and β denote hyperparameters of the Dirichlet prior distribution. Meanwhile, by designing the factor function as

and

a factor graph representation of LDA (i.e., Figure 2) can be summarized as

Equation (4) shows that the factor graph representation encodes exactly the same information as Eq. (1). From the factor graph representation, it is possible to reinterpret LDA using a Markov random field framework and thus infer messages for words in a document using loopy belief propagation. The essence of their paper can be summarized by a message updating equation of the form

Our goal in this paper is to connect the factor graph approach of LDA with the neighborhood aggregation algorithm.

3.2 Neighborhood Aggregation Algorithm

The goal of node embedding is to represent nodes as low-dimensional vectors summarizing the positions or structural roles of the nodes in a network [Hamilton et al., 2017]. The neighborhood aggregation algorithm is a recently proposed algorithm used in node embedding [Dai et al., 2016; Hamilton et al., 2017], which tries to overcome limitations of the more traditional direct encoding [Perozzi et al., 2014] approaches. The heart of a neighborhood aggregation algorithm lies in designing encoders that summarize information gathered by a node's local neighborhood. In a neighborhood aggregation algorithm, it is easy to incorporate a graph structure into the encoder, leverage node attribute information and add nonlinearity to the model. The parameters defining the encoder can be learned by minimizing the supervised loss [Dai et al., 2016; Hamilton et al., 2017]. In [Dai et al., 2016], it was shown that these algorithms can be seen as replacing message passing operations with differential neural networks. We refer to [Dai et al., 2016] for a more detailed explanation.

The essence of neighborhood aggregation algorithms is characterized by three steps: aggregation, combination and normalization. In the aggregation step, we gather information from a node's local neighborhood. A concatenation, sum-based procedure or elementwise mean is usually employed. In the combination step, we combine a node's attribute information with the gathered information from the aggregation step. After passing the combined information through a nonlinear transformation, we normalize the message so that the new updated messages can be further used by neighboring nodes.

The overall process is succinctly summarized as

where µi(t+1) denotes the message of node i at iteration t+ 1; Nei(i) denotes neighboring nodes of node i; Agg, Comb and Norm respectively denote aggregation, combination and normalization functions; WA and WC denote parameters used in the combination step (e.g., Comb(vi, xi) = WCvi + WAxi) and σ is an elementwise nonlinear transformation. With enough update iterations, the model converges and the desired messages are learned.

Suppose that we are given a training dataset D = {X, y1:D}, where X is the observed network and y1:D is the supervisory signal attached to each node. For a classification problem {yd ∈ 1, · · · , S}, we use a simple neural network with softmax output to transform the learned messages to probabilities and to minimize the cross-entropy loss. This is summarized as

where SA, SB, SC are either H1 ×K, H2 ×H1 and S ×H3 weight matrices transforming the messages and TA, TB and TC are bias vectors of either size H1, H2 and S, ysd = 1 if the label of node d is s and zero otherwise and σ is an elementwise nonlinear transformation. We use the classic sigmoid function in this paper.

For a regression problem yn ∈ R, parameters can be learned by minimizing the sum-of-squared loss:

4 Model

4.1 LDA and Neighborhood Aggregation

The message-passing equation (Eq. (5)) of LDA can be seen as taking an elementwise product of two neighborhood aggregation operations and normalizing the result for probabilistic interpretation. We clarify this point with an illustrative example. Figure 3 shows a hypothetical corpus with three documents "platypus, clinic, ill", "platypus, Australia, east" and "platypus, eat, shrimp". Each document-word pair in the corpus has an associated message like the upper-right vector depicted next to the word "ill" in document 1. The red bold lines indicate a relationship describing co-occurring words in a document while the blue dotted lines indicate a relationship describing the same word in the corpus.

Suppose that we want to update the message for the word "platypus" found in document 2. According to the message passing equation (Eq. (5)), this message is updated by the elementwise product of messages gathered from edges describing co-occurring words in document 2 (as denoted by the red bold arrows) and messages gathered from edges describing the same word in the corpus (as denoted by the blue dotted arrows). In fact, assuming that the aggregation function in a document is

where Neid(d,w) denotes all other words in the same document, there are no additional node features to combine and, using Aggd to denote [Agg1 d · · ·AggK d ], the neighborhood aggregation operation from co-occurring words in a document can be written as

where NormK is defined to be a normalization function dividing by the sum over topics 1 ≤ k ≤ K.

Following similar reasoning, it is assumed that the aggregation function for the edges describing the same word in the corpus is

where Neiw(d,w) denotes all document-word pairs in the corpus that is the same word as w (i.e. nodes in the blue dashed square shown in Figure 3) and there are no additional node features to combine (such as pre-trained word embedding vectors described below). The neighborhood aggregation operation for this neighborhood system can be summarized as

where NormW is defined as a normalization function dividing by the sum over words 1 ≤ w ≤ W.

The message update equation for a document-word pair (d, w) can thus be summarized by

where ⊙ denotes elementwise multiplication. The messages are normalized in the final step so that they can be used as a proper probability distribution.

This explanation shows that the message passing equation (Eq. (5)) of LDA can be seen as an element-wise product of two neighborhood aggregation operations with an additional normalization step for probabilistic interpretation.

4.2 Supervised LDA and Neighborhood Aggregation

We extend the above formulation to incorporate supervisory signals. Supervisory signals such as label information can be thought of as defining an additional neighborhood system in the above formulation. We therefore add edges reflecting this additional neighborhood system. For example, in a classification problem where we have label information for each document, we add edges among document-word pairs that belong to other documents with the same label. Figure 4 illustrates the same hypothetical corpus as shown in Figure 3. The green dashed arrows are the new edges added by the supervisory signal.

We define the aggregation function for this neighborhood system as

where the indicator function 1l(d′ )=l(d) is used to select documents with the same label and l is a function that maps a document index d to its corresponding label s. Neighborhood systems can easily be extended to the regression case where documents do not necessarily have exactly the same output value (e.g., real numbers). In this case, we replace the indicator function as 1|l(d′ )-l(d)|<ϵ for a given ϵ > 0.

From this formulation, the neighborhood aggregation operation from the supervisory signal can be written as

where Neis(d,w) denotes the neighborhood system of s for document-word pair (d,w) and Ws denotes a diagonal matrix with positive entries only. We train Ws using a supervisory signal as is done in neighborhood aggregation algorithms [Hamilton et al., 2017].

In the standard sum-product algorithm, we take the product of all messages from factors to variables. However as was noted in [Zeng et al., 2013], a product operation cannot control the messages from different sources (i.e. supervised part and unsupervised part) and we, therefore, take the weighted sum of the two neighborhood aggregation operations to rebalance the two effects. The whole message updating equation can be summarized as

where η controls the strength of the supervisory signal in inferencing the topic label configuration. This type of rebalancing of the effect from the supervised part and unsupervised is also taken in the traditional supervised LDA approaches where it is well known that the effect of the supervised part is reduced for documents with more words in the standard formulation [Katharopoulos et al., 2016]. Hence, Eq. (16) is quite a standard technique, contrary to what it might appear at first glance.

4.3 PretrainedWord Embedding Vectors

It is natural to assume that there is an association between pre-trained word embedding vectors [Mikolov et al., 2013] and topics, so long as topic models are used to summarize the semantic information in a document. Hence, learning the association between pre-trained word embedding vectors and topics is important especially at the time of testing when there might be many unseen words in the held-out documents. If these unobserved words are included in the pre-trained word embedding dictionary and we have trained a function mapping the word embedding vectors to the topics, we can leverage the pre-trained word embedding vectors to predict the topic label configuration more accurately. This issue was addressed in several recent papers [Das et al., 2015].

Within the neighborhood aggregation framework, the pre-trained word embedding vectors can be modeled as a node attribute vector vi in the notation of Eq. (6). We define the word embedding in the topic distribution transformation as

where g is a function that maps the document-word index to the word index in the pre-trained word embedding vector dictionary, WC is the K×E weight matrix transforming word embedding vectors (which we assume to have dimension E) to the topic and σ is an element-wise nonlinear transformation.

Using the transformed vector ug(d,w), we define the combine function as

The entire message updating equation is now summarized as

This update equation is a particular instance of Eq. (6) assuming identity mapping for σ and applying elementwise multiplication for the combine function.

4.4 Training

Inference of the messages (i.e., µkd,w ) and estimation of the parameters (i.e., WS, WC, SA, SB, SC, TA, TB and TC) can be performed by alternately inferring the topic label configurations given parameters of the model and updating the parameters minimizing the supervised loss given a topic label configuration. In each iteration, we first infer the topic label configuration for the randomly sampled document-word pairs, then update all the parameters by creating a mini-batch of the document-word pairs and update them using standard stochastic gradient descent technique. The number of the document-word pairs are set to 5000. For the regularization parameter governing WS and WC, we set it to 0.001, and for the output function, we set a dropout probability of 0.5 for regularization. We also set η in our model to be 0.2 for the economic watcher survey and 0.05 for the rest, and set the number of hidden units in Eq.(7) to be H1 = 50 and H2 = 50. We repeat this iteration until convergence. We could also use sparse message passing in the spirit of [Pal et al., 2006] to enhance the coherence of the topics, but we leave this for future works.

5 Experiments

5.1 Datasets

We conduct experiments showing the validity of our approach. We use three datasets in our experiments. Two of our datasets are data of multiclass-label prediction tasks focusing on sentiment (economic assessments and product reviews) while the other dataset is data of a binary label prediction task focusing on subjectivity. We summarize the datasets below.

• The economic watcher survey, in the table abbreviated as EWS, is a dataset provided by the Cabinet Office of Japan 1 . The purpose of the survey is to gather

1 The whole dataset is available at http://www5.cao.go.jp/keizai3/ watcher index.html

economic assessments from working people in Japan to comprehend region-by-region economic trends in near real time. The survey is conducted every month, and each record in the dataset consists of a multi-class label spanning from 1 (i.e., the economy will get worse) to 5 (i.e., the economy will get better) and text describing the reasons behind a judgment. We use a subset of this dataset describing the assessment of future economic conditions. We use the 3000 most frequently used words in the corpus and further restrict our attention to records that have more than a total of 20 words in the pre-trained word embedding dictionary to perform a fair comparison among the models 2 . We randomly sample 5000 records for training, development, and testing. For the pre-trained word embedding vectors describing Japanese words, we use word2vec vectors provided by [Suzuki et al., 2016] 3 .

• Amazon review data are a dataset of gathered ratings and review information [McAuley et al., 2015]. We use a subset of the five-core apps for android dataset 4 and use the 3000 most frequently used words in the corpus. We randomly sample 5000 records for training, development, and testing focusing on reviews that have more than 20 words excluding stop words in the pre-trained word embedding dictionary 5 . For the pre-trained word embedding vectors describing English words, we use the word2vec embedding vectors provided by Google [Mikolov et al., 2013].
• Subjectivity data are a dataset provided by [Pang and Lee, 2004], who gathered subjective (label 0) and objective (label 1) snippets from Rotten Tomatoes and the Internet Movie Database. We focus on snippets that have more than nine words and sample 1000 snippets each for training, development and testing 6 . Other settings are the same as those of the Amazon review data.

5.2 Classification Performance

The main goal in this section is to compare the classification performance of our proposed models with that of the state-of-the-art supervised LDA implementations (which we denote as SLDA[Katharopoulos et al., 2016] and MedLDA[Zhu et al., 2009]), nonlinear prediction using pre-trained word embedding vectors only (which we denote as WE-MLP) and topic models incorporating pre-trained word embedding vectors (which we denote as LFLDA[Nguyen et al., 2015] and LFDMM[Nguyen et al., 2015]. The difference between LFLDA and LFDMM is that LFDMM assumes that all the words in a document share the same topic. We use the default settings defined in the codes provided by the authors. We also fix the number of topics to 20 for all experiments performed in this section. We compare the performance of these models which we denote as NA-WE-SLDA.

2 We use standard morphological analysis software [Kudo et al., 2004] to annotate each word.

3 These are available for download at http://www.cl.ecei.tohoku. ac.jp/∼m-suzuki/jawiki vector/

4 The term five-core implies a subset of the data in which there are at least five reviews for all items and reviewers.

5 The whole dataset is available at http://jmcauley.ucsd.edu/data/ amazon/.

6 The whole dataset is available at http://ws.cs.cornell.edu/ people/pabo/movie-review-data.

For WE-MLP, we use the average of all the pre-trained word embedding vectors found in a document as the feature describing a document and we connect this feature to our supervisory signal using a simple multilayer perceptron with softmax output. For LFLDA and LFDMM we use the document topic distributions as the feature describing a document and also connect this feature to our supervisory signal using a simple multilayer perceptron with softmax output. Parameters (e.g., the number of hidden units) of these models was found by utilizing the development dataset.

Table 1 summarizes the results. Cross-entropy denotes the cross-entropy loss and accuracy denotes the proportion of correct predictions in the held-out documents. For MedLDA we only report accuracy because the method does not use the softmax output function. We see that NA-WE-SLDA is the best performing model in all cases and sometimes beats SLDA and MedLDA substantially, especially for multi-label classification tasks when measuring with accuracy.

5.3 Comparison with Deep Learning

Here we compare our model with the state-of-the-art deep learning method [Kim, 2014] varying the number of documents from 1000 to 5000. We chose the filter sizes, number of filters and dropout probability by using development data set. We see that for smaller document size our method performs better than the deep learning method, but as the number of the document increases the power of deep learning catches up, and our model is out-beaten as expected. However, our performance for a smaller number is worth noting.

5.4 Topic Coherence

Our goal in this section is to compare topic coherence among the supervised topic models that we evaluated in the previous section and the plain LDA model [Blei et al., 2003]. The purpose of this experiment is that adding complex structures to topic models might have a side effect of downgrading interpretability [Chang et al., 2009]. We want to examine whether our models sacrifice interpretability to achieve better predictive performance.

Traditionally, topic coherence is evaluated on the basis of the perplexity measure, which is basically a measure of how likely a model predicts words in the held-out documents. However, as was shown in [Chang et al., 2009], higher perplexity does not necessarily correlate to better human interpretability and various topic coherence measures have been proposed to address this issue [R¨oder et al., 2015]. The present paper uses the CV measure proposed in [R¨oder et al., 2015], which is a coherence measure that combines existing basic coherence measures. It was shown that the CV measure is the best coherence measure among 237,912 coherence measures compared in their paper [R¨oder et al., 2015]. We employ the source code provided by the authors 7 to calculate the coherence measure in our experiment focusing on the top 15 words in the topic distribution 8 .

Table 2 summarizes the results. We see that our method has a price to pay for its better predictive accuracy. For all the data set, our model shows inferior coherence compared to other methods. Our second observation is that except for the economic watcher survey dataset and our model, models with word embedding vectors slightly outperforms the model without word embedding vectors. Our model even utilizing word embedding vectors, could not compensate for the loss made from the nonlinear output function and lack of sparsification, but still provides comparable performance.

7 The code is available at https://github.com/dice-group/Palmetto

8 For the economy watcher survey, we translated words from Japanese to English.

To gain further insights into our learned topics, in Table 4, Table 5 and Table 6, we report the top six words of the 20 topics learned by LDA, SLDA, and NA-WE-SLDA respectively for the economic watcher survey dataset 9 . First of all we see several topics that does not appear in LDA such as "ecology, payment, automobile, subsidy, sale, car" (i.e., Topic 1) in SLDA and "ecology, subsidy, car, payment, last, year" (i.e., Topic 8) in NA-WE-SLDA highlighting the effect of the supervisory signals. It is also worth mentioning that although the topic corresponding to the great east Japan earthquake does seem to appear in LDA as in Topic 7 (i.e., "customer, last, year, situation, disaster, east"), the topic corresponding to the earthquake event is easier to spot in SLDA and NA-WE-SLDA, where the former focuses more on the impact on the car industry (i.e., Topic 13) and the latter focuses more on the nuclear incident (i.e., Topic 9). The difference between SLDA and NA-WE-SLDA could be seen in topics that focus on temporary dispatch workers 10 . While Topic 15 in SLDA seems to focus on dispatch workers in small and medium-sized companies, Topic 16 in NA-WE-SLDA focuses more on the end of the fiscal year which corresponds to the timing when dispatch workers are more susceptible to layoffs.

6 Conclusion

We showed that topic modeling with word embedding can be viewed as implementing a neighborhood aggregation algorithm where the messages are passed through a network defined over words. By exploiting the network view of topic models, we proposed new ways to model and infer supervised topic models equipped with a nonlinear output function. Our extensive experiments performed over a range of datasets showed the validity of our approach.

Acknowledgements

This work was supported by JSPS KAKENHI Grant Number JP17K17663. We thank Glenn Pennycook, MSc, from Edanz Group (www.edanzediting.com/ac) for editing a draft of this manuscript.

9 Stop words are omitted.

10 Temporary workers sent by staffing agencies.