Distraction-Based Neural Networks for Modeling Document

1 Introduction

Modeling the meaning of text lies in center of natural language understanding. Distributed representation learned with neural networks has recently shown to be effective in modeling fine granularities of text, including words [Collobert et al., 2011; Mikolov et al., 2013; Chen et al., 2015], phrases [Yin and Sch¨utze, 2014], and arguably sentences [Socher et al., 2012; Irsoy and Cardie, 2014; Kalchbrenner et al., 2014; Tai et al., 2015; Zhu et al., 2015a; 2015b].

Whether and how such an approach can be extended to help model larger spans of text, e.g., documents, is intriguing, and further investigation would still be desirable, although there has been interesting research conducted recently along this line [Li et al., 2015; Hu et al., 2015; Wang and Cho, 2015; Hermann et al., 2015]. A typical problem of documentlevel modeling is automatic summarization [Mani, 2001; Das and Martins, 2007; Nenkova and McKeown, 2013], in which computers generate summaries for documents, based on their shallow or deep understanding of the documents.

If one regards the process of representing input documents, generating summaries, and the interaction between them to be a (complicated) function, fitting such a function could expect to have a large-scale annotated dataset to estimate a large set of parameters, while on the other hand, hardcoding summarization knowledge (in different forms) or limiting the number of model parameters (e.g., as in many extractive summarization models) are often adopted when there are no enough training data. This work explores the former direction and utilizes relatively large datasets [Hu et al., 2015; Hermann et al., 2015] to train neural summarization models. In general, neural networks, as universal approximators, can fit very complicated functions and have shown to be very effective on many problems recently.

Understanding the input documents and generating summaries are both challenging. On the understanding side, much recent work seems to have suggested that distributed representations (often vectors) by themselves may not be adequate for representing sentences, let along with longer documents. Additional modeling such as soft or hard attention has been applied to retrospect subsequences or even words in input text to remedy the limits, which has shown to improve performances of different tasks such as those discussed in [Bahdanau et al., 2014; Luong et al., 2015; Rush et al., 2015] among others. We regard this to be a mechanism that provides a connection between input document modeling (encoding) and summary generating (decoding), which could model a level of cognitive controls-human summarizers themselves often move between the input documents and summaries when they summarize a document.

We consider this control layer to be important, and in this paper we focus on better designing this control layer for summarization. We propose neural models to train computers not just to pay attention to specific regions and content of input documents with attention models, but also distract them to traverse between different content of a document so as to better grasp the overall meaning for summarization.

Without engineering any features, we train the models with two large datasets. The models achieve the state-of-the-art performance and they significantly benefit from the distraction modeling, particularly when input documents are long. We also explore several technologies that have been applied to sentence-level tasks and extend them to document summarization, and we present in this paper the technologies that showed to help improve the summarization performance. Even when it is applied onto the models that have leveraged these technologies, the distraction models can further improve the performance significantly. In general, our models here aim to perform abstractive summarization.

2 Related work

Distributed representation Distributed representation has shown to be effective in modeling fine granularities of text as discussed above. Much recent work has also attempted to model longer spans of text with neural networks [Li et al., 2015; Hu et al., 2015; Lin et al., 2015; Wang and Cho, 2015; Hermann et al., 2015]. This includes research that incorporates document-level information for language modeling [Wang and Cho, 2015; Lin et al., 2015] and that answers questions [Hermann et al., 2015] by comprehending input documents with attention-based models. More relevant to ours, the work of [Li et al., 2015] learned distributed representation for short documents with the averaged length of about a hundred word tokens, although the objective is not summarization. Summarization typically faces documents longer than those, and summarization may be more necessary when documents are long. In this paper, we propose neural models for summarizing typical news articles with up to thousands of word tokens. We find it is necessary to enable computers not just to pay attention to specific content of input documents with attention models, but also distract them to traverse between different content so as to better grasp the overall meaning for summarization, particularly when documents are long.

Neural summarization models Automatic summarization has been intensively studied for both text [Mani, 2001; Das and Martins, 2007; Nenkova and McKeown, 2013] and speech [Zhu and Penn, 2006; Zhu et al., 2010]. Most stateof-the-art summarization models have focused on extractive summarization, although some efforts have also been exerted on abstractive summarization. Recent neural summarization models include the recent efforts of [Rush et al., 2015; Lopyrev, 2015; Hu et al., 2015]. The research performed in [Rush et al., 2015] focuses on neural models for sentence compression and rewriting, but not full document summarization. The work of [Lopyrev, 2015] leverages neural networks to generate news headline, where input documents are limited to 50 word tokens, and the work of [Hu et al., 2015] also deals with short texts (up to dozens of word tokens), in which summarization problems such as content redundancy is less prominent and attention-based models seem to be sufficient. However, summarization typically faces documents longer than that and summarization is often more needed when documents are long. In this work we attempt to explore neural summarization technologies for news articles with up to thousands of word tokens, in which we find distraction-based summarization models help improve performance. Note that our improvement is achieved over the model that has already outperformed the attention-based model reported in [Hu et al., 2015] on short documents.

3 Our approach

3.1 Overview

We base our model on the general encoder-decoder framework [Sutskever et al., 2011; 2014; Cho et al., 2014] that has shown to be effective recently on different tasks. This is a general sequence-to-sequence modeling framework in which the encoding part can be devoted to model the input documents and the decoder to generate output.

We believe the control layer that helps navigate the input documents to optimize the generation objectives would be of importance, and we will focus on the control layer in this paper and enrich its expressiveness. Specifically for summarization, unlike much recent work that focuses more on attention in order to grasp local context or correspondence (e.g, in machine translation and sentence compression), we force our models to traverse between different content of a document to avoid focusing on a region or same content, to better grasp the overall meaning for the summarization objective.

We also explore several popular technologies that have been applied to sentence-level tasks and extend them to document summarization, and we present those that help improve the summarization performance.

3.2 GRU-based encoding and decoding

Encoding The general document modeling and summarizing framework takes in an input document x = x1↪ · · · ↪ xTx and write the summary of the document as the output y = y1↪ · · · ↪ yTy . The summarization process is modeled as finding the output text y⇤ that maximizes the conditional probability argmaxy p(y♣x), given gold summary sequences. As discussed above, such a model has been found to be very effective in modeling sentences or sub-sentential text spans. We will address the challenges faced at the document level.

On encoding we do not restrict the encoders' architectures as if it is a recurrent neural network (RNN). The recent literature shows long-short termmemory (LSTM) [Hochreiter and Schmidhuber, 1997; Sutskever et al., 2014] and gated recurrent units (GRU) [Bahdanau et al., 2014] are both good architectures. In developing our systems we empirically found GRU achieved similar performance as LSTM but it is fast to train; we will therefore describe the GRU implement of our neural summarization models.

In the simplest uni-directional setting, when reading input symbols from left to right, a GRU learns the hidden annotations hi at time i with

where the hi 2 Rn encodes all content seen so far at time i which is computed from hi-1 and e(xi), where e(xi) 2 Rm is the m-dimensional embedding of the current word xi. The forward propagation of GRU is computed as follows.

where Wu, Wr, W 2 Rn⇥m and Uu, Ur, U 2 Rn⇥n are weight matrices, n is the number of hidden units, and ⊙ is element-wise multiplication.

In our work, we actually applied bi-directional GRUs (biGRUs), which we found achieving better results than single directional GRUs consistently. As its name suggests, in a bi-GRU unit, the annotation vector ht encodes the sequence from two directions, modeling both the left and right context. The bottom part of Figure 1 shows the encoder intuitively, while for more details, readers can refer to [Bahdanau et al., 2014] for further discussion.

Generation When generating summaries, the decoder predicts the next word yt given all annotations obtained in encoding h = h1↪ · · · ↪ hTx , as well as all previously predicted words y1↪ · · · ↪ yt-1. The objective is a probability over the summary y with decomposition into the ordered conditionals:

where Equation (6) depicts a high level abstraction of generating a summary word yt over previous output as well as input annotations h = h1↪ · · · ↪ hTx , and yt is a legal output word at time t, while y⇤ is the optimal summary found by the model.

The conditional probability is further rewritten as Equation (7) to factorize the model according to the structures of neural networks. The function g(yt-1↪ st↪ ct) is a nonlinear function that computes the probability vector for all legal output words at output time t, and g(yt-1↪ st↪ ct)♣yt takes the element of the resulting vector corresponding to word yt, i.e., the predicated probability for word yt.

The vector st and ct are the control layers that connect output y and input h, which we will discuss in details in Section 3.3. For completeness, function g(▷) is computed with:

where σ is a softmax function; Wo 2 RK⇥n , Uo 2 Rn⇥n , Vo 2 Rn⇥m and Co 2 Rn⇥2n are weight matrices; K is the vocabulary size; e(yt-1) 2 Rm is the m-dimensional embedding of the previously predicted word yt-1.

3.3 The control layers

The document modeling and summary generation are described above as two components: input document encoding and summary generation. A core problem is how these two components are associated. In sentence-level modeling such as machine translation and speech recognition, attention model is often applied to grasp local context and correspondence between input and output texts. For example, in translation attention is shown to be very useful for aligning the words of the target language (the language being translated to) to the corresponding source words and their context.

Attention can be regarded as a type of cognitive controls. In modeling documents, we take a general viewpoint on this control layer and propose distraction modeling to enable the model to traverse over different content of a long document, and we will show it improves the summarization performance significantly. In general, the control layer allows a complicated examination over the input. In this section we describe the controls that consider both attention and distraction to navigate input documents and to generate summaries.

Two-layer hidden output We first extended the recent twolevel hidden output model [Luong et al., 2015] to our summarization models. As presented later in the experiments, the two-level hidden output model consistently improves the summarization performance on different datasets. More specifically, the updating of st follows a two-layer GRU architecture shown in the top part of Figure 1.

The forward propagation of GRU1 and GRU2 are computed similar to Equation (1) above. GRU1 and GRU2 use untied parameter matrices. The two-layer model allows for capturing a direct interaction between s0t and ct, with the former encoding the current and previous output information and the latter encoding the current input content that is primed with distraction and attention. We will discuss how these vectors are computed below.

Distraction in training We propose to enforce distraction from two perspectives: adding the distraction constraints in training as well as in decoding. We first discuss the distraction in training.

Distraction over input content vectors In training we force the model not to pay attention to the same content or same part of the input documents too much. We accumulate the previously viewed content vector as a history content vector Pt-1 j=1 cj and incorporate it into the currently computed c0t . We refer to this model as M1.

where Wc 2 R2n⇥2n and Ua 2 R2n⇥2n are diagonal matrices. And c0t is input content vectors that have not been directly penalized with history yet; c0t is directly computed with conventional equation as follows:

where hi are annotation vectors that encode the current input word and its context with the input GRU described above in Equation (1). And ↵t↪i is the attention weight put on hi at the current output time t. The distraction-based ct computed in Equation (11) can then be incorporated in Equation (8).

Distraction over attention weight vectors We also propose to add distraction directly on the attention weight vectors. Similarly as above, we accumulate the past attention weights as a history attention weight Pt-1 j=1 ↵j↪i and use it to prime the current attention weights. The model in [Tu et al., 2016] also uses history attention weights, but we use history here to force distraction in order to avoid redundancy, which is not a concern in the machine translation task. We refer to the model as M2.

where Wa 2 Rl⇥n , Ua 2 Rl⇥2n , ba 2 Rl , and va 2 Rl are the weight matrices, and l is the number of hidden units. Note that Was0t + Uahi in the equation computes the conventional attention without penalizing attention history. ↵0 is often normalized with a softmax to generate attention weights ↵t↪i below, which is in turn used in Equation (12).

Distraction in decoding In the decoding process, we also enforced different types of distraction, one by computing the difference between the distribution of the current attention weight ↵t and that of all previous attention weights ↵1↪ · · · ↪ ↵t-1. Since ↵ can be seen as a proper probabilistic distribution, normalized in Equation (14), we used KullbackLeibler (KL) divergence to measure their difference with Equation (15), which was found to be consistently better than several other distance metrics we tried on the held-out data.

We also enforced distraction in a similar way on the attention-primed input content vector ct, as well as on the hidden output vector st. Both ct and st are not normalized but are regular content vectors, where the cosine similarity was found achieving a better performance than several alternatives (e.g., l1 and l2 distances) on the held-out data.

The distraction score d⇤↪t was then added into the output probability and the beam search in order to encourage the model to avoid redundant content.

where scoret was used as follows in the beam search with distraction algorithm, and parameter λ1, λ2, and λ3 were determined on the development set. We refer to this model as M3.

Algorithm 1 Beam search with distraction

Require: Vocabulary size K, beam size B, max output length N

• . Computed probabilities of all the words in vocabulary
• . Choose the B most likely words and initialize the B hypotheses

for i = 1 : N do

• . For each hypothesis, compute the next conditional probabilities, then have B ⇥K candidates with the corresponding probabilities
• . Use the distraction-primed value score to choose B most likely candidates

end for

Unknown word replacement for summarization We borrowed the unknown word replacement [Jean et al., 2015] from machine translation to our summarization models and found it improved the performance when summarizing long documents. Specifically, due to the time complexity in handling a larger vocabulary in the softmax layer in summary generation, infrequent words were removed from the vocabulary and were replaced with the symbol hUNKi. The threshold of vocabulary size is data-dependent and will be detailed later in the experiment set-up section.

After the first-round summary generated for a document, a token labeled as hUNKi will be replaced with a word in the input documents. More specifically, we obtained the replacement using Equation (14); i.e., we used the largest element in ↵t to find the source location for the current hUNKi.

4 Experiment set-up

4.1 Data

We experiment with our summarization models on two publicly available corpora with different document lengths and in different languages: a CNN news collection [Hermann et al., 2015] and a Chinese corpus made available more recently in [Hu et al., 2015]. Both are large datasets appropriate for training neural models, which, as discussed above, employ a large number of parameters to fit the potentially complicated summarization process involving representing input documents, generating summaries, and interacting between them.

CNN data The CNN data [Hermann et al., 2015] have a human-generated real-life summary for each news article. The dataset collected in was made available at GitHub. 1 The data was preprocessed with the Stanford CoreNLP tools [Manning et al., 2014] for tokenization and sentenceboundary detection; all capital information is kept. To speed up training, we removed the documents that are too long (over 2,500 word tokens) from the training and validation set, but kept all documents in the test set, which does not change the difficulty of the task.

1 https://github.com/deepmind/rc-data

LCSTS data The second corpus is LCSTS, which is a Chinese corpus made available more recently in [Hu et al., 2015]. The data is constructed from the Chinese microblogging website, Sina Weibo. We used the original training/testing split mentioned in [Hu et al., 2015], but additionally randomly sampled a small part of the training data as our validation set.

Table 1 gives more details about the two datasets. We can see from the table that averaged document length of the CNN corpus is about seven time as long as the LCSTS corpus, and the summary is about 2-3 times longer.

4.2 Training details

We used mini-batch stochastic gradient descent (SGD) to optimize log-likelihood, and Adadelta [Zeiler, 2012] to automatically adapt the learning rate of parameters (✏ = 10-6 and ⇢ = 0▷95).

For the CNN dataset, training was performed with shuffled mini-batches of size 64 after sorting by length. We limit our vocabulary to include the top 25,000 most frequent words. Other words were replaced with the token hUNKi, as discussed earlier in the paper. Based on the validation data, we set embedding dimension to be 120, the vector length in hidden layers to be 500 for uni-GRU and 600 for bi-GRU. An end-of-sentence token was inserted between every sentence, and an end-of-document token was added at the end. The beam size of decoder was set to be 5.

For the LCSTS data, a larger mini-batch size 256 was found to be better based the observation on the validation set. Same as in [Hu et al., 2015], we used characters rather than words as our tokens. The vocabulary size is 4000, embedding dimension is 500, and the vector size of the hidden-layer nodes is 500. Beam search size is 5, same as in the CNN dataset.

We make our code publicly available 2 . Our implementation uses python and is based on the Theano library [Bergstra et al., 2010].

5 Experimental results

5.1 Results on the CNN dataset

Overall performance Our results on the CNN dataset are presented in Table 2. We used Rouge scores [Lin, 2004] to measure performance. Since the summary lengths are not preset to be the same, we report F1 Rouge. The upper part of the table includes the baseline results of a number of typical summarization algorithms, which we listed in the table as Luhn [Luhn, 1958], Edmundson [Edmundson, 1969], LSA [Steinberger and Jezek, 2004], Lex-rank [Erkan and Radev, 2004], Text-rank [Mihalcea and Tarau, 2004], Sumbasic [Vanderwende et al., 2007], and KL-sum [Haghighi and Vanderwende, 2009]. These baseline results are implemented in the open-source tool SUMY 3 .

2 Our code is available at https://github.com/lukecq1231/nats

The results at the lower half of the table show that the bi-GRU encoder achieves a better performance than the uniGRU encoder. This is consistent with the results on the LCSTS dataset reported later in Table 4. We show that two-level output model we discussed in the method section is beneficial, which is also consistent with the results on the LCSTS dataset. In addition, the unknown replacement technique yields an additional improvement.

Over the strong model that has used these technologies (the row marked as "+UNK replace"), the model in the last row that incorporates all distraction modeling (M1, M2 and M3) finally achieves a Rouge-1 score of 27.1, a Rouge-2 score of 8.2, and a Rouge-L score of 18.7, significantly improving the three Rouge scores by 5.8, 1.9, and 2.3, respectively. These are also the largest improvement presented in the table, compared with the other techniques listed. The table also lists the details of how the model M1, M2, and M3 improve the performance additively. Again, the neural models do not engineer any features and use only content but not any additional formality features such as locations of input sentences, which may bring additional improvement.

Performance on different lengths of documents To observe the effectiveness of the distraction model over different document lengths, we further selected all short documents from the CNN training dataset into a subset (subset-1) with average length at 335 word tokens, and a subset of data (subset-2) that have the same number of documents as the subset-1, with averaged document length at 680 word tokens. As shown in Table 3, on the data subset-2, the distraction model improves the results more significantly. The relative improvement is 29.0%, 25.6%, and 10.8%, compared with 25.9%, 20.5%, and 8.1% on subset-1, respectively. In general the best performance on both dataset is lower than that using all training data, suggesting using more training data can improve summarization performance.

3 https://pypi.python.org/pypi/sumy

5.2 Results on the LCSTS dataset

We experiment with the proposed model on public LCSTS corpus. The baseline is the best result reported in [Hu et al., 2015] 4 . Our modified uni-GRU achieves a slight improvement over the reported results. The Bi-GRU attention-based model achieves a better performance, confirming the usefulness of bi-directional models for summarization as well as that our implementation is the state-of-the-art and serves as a very strong baseline in the CNN dataset discussed above. Note that since the input text length of LCSTS is far shorter than the CNN documents, each containing about 100 words and roughly 6-8 sentences, we show that distraction does not improve the performance, but in contrast, when documents are longer, its benefits are significant, achieving the biggest improvement as discussed earlier. This suggests the effectiveness of distraction modeling in helping summarize the more challenging longer documents, where summarization is often more necessary than for short texts.

4 We thank the authors of [Hu et al., 2015] for generously sharing us the latest output of their models, which achieves a better performance than the results reported in [Hu et al., 2015]. We reported here the updated scores higher performance as our baseline.

We also compare our models with the simple baseline that selects the first N numbers of word tokens from the input documents, which reaches its maximal Rouge scores when the first 30 tokens were taken, and achieves Rouge-1, Rouge2, and Rouge-L at 25.5, 14.1 and 21.4. And our models are significantly better than that. For the CNN data set, choosing the first three sentences achieves the best results, which reach Rouge-1, Rouge-2, and Rouge-L at 26.1, 9.6 and 17.8, respectively. Since the CNN data is news data, the baseline of selecting first several sentences has known to be a very strong baseline. Again, the models we explore here are towards performing abstractive summarization.

6 Conclusions and future work

We propose to train neural document summarization models not just to pay attention to specific regions of input documents with attention models, but also distract the models to different content in order to better grasp the overall meaning of input documents. Without engineering any features, we train the models on two large datasets. The models achieve the stateof-the-art performance and they significantly benefit from the distraction modeling, particularly when the input documents are long. We also explore several recent technologies for summarization and show that they help improve summarization performance as well. Even if applied onto the models that have already leveraged these technologies, the distraction models can further improve the performance significantly.

From a more general viewpoint, enriching the expressiveness of the control layers that link the input encoding layer and the output decoding layer could be of importance to remedy the shortcomings of the current models. We plan to perform more work along this direction.

Acknowledgments

The first and the third author of this paper were supported in part by the Science and Technology Development of Anhui Province, China (Grants No. 2014z02006), the Fundamental Research Funds for the Central Universities (Grant No. WK2350000001) and the Strategic Priority Research Program of the Chinese Academy of Sciences (Grant No. XDB02070006).