Learning towards Abstractive Timeline Summarization

1 Introduction

Timeline summarization aims at concisely summarizing the evolution trajectory of input events along the timeline. Existing timeline summarization approaches such as [Li and Li, 2013; Ren et al., 2013] are all based on extraction methods. However, these methods rely on human-engineered features and are not as flexible as generative approaches. Herein, we propose the abstractive timeline summarization task which aims to concisely paraphrase the event information in the input article. An example case is shown in Table 1, where the article consists of events of a greatest entertainer in different periods, and the summary correctly summarizes the important events from the input article in order.

Abstractive summarization approaches including [See et al., 2017; Hsu et al., 2018] have been proven to be useful recently thanks to the development of neural networks. However, unlike traditional document summarization, timeline summarization dataset consists of a series of time-stamped events, and it is crucial for timeline summarization model to capture this time series information to better guide the chronological generation process. Besides, as the example in Table 1 shows, bad summary confuses the birthplace and the residence, the first album and the best-selling album of the celebrity. As we found in experiment, such infidelity problem is a commonly-faced problem in summarization tasks.

∗Equal contribution. Ordering determined by dice rolling.

†Contact Author.

To tackle above challenges, we come up with a memorybased timeline summarization (MTS) model. Specifically, we first use an event embedding module with selective reading units to embed all events. Then, we propose a key-value memory module storing time series information to guide the summary generation process. Concretely speaking, the key in memory module is the time position embedding that represents the time series information, and the value is the corresponding event representation. Keys together forms a timeline and we use the time position of events on the timeline to guide generation process. Finally, in each decoding step, we introduce event-level attention and use it to determining word-level attention so as to avoid confusion between events.

Overall, our contributions can be summarized as follows:

• We propose the generative timeline summarization task.

• To tackle this task, we first come up with a time-event memory modeling time series information to guide chronological generation process.
• In each decoding step, we incorporate event-level information to assist in determining word-level attention so that the generated summary can avoid confusion between events.
• We also release the first real-world large-scale timeline summarization dataset 1 . Experimental results on the dataset demonstrate the effectiveness of our framework.

2 RelatedWork

We detail related work on timeline summarization, abstractive summarization, and key-value memory network.

Timeline summarization. Timeline summarization task is firstly proposed by [Allan et al., 2001] which extracts a single sentence from each event within a news topic. Later, a series of works [Yan et al., 2011b; Yan et al., 2011a; Yan et al., 2012; Zhao et al., 2013] furthur investigate timeline summarization task. There are also works focusing on tweets summarization that are related to timeline summarization. For example, [Ren et al., 2013] considered the task of time-aware tweets summarization, based on a user's history and collaborative social influences from "social circles". However, all above works are based on extractive methods, which are not as flexible as abstractive approaches.

Abstractive summarization. Recently, with the emergence of strong generative neural models for text [Bahdanau et al., 2014], abstractive summarization is also becoming increasingly popular [Nallapati et al., 2017; See et al., 2017]. Most recent work includes [Hsu et al., 2018], where they use sentence-level attention to modulate the word-level attention such that words in less attended sentences are less likely to be generated. Their sentence-level attention is static during the generation process, while in our model, the high-level attention changes in each decode step depending on current generated word which is more reasonable.

Key-value memory network Key-value memory proposed by [Miller et al., 2016] is a simplified version of Memory Networks [Weston et al., 2015] with better interpretability and has been applied in document reading [Miller et al., 2016], question answering [Pritzel et al., 2017], language modeling [Grave et al., 2017], and neural machine translation [Kaiser et al., 2017]. In our work, we apply the key-value memory network on timeline summarization task and fuses it into the generation process.

3 ProblemFormulation

MTS takes a list of events X = {x1, ..., xTe } as inputs, where Te is the number of events. Each event xi is a list of words: xi = {wi1 ,wi2 , ...,wiT iw }, where T iw is the word number of event xi. The goal of MTS is to generates a summary ˆ Y = {ˆ y1, ..., ˆ yTy } that is not only grammatically correct but also consistent with the event information such as occurrence place and time. Essentially, MTS tries to optimize the parameters to maximize the probability P (Y |X) = ∏Ty t=1 P (yt|X), where Y = {y1, ..., yTy }is the ground truth answer.

1 http://tiny.cc/lfh56y

4 Model

4.1 Overview

In this section, we introduce our memory-based timeline summarization model in detail. The overview of MTS is shown in Figure 1 and can be split into three modules: (1) Event Embedding Module (See § 4.2): We employ a recurrent network with Selective Reading Units (SRU) to learn representation of each event. (2) Time-Event Memory (See § 4.3): we propose a time-event memory to establish a timeline, and use the time position of events in the timeline to guide generation process. (3) Summary Generator (See § 4.4): Eventually, we use an RNN-based decoder to generate the answer incorporating memory information, event-level information, and word-level information.

4.2 Event EmbeddingModule

To begin with, we use an embedding matrix e to map onehot representation of each word in xi into a high-dimensional vector space. We then employ a bi-directional recurrent neural network (Bi-RNN) to model the temporal interactions between words:

where ";" denotes the concatenation between vectors, hit denotes the hidden state of t-th word in Bi-RNN for event xi. To capture the sequential information of events, we randomly initialize a time position encoding vector pi of i-th event to be included in the Bi-RNN input. Apart from gaining word representation hit , we also need to gain event representation. Simply taking the final state of Bi-RNN hiTw as the representation of the whole event cannot fully capture the feature of the whole event. Thus, we establish a second RNN made of SRU proposed in [Chen et al., 2018] to gain new event representation ai :

Generally speaking, SRU replaces the update gate in original GRU with a new gate taking each input hit and coarse event representation hiTw into consideration. We omit the details here due to limited space and readers can refer to [Chen et al., 2018] for details.

So far, we gain the representation of i-th event ai and t-th word in ai , i.e., hit .

4.3 Time-Event Memory

As stated in Introduction, in timeline dataset, the generated summary should capture the time series information to guide the chronological generation process. Hence, we propose a key-value memory module where keys together forms a timeline, and this time series information is used to guide generation process as shown in Figure 2.

The key in this memory is the time position encoding pi introduced in § 4.2. In § 4.4, we will use this key as time guidance to extract information from memory. As for the value part, it stores event information of local aspect in local value v1 and global aspect in global value v2. v1 simply stores the event representation ai as vi1 , which means that v1 only captures event information from current input article. On the other hand, v2 is responsible for learning the global characteristics of events in different time position. Thus, we first randomly initialize vi2 for i-th event in the same way as time position encoding. Then we establish a gate ν to combine average event representation in current batch ˙a as a sub-global information:

where σ is the sigmoid function and · is dot product. In this way, the memory learns itself the global feature of each event in different position and stores it in v2.

4.4 Summary Generator

To generate a consistent and informative summary, we propose an RNN-based decoder which incorporates outputs of time-event memory module and event representation as illustrated in Figure 2.

Following [Li et al., 2018], we randomly initialize an LSTM cell taking the concatenation of all event representations as input, and use the output as decoder initial state:

where hc is a random variable. Next, following traditional attention mechanism in [Bahdanau et al., 2015], we summarize the input document into context vector ct-1 ′ dynamically, and the t-th decoding step is calculated as:

where h′t is the hidden state of t-th decoding step, and will be modified by output from memory module in Equation 22. Context vector c′t-1 is calculated as:

where we first use the decoder state h′t-1 to attend to each states hij and resulting in the attention distribution αt,i,j ∈ RT d , shown in Equation 9. hij denotes the representation of j-th word in event xi. Then we use the attention distribution αt,i,j to get the weighted sum of document states as the context vector c′t-1 .

Context vector c′t-1 here only takes the word-level attention into consideration without considering event-level information. However, in timeline summarization, it is important for the model to be aware of which event it is currently describing, or it may confuse information from different events and results in an unfaithful summary. Hence, we introduce an event-level attention β similar to the calculation of word-level attention and use it to adjust word-level attention:

The new context vector ct (replacing c′t in Equation 10) is now calculated as:

Apart from using event-level attention to directly guide wordlevel attention, we also use it to gain the weighted sum of event representation to be concatenated in the projection layer in Equation 23:

So far, we have finished the calculation of context vector. Next, we introduce how to incorporate the guidance from memory. We first use hidden state h′t to attend to each key in memory. As stated in § 4.3, keys, i.e., time position embeddings, conform the timeline that represents the time series information. Thus, we let the model take advantage of this sequential information, and calculate the relevance between position encoding and current state as time-attention π(pi , h′t ):

Time-attention is then used to gain the weighted sum of local value v1 and global value v2 in the memory:

m1′ t and m2′ t stores information from different level, thus should play different roles in generator.

By a fusion gate, local value m1′ t is changed tom1t and will be incorporated into the projection layer in Equation 23.

We place the local value in the projection layer since m1′ t stores the detailed information rather than the global feature in the input, thus should play an important part when generating each word.

As for the global value m2′ t , it stores the global feature of event in different position, thus should influence the whole generation process. Concretely, information from m2′ t is fusioned into the decoding state h′t-1 by a gate:

Finally, an output projection layer is applied to get the final generating distribution Pv over vocabulary:

We concatenate the output of decoder LSTM h′t , the context vector ct, and memory vector mt as the input of the output projection layer.

In order to handle the out-of-vocabulary (OOV) problem, we equip the pointer network [Gu et al., 2016; See et al., 2017] with our decoder, which enables the decoder capable of copying words from the source text. The design of the pointer network is the same as the model used in [See et al., 2017], thus we omit this procedure due to limited space.

Our objective function is the negative log likelihood of the target word yt, shown in Equation 24:

The gradient descent method is employed to update all parameters to minimize this loss function.

5 Experimental Setup

5.1 Research Questions

We list four research questions that guide the experiments: RQ1 (See § 6.1): What is the overall performance of MTS? Does it outperform other baselines? RQ2 (See § 6.2): What is the effect of each module in MTS? RQ3 (See § 6.3): Is the time position embedding useful so that time-event memory can correctly guide generation process? RQ4 (See § 6.4): Can event-level attention correctly guide word-level attention in decoding process?

5.2 Dataset

We collect a large-scale timeline dataset from the world's largest Chinese encyclopedia 2 . The character subsection of this website consists of celebrities at all times and in all countries or lands. On each website page, there is a timeline summary for each character, and in the character experience section of this page, each event is set as a paragraph with explanation and details, which is selected as input article. We filter out irrelevant content such as cited sources and figures. In total, our training dataset amounts to 169,423 samples with 5,000 evaluation and 5,000 test samples. On average, there are 352.22 words and 61.16 words in article and summary respectively.

5.3 ComparisonMethods

We first conduct ablation study to prove the effectiveness of each module in MTS. Then, to evaluate the performance of our proposed dataset and model, we compare it with the following baselines:

(1) Pointer-Gen: Sequence-to-sequence framework with pointer mechanism proposed in [See et al., 2017]. (2) FTSum: A summarization model proposed in [Cao et al., 2018]. Since there is no open information extraction tool in Chinese, we use POS tagging to extract entities and verbs to replace it. (3) Unified: State-of-the-art generative summarization model proposed in [Hsu et al., 2018]. (4) LEAD3: a commonly used baseline, which selects the first three sentence of document as the summary. (5) TextRank: [Mihalcea and Tarau, 2004] propose to build a graph, then add each sentence as a vertex and use link to represent semantic similarity. (6) ITS: One of state-of-the-art extractive summarization models proposed in [Chen et al., 2018].

5.4 EvaluationMetrics

For evaluation metrics, we adopt ROUGE score in [Lin, 2004] which is widely applied for summarization evaluation [Sun et al., 2018; Chen et al., 2018]. The ROUGE metrics compare generated summary with the reference summary by computing overlapping lexical units, including ROUGE1 (unigram), ROUGE-2 (bi-gram) and ROUGE-L (longest common subsequence).

[Schluter, 2017] notes that only using the ROUGE metric to evaluate summarization quality can be misleading. Therefore, we also evaluate our model by human evaluation. Three highly educated participants are asked to score 100 randomly sampled summaries generated by Unified and MTS. Statistical significance of observed differences between the performance of two runs are tested using a two-tailed paired ttest and is denoted using ▲(or ▼) for strong significance for α = 0.01.

2 https://baike.baidu.com/

5.5 Implementation Details

We implement our experiments in TensorFlow [Abadi et al., 2016] on NVIDIA GTX 1080 Ti GPU. The word embedding dimension is set to 128 and the number of hidden units is 256. For time-event memory, the dimension of key, global value, and local value is 128, 512, and 256 respectively. We initialize all of the parameters randomly using an uniform distribution in [-0.02, 0.02]. The batch size is set to 16, and the event number is set to 8. We use Adagrad optimizer [Duchi et al., 2010] as our optimizing algorithm and the learning rate is 0.15. In decoding, we employ beam search with beam size 4 to generate more fluency summary sentence.

6 Experimental Results

6.1 Overall Performance

For research question RQ1, we examine the performance of our model and baselines in terms of ROUGE as shown in table 2. Firstly, generative models outperform extractive models by a substantial margin, demonstrating the necessity of generative timeline summarization approaches. Secondly, the state-of-the-art model on CNN/DailyMail summarization dataset, Unified, still gets the best performance among baseline models on our timeline summarization dataset and outperforms the Pointer-Gen by 4.45% in ROUGE-1, which demonstrates the effectiveness of baselines. Finally, MTS achieves better performance with 4.02%, 1.32% and 3.48% increment over Unified and 8.65%, 4.16% and 9.21% over Pointer-Gen in terms of ROUGE-1, ROUGE-2 and ROUGEL respectively, which proves the superiority of our model.

As for human evaluation, we ask three highly educated participants to rank generated summaries in terms of fluency, informativity, and fidelity. We pick FTSum and Unified as baselines since their performance is relatively high compared to other baselines. The rating score ranges from 1 to 3 and 3 is the best. The result is shown in Table 3, where MTS outperforms Unified by 5.44%, 3.61% and 18.09% in terms of fluency, informativity, and fidelity. It is worth noticing that the infidelity problem is a serious problem existing in timeline summarization, and MTS greatly alleviates such problem. We also conduct the paired student t-test between our model and Unified (row with shaded background), and result demonstrates the significance of the above results. The kappa statistics is 0.54 and 0.57 respectively, which indicates moderate agreement between annotators 3 .

6.2 Ablation Study

Next, we turn to research question RQ2. We conduct ablation tests on the usage of event-level attention, global and local value in time-event memory, corresponding to MTS w/o EA, MTS w/o LV, MTS w/o GV respectively. The ROUGE score result is shown in Table 4. Performances of all ablation models are worse than that of MTS in terms of all metrics, which demonstrates the necessity of each module in MTS. Concretely, local value and global value both make great contribution to overall performance, demonstrating that time series information is indeed helpful in extracting information to guide generation process. Besides, event-level attention also plays an important part. Without guidance from this level, word level attention has difficulty in focusing on input article and that leads to a 3.71% drop in ROUGE-L.

6.3 Analysis of Time Position Embedding

We then address RQ3, the usefulness of time position embedding is reflected by time-attention. We visualize the attention map of two randomly sampled example as shown in Figure 3. The figure above is the attention map in the first decoding step, and the figure below is in the final decoding step. The darker the color is, the higher the attention is. Due to limited space, we omit the corresponding event descriptions. When decoding starts, MTS learns to pay attention to the first two events, which always consist of parallel information such as the birthplace and birth date of the character. The attentions on last several events are low since it does not need this information in advance. When decoding ends, MTS focuses more on the last several events. However, it also pays attention to the first few events, since timeline summarization is a process of information accumulation, and latter sentences should consider previous information. Above example demonstrates the effectiveness of time position embedding.

3 [Landis and Koch, 1977] characterize kappa values < 0 as no agreement, 0-0.20 as slight, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial, and 0.81-1 as almost perfect agreement.

7 Conclusion and FutureWork

In this paper, we propose a framework named MTS which aims to generate summaries that concisely summarize the evolution trajectory along the timeline. we first propose an event embedding module with selective reading units to embed all events. Then we propose a time-event memory module storing structral evolutinary event information to guide generation process. Finally, in each decoding step, we unify the current sentence-level attention and word-level attention together to avoid confusion between events. Our model outperforms state-of-the-art methods in terms of ROUGE and human evaluations by a large margin. In the near future, we aim to propose a time-aware timeline summarization that can summary the a specific time period of an whole article.

Acknowledgments

We would like to thank the anonymous reviewers for their constructive comments. This work was supported by the National Key Research and Development Program of China (No. 2017YFC0804001), the National Science Foundation of China (NSFC No.61876196, No. 61672058), Alibaba Innovative Research (AIR) Fund. Rui Yan was sponsored by CCF-Tencent Open Research Fund and Microsoft Research Asia (MSRA) Collaborative Research Program.

We now turn to RQ4, whether event-level attention can guide word-level attention. We first conduct a case study to visualize the two level attentions, as shown in Figure 4. The figure above is the event-level attention, and three figures below are word-level attention corresponding to the first three events. We only show first 11 words in an event. The result shows that the third event is the most important event in this decoding step, and weights of the words in this event are also greater than other words on average. Above observation demonstrates that event-level attention gives the correct guidance for word-level attention.

Apart from the visualization, we also conduct quantitative analysis to measure how greatly the word-level attention is influenced by event-level information, which is reflected by inconsistency loss. We adjust the inconsistency loss proposed in [Hsu et al., 2018] into MTS, and the new consistency loss at t-th decoding step is the negative log-likelihood of the product of attention value of most attended three words and their corresponding event-level attention. The intuition is to verify whether the event-level attention is high too when word-level attention is high. When training starts, the inconsistency loss is around 4.8, and when training ends, the loss drops to 2.6. This means that event-level information greatly influences the word-level attention and the model learns to unify these two attentions. We did not directly add inconsistency loss to training because we found that made MTS perform worse. Instead, we let the model learn by itself to unify these two attentions.

We also show a case study in Table 5. We can observe that baseline Unified confuses the description of events for twice. It is the movie "The Flowers Of War" that wins Golden Globe Award instead of the actor. While in summaries generated by MTS, the important events and their corresponding descriptions are all correctly included.