Point at the Triple: Generation of Text Summaries from Knowledge Base Triples (Extended Abstract)

1 Introduction

Natural Language Generation (NLG) is the task of generating text that captures the content of structured-data records in a human-readable way [Reiter and Dale, 2000]. In the context of knowledge graphs, structured data takes the form of triples.

NLG systems for knowledge graphs take as input a subset of the graph's triples and output a text summary [BouayadAgha et al., 2014]. Earlier attempts to generate text from triples tend to use rules or template-based techniques, which work well in domains with a regular structure and limited vocabulary [Bouayad-Agha et al., 2012]. More recent proposals leverage deep learning, which was successful in similar text-generative tasks, including machine translation [Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2014] and text summarisation [Rush et al., 2015; See et al., 2017]. Systems such as [Chisholm et al., 2017; Lebret et al., 2016; Vougiouklis et al., 2018] are able to generate coherent, relevant text for Wikipedia from structured data, without any input from linguists or domain experts. However, these newer systems are not without their challenges: while they could, in principle, scale to open-domain tasks, they only report their performance on one domain, people's biographies. In addition, they struggle to verbalise rare or previously unseen entities, which are represented as placeholder tokens in the output and are meant to be replaced in a post-processing step. This introduces a degree of stochastic behaviour when multiple relations from the input match the predicted placeholders.

∗This paper is an extended abstract of an article entitled "Point at the Triple: Generation of Text Summaries from Knowledge Base Triples" in the Journal of Artificial Intelligence Research.

† Work done while at University of Southampton.

In this paper, we present an approach that addresses these concerns. Table 1 presents a canonical NLG task. Our aim is to automatically generate a textual summary describing the graph about Atlas Shrugged. The input contains triples in which the given entity, in this case Atlas Shrugged, is the subject or the object of the triples. Our approach is inspired by pointer-generator networks which have been recently introduced in text summarisation [Gu et al., 2016; See et al., 2017]. Our systems jointly learn to: (i) verbalise the entities from pointed triples in several ways; (ii) copy the label or the number in the case that the pointed triple consists of either infrequent entities or numbers; and (iii) generate words or other human-readable realisations of entities from a fixed vocabulary.

Following the methodology of Vougiouklis et al., we create a dataset encompassing the entirety of Wikipedia rather than just the biographies. We use this dataset to demonstrate our model's ability to generalise on a much more challenging task. We evaluate our approach in two ways: automatically and manually. For the former, we use BLEU, ROUGE and METEOR in order to evaluate the performance of our approach in both the widely cited task of biographies generation [Chisholm et al., 2017; Lebret et al., 2016; Liu et al., 2018; Vougiouklis et al., 2018; Yeh et al., 2018], and the generation of open-domain Wikipedia summaries. Furthermore, we run a user study in which we explore the fluency and coverage of the summaries, as well as the presence of contradictions. In all scenarios, our systems outperform a variety of competing baselines of different natures. Our dataset along with the code of our systems is available at: github.com/pvougiou/Point-at-the-Triple.

2 Our Model

We assume our model is trained on records consisting of an English language summary and a set of triples. Let Fz = {f1, . . . , fE : fi = (si, pi, oi)} be the set of triples f1, . . . , fE about the entity z, where si, pi and oi are the onehot vector representations of the respective subject, predicate and object of the i-th triple. We build a model that computes the probability of generating a sequence of tokens y = y1, y2, . . . , yT , given the initial set of triples f1, f2, . . . , fE.

We build upon architectures from the literature, in particular See et al. and our own work in Vougiouklis et al.. The former introduces a pointer-generator network capable of both copying tokens from the input sequence and generating words for the fixed vocabulary of the decoder. While this model handles sequential inputs and outputs (i.e. sentences), in our case the sets of input triples are un-ordered, and not sequentially correlated. For this reason, we use the encoder proposed in the latter work of Vougiouklis et al., and we compute the attention scores on top of this feed-forward architecture.

In many cases, the entities that participate in the properties contained in the input triples cannot be directly copied to the generated text. For example, the entities of dbr:Actor and dbr:United States could be expressed, based on the context, as both "actor" or "actress", and "United States" and "American", respectively. To tackle this, we propose a technique that enables our model to learn different realisations for the entities of the pointed triples. The technique also helps with handling rare entities, for which we do not have good vector representations, numerical values that are in the third, object position of the pointed triples, and their labels.

Decoder. We implement the decoder as a multi-gated RNN variant with Gated Recurrent Units (GRUs). Let hlt ∈ Rm be the aggregated output of a hidden unit at timestep t = 1 . . . T and layer depth l = 1 . . . L.

2.1 Triple Encoder

We compute the vector representation hfi of the i-th triple by forward propagating the triple encoder as follows:

where [. . . ; . . .] represents vector concatenation, Wx→˜h : R|N| → Rm and W˜h→h : R3m → Rm are unbiased linear mappings.

Attending the Triples. We implement an attention mechanism over all triples, based on Dong and Lapata's formulation. Our model uses the attention scores to compute a context vector, ct, as a weighted sum over the representations of the encoded triples. The alignment between this context vector and the information that has already been processed in a generated summary are jointly learned through trainable weightsWc : Rm →Rm andWh : Rm →Rm as follows:

2.2 Dynamically Expanding the Vocabulary

As discussed earlier, the pointer-generator network from See et al. learns to copy only a single representation per input token. This means that the system uses the same label for each copied entity regardless of the context in the text, which impacts fluency [Vougiouklis et al., 2018]. Our approach addresses this concern by learning multiple realisations for the entity of a pointed triple.

Our approach is partially inspired by how humans would perform on the same task. When provided with a set of triple-facts which they are asked to summarise in text, people would start summarising by using their own known vocabulary. However, they would focus their attention on a particular triple when they would want to realise an entity's name or a number in the text.

We define Gkd = {gkd 1 , gkd 2 , . . . , gkd Q } as the set of all possible verbalisations through which our model learns to express an entity kd in the generated summary. Q is a datasetspecific hyper-parameter for our model, and is calculated by averaging the number of possible realisations qkd ∀kd ∈ K.

Let H(f) and H(i) be the sets of all the frequent and infrequent entities that participate in the triples. In addition, let E = {e1, e2, . . . , eE} s.t. ej ∈ (sj, oj) and ej ̸= z ∀j ∈ [1,E] be the set of all the items (numerical values or entities) other than entity z that participate in the corresponding relationships in F. We assume a fixed target vocabulary V † . In comparison to similar pointer-generator networks that expand the decoder's fixed vocabulary by the length of their input E, we expand it by Q · E, and we define the dynamic vocabulary extension (where the values are based on the input triples), V ext = {vext 1 , vext 2 , . . . , vext Q·E } along with its subsets V f ext, V copy ext and V null ext , s.t.:

∀j ∈ [1, Q · E], where ⌈. . .⌉ represents the ceiling function.

During both training and testing, for each set of input triples we form the values of V ext . Each triple is provided with Q slots in V ext . In case a rare entity is either the subject or the object of a triple in the triple set, it is replaced by its corresponding instance type token before it is provided to our model. In such scenario, all values of V ext that correspond to this particular triple are filled with null, except the first one which refers to the copy of the label of this rare entity. A similar methodology is used for numbers.

2.3 Summarising By Pointing and Generating

The probability distribution qt for each entry in the vocabulary extension V ext after distributing the attention scores over the realisations of the relevant triples is computed as follows:

∀i ∈ [1,Q · E]. We adopt the notion of generation probability pgen t ∈ [0, 1], to simulate a soft switch at each timestep t between generating a token from the fixed vocabulary or copying either the surface form or the label of an entity from the highlighted triple [See et al., 2017]. Our model computes the following probability distribution for each entry w in the extended vocabulary V = V † ∪ V ext as follows:

where P voc t = softmax(WyhL+1 t ), and Wy : Rm → R|X|+Q·E is a trainable weight matrix.

3 Experiments

We trained and evaluate our system on two corpora. The first is the D1 Biographies corpus provided by Vougiouklis et al., which consists of triples from DBpedia aligned with Wikipedia biographies. The second corpus, which we refer to as the Full corpus, uses the same methodology as in [Vougiouklis et al., 2018], applied, however, to the entire Wikipedia. We compute a Q value of 2 (for the D1 Biographies dataset), and 3 (for the Full corpus), which results in ∼ 98% coverage of the total number of textual realisations of the triples' entities for both corpora (see Section 2.2).

For each dataset we ran two sets of experiments: one in which the surface form tuples [Vougiouklis et al., 2018] were part of the fixed vocabulary of the decoder, and one in which they were treated as regular words. We evaluated our approach by comparing it against a set of competitive baselines: (i) Random, (ii) Kneser-Ney (KN) language model, (iii) Information Retrieval (IR), (iv) Triples2GRU and Triples2LSTM from Vougiouklis et al., and (v) PointerGenerator, which has been adapted from See et al..

3.1 Automatic Evaluation

We evaluated our performance in terms of the following three metrics: (i) BLEU (Bilingual Evaluation Understudy),

(ii) ROUGE (Recall-Oriented Understudy for Gisting Evaluation), and (iii) METEOR.

In almost all scenarios, we outperformed the baselines. On the Full corpus, both our systems achieved an improvement that ranges from 0.59 to 0.79 and 0.16 to 0.47 BLEU 4 and ROUGE points, respectively, in comparison to PointerGenerator, which is our strongest competitor. On the Biographies corpus, we can show small improvements of at least 0.02 in terms of both BLEU and ROUGE points. We believe that the lower performance gain on the smaller corpus is mainly a function of the limited linguistic variability of biographies-in 92.67% of the cases, entities from the triples in that corpus were realised in the text with their most frequent surface form. The advantages of our approach become more clear on the Full corpus, which is linguistically more challenging and includes entities with varied realisations.

3.2 Human Evaluation

The three metrics we used in the automatic evaluation do not capture performance well in tasks where the input and the output are loosely correlated [Reiter, 2010]. To understand how well our approach would do in practice, we undertook two user studies with participants recruited from the Figure Eight crowdsourcing platform. In the first one, we looked at the performance of our networks against the two most competitive baselines, Triples2GRU and Pointer-Generator, on the open-domain task. In the second one, we explored whether training our systems on the Full dataset with the same hyper-parameters (except the size of the input/output vocabularies) would yield results comparable to systems that were trained on the biographies task.

Inputs and Outputs

We included only input sets of at least six triples. For the first study we sampled 32 sets of triples according to the instancetype distribution of the main discussed entities in the Full dataset. For the second study, we used a random sample of the same size. In both cases, we took the summaries generated by the four systems and asked 10 participants to assess them against three criteria: (i) fluency, (ii) coverage, which is concerned with the triples in the input whose content is mentioned either implicitly or explicitly in the text, and (iii) contradiction, which refers to information that is conveyed by the sentence, but conflicts with one or more of triples from the input set.

For each of the 32 sets of triples used in each study, we also compiled a gold standard of 8 additional sets of triples with manually written summaries. We designed three crowdsourcing tasks, one for fluency, one for coverage, and one for contradiction.

Results of the First Study

The results of the two studies are in alignment with the results of the automatic evaluation.

Fluency. In the first study, which focused on the performance on the Full dataset, the summaries generated by both our systems were ranked significantly higher than those of the the two baselines (one-way ANOVA test, p < .05). Among our two systems, the one with surface form tuples was also found to generate significantly more fluent summaries than the one without.

Coverage. Both our systems realised more information from the input triples in the generated summaries; the coverage of both systems is also significantly better than Triples2GRU. Figure 2 shows the extent to which different number of predicates from the input triples are covered in the text. Our architectures are able to realise not only more predicates than the competition but also to address these predicates with higher consistency in the summaries.

Contradiction. All systems scored low with respect to the amount of information in the text that is contradicted by the triples, and no statistically significant differences were noted.

Results of the Second Study

Training our systems on a much more challenging corpus results in minimal performance differences in comparison to the performance of the same systems when trained solely on Biographies. Nonetheless, we observed that when trained on the Full corpus and tested on biographies, the system equipped with surface form tuples is significantly (p < .05) more fluent than the one without.

4 Conclusion

We presented a data-driven approach to generate opendomain text summaries from knowledge base triples. We proposed a pointer-generator network that jointly learns to verbalise in a different number of ways the content from the triples, while retaining the ability to generate regular words from a fixed target vocabulary. We trained and evaluated two system variants on two different datasets of aligned DBpedia triples with Wikipedia summaries with respective vocabulary sizes of 400k and 1114k words.

We evaluated our approach using well-established automatic text similarity metrics and conducted two user studies. Our approach outperformed the state of the art; in particular, compared to other encoder-decoder architectures, our summaries are significantly more fluent and convey a greater share of the content of the input triples.

Acknowledgments

This research is partially supported by the Data Stories project, funded by EPSRC research grant No. EP/P025676/1.