Triple-to-Text Generation with an Anchor-to-Prototype Framework

1 Introduction

We consider the problem of automatically generating appropriate natural sentences for a given set of RDF triples, namely triple-to-text generation. Defined by the W3C standard [Miller, 2001], a Resource Description Frameworks (RDF) triple is in the form of 〈subject, predicate, object〉, which can be used to represent the relation between resources (the subject and object) in a structured knowledge base. Generating textual description from RDF triples can be applied to many scenarios for better user interaction such as question answering [Bordes et al., 2014; Fader et al., 2014], search engines [Ding et al., 2004] and profile summarizing [Chisholm et al., 2017].

Great efforts have been devoted to the sequence-tosequence [Sutskever et al., 2014] based neural models for triple-to-text generation, which often generate sentences from scratch in a left-to-right manner. [Vougiouklis et al., 2018; Distiawan et al., 2018; Zhu et al., 2019; Koncel-Kedziorski et al., 2019; Moryossef et al., 2019]. However, due to the huge gap between the structured input and the unstructured output, the input triples alone are insufficient to decide and acquire an informative and specific description. As a result, these models prefer generic utterances and tend to produce repeated and irrelevant words.

∗ indicates equal contribution.

†Corresponding authors.

To alleviate such problems, given the input triples, we can retrieve similar triples and the associated description, termed as prototype triples and prototype description, to guide tripleto-text generation. Intuitively, a human-written prototype description is informative and expressive, which can provide concrete patterns that faithfully represent the information encoded in the prototype triples. For example, to describe 〈William, mission, NASA〉, a prototype triple 〈Alan, mission, Apollo12〉 and its description "Alan was a crew member of Apollo12" are first retrieved from the training data. Then a model can imitate the prototype to use the pattern "a crew member of " to express the mission relation and write its own description as "William served as a crew member of NASA".

Despite the benefits of leveraging prototypes, as the number of triplets in the input increases, both the reference and the retrieved prototypes will become more complex and verbose. In this case, the model is unable to precisely learn writing guidance from the complicated prototypes. Consequently, the model tends to overlook the retrieved prototypes and degrades to a vanilla seq2seq model [Cai et al., 2019b]. As we made statistic on the WebNLG dataset [Gardent et al., 2017], samples with multiple triples accounts for over 76%. Such proportion cannot be ignored.

To this end, we propose a novel framework, anchor-toprototype, to facilitate triple-to-text generation. Given the input triples along with the retrieved prototype triples and description, we first employ a transformer-based triple encoder to represent the input triples. Then, to extract writing patterns from the retrieved prototypes, a guidance extractor is used to extract a guidance vector based on the difference among the input triples, the prototype triples and the prototype description.

As mentioned before, with multiple triples as input, the retrieved prototypes could become too complex to be made precise use. Thus, before incorporating the guidance vector into our decoder, we additionally apply a triple anchor to align the input triples into groups to better match the retrieved prototypes. A concrete example illustrating our idea is shown in Fig 1. Finally in the decoding stage, the hidden state of the triple anchor is used to revise the guidance vector via a gate mechanism, so as to dynamically control the information flows from the prototypes to the decoder.

Our experiments are conducted on both English and Chinese datasets towards triple-to-text generation. Experimental results show that our model outperforms state-of-the-art approaches in automatic metrics as well as human evaluation. In addition, we also conduct extensive ablation studies to quantify the improvement from each designed component of our model.

2 RelatedWork

Recently, neural models have become the mainstream for triple-to-text generation. As a pioneer, Vougiouklis et al. [2018] first employ a feed-forward neural network to encode the input triples and add them into a LSTM decoder to generate words. Based on Vougiouklis's work, various techniques have been developed to make further improvement, including encoding relationships among triples [Distiawan et al., 2018; Koncel-Kedziorski et al., 2019; Cai and Lam, 2020], modifying training objective [Zhu et al., 2019] and explicitly modeling text planning [Moryossef et al., 2019]. The main drawback of these models is that they often generate sentences from scratch, while the input triples alone are insufficient to decide an expressive and specific description.

Several studies design templates (prototypes) to guide the text generation from triples [Kukich, 1983; Duma and Klein, 2013; Wiseman et al., 2017], but they mostly use either fixed or hard templates, which are limited in scalability and flexibility. By contrast, our model is guided by retrieved prototypes that served as soft templates. Additionally, the idea of utilizing retrieved prototypes has also been exploited in other generation task such as unconditional text generation [Guu et al., 2018], dialogue response generation [Wu et al., 2019; Cai et al., 2019a; Cai et al., 2019b] and abstractive summarization [Cao et al., 2018; Wang et al., 2019]. Different from these methods, to precisely learn writing guidance from complicated prototypes, our framework models a grouping stage to control the information flows from the prototypes.

3 Models

The input to our model is a set of RDF triples x = {t1, t2, ..., tN}, where ti = 〈si, pi, oi〉 is the i-th triple in x (elements si, pi and oi correspond to subject, predicate and object respectively). Each element can contain multiple words. While the output y = y1y2...yM (yj is the j-th word) is a paragraph that faithfully represents the information encoded in x.

To facilitate the generation from x to y, a similar set of triples x′ and its associated description y′ are retrieved from the training data, which are called prototype triples and prototype description respectively. Then the goal of our model is to maximize p(y|x, x′ , y′ ).

As shown in Fig 2, our model consists of three main components: (a) triple encoder, which encodes the input triples into vector representation via a set of multi-head attention layers. (b) guidance extractor, which learns a guidance vector from the retrieved prototype to guide the generation. (c) triple anchor, which groups the input triples to match the prototype so that the decoder can make a better use of the learnt guidance.

3.1 Triple Encoder

The triple encoder converts the input triples x into embedding vectors and then encodes the vectors via multihead attention layers. We first flatten the input triples x = {t1, t2, ..., tN} into a sequence of words ¯x = {w1,1,w1,2, ...,w1,|t1|, ...,wN,|tN|}, where wi,j is the j-th word in the i-th triple. To facilitate input representation, a special token [CLS] is added into the beginning of ¯x .

For each word in the input, its input embedding is the sum of its word and position embeddings. Unlike the conventional position embedding that only records word indices in the input sequence, we employ local and global position to encode both the intra and inter positional information among triples.

Specifically, for a word wi,j in ¯x, its local position p′i,j = j is the word index within the associated triple while the global position pi,j = i corresponds to the triple index in the input. Then we embed the word, local position and global position into vector representations (denoted as ¯x, p′ and p) and computed the embedding vectors of x as the summation of them:

Besides, the entity type information of each element in the input is also considered. We obtain a sequence of type labels xtype = {l1,1, ..., lN,|tN|} provided by the datasets and map xtype into embedding vectors xtype, which will be utilized to enhance representation in the encoder layers. The input embedding x and type embedding xtype share the same number of dimensions dk and both are optimized as parameters during the training process.

After obtaining the embedding vectors of the input, we feed the vectors into our triple encoder. The encoder is based on transformer [Vaswani et al., 2017], which consists of a stack of identical multi-head attention layers. Each layer takes queries Q, keys K, and values V as input and is for-

where W , W Q i , W K i andW V i are trainable parameters, [; ] represents the concatenation operation and H is the number of attention heads. A fully connected feed-forward network with residual connection and a layer normalization are then added behind the MultiAttn to form an encoder layer Block(Q, K, V ).

Finally, based on the above calculating process, we stack L layers to encode the input from both the input embedding x and type embedding xtype, which can be shorten as:

where zL is the final representation of the input x, which will be fed into the following modules of our model.

3.2 Guidance Extractor

The guidance extractor aims to extract a guidance vector based on the difference among the input triples x, the prototype triples x′ and the prototype description y′ . According to [Wu et al., 2019], the extracted vector should absorb appropriate content from the prototype while discarding irrelevant words regarding the current input.

Following this idea, to construct the guidance vector, we first define two word sets: insertion words I = {w|w ∈ y′ ∩ w /∈ x′ } and deletion words D = {w|w ∈ x′ ∩ w /∈ x}. The insertion words I consider words in the writing patterns derived from the prototype description y′ , while the deletion words D correspond to words that only specific to the prototype triples x′ . Then, we concatenate the representation of the two word sets to form a textual difference vector g∗ :

where e(·) converts a word into its word embedding. The value αwI is the weight of a word wI in the insertion words, which is derived by an attention mechanism with vI and WI as parameters:

Here, zL 0 is the hidden state of the [CLS] token in Eq. 5, which can be regarded as a sentence representation of the input x. In addition, βwD , the weight of a deletion word wD in D, can be obtained with a similar process:

Finally, we use a dense layer with parameters WG and bG to transform g∗ to the guidance vector g:

then the guidance vector g will be fed into the sentence decoder.

3.3 Triple Anchor

As discussed in Section 1, the retrieved prototypes could be too complex for the decoder. Thus, before moving on to the decoding stage, the triple anchor splits input triples into groups to align the prototypes, where a group corresponds to a sentence in the description. Then the decoder can make a more precise use of the extracted guidance vector.

Inspired by the content selection applied in [Hua and Wang, 2019; Shao et al., 2019], we extend the Memory Network [Sukhbaatar et al., 2015] to model the grouping of triples. The whole process can be regarded as selecting a subset of the input triples to be covered for the current group in each time step.

Given the representation of the input triples zL from Eq. 5, we first sum up all the elements in the same triple in zL to obtain triple representations T = {t1, t2, ..., tN}, where ti = ∑j zL i,j is the representation of the i-th triple. To control the start and end of the selection process, two special entries [START] and [END] are also added into T. Then, we transform T to hidden vectors {hei |hei = [-→h ei ; ←- h ei ]}N+2 i=1 through a biLSTM, where hei is the hidden state of the entry ti.

After obtaining the hidden state of each entry in T , we use a LSTM decoder to make selection of triples. Starting with the first group that only contains the [START] entry, the decoder recurrently selects triples for the next group until the [END] entry is selected. Formally, at time step t, the hidden state of the decoder hdt is computed as:

where selt,i ∈ {0, 1} is the selection for the i-th entry in the t-th group, and st is the summation of all the currently selected entries. Then, the hidden state hdt along with all the previously selected entries pt are used to calculated the selection for the next group, which is treated as a binary prediction for each input entry:

where σ is the sigmoid function. pt records entries that have been selected before, which can prevent the model from repeatedly picking the same triples.

3.4 Sentence Decoder

Our sentence decoder basically follows the same spirit of the Transformer decoder, in which L decoder layers are stacked. Each decoder layer has the similar structure with the block in the encoder, while with an addition of masked multi-attention layer before the feed-forward layer.

At time step t, the decoder is fed with the input representation zL learnt from the encoder and the embedding of the predicted output y<t, and outcomes a sequence of hidden states yL. Then the probability of generating the next target word yt is jointly decided by the t-th hidden state yL t , the guidance vector g and the triple anchor's state hd . Specifically, we first calculate the weighted sum of hd to form an anchor representation c based on the guidance vector g, which is given as:

Then we use the anchor representation c to calculate a gate probability that determines how much information of the guidance vector should flow into the output sequence:

whereWH and bH are trainable parameters. Finally, the output probability of the next word yt is estimated by y∗ t via a softmax function over the target vocabulary.

3.5 Training Objective

There are two optimization objectives in our work:

1. loss1 is the cross entropy for sequence generation between the predicted token distribution and the reference distribution.
2. loss2 is the loss function of the triple anchor which utilizes the binary cross-entropy loss with gold-standard selection as criterion over the training set. The goldstandard selection is manually annotated so that the grouping of triples can be trained with supervision.

The total loss training objective is given by:

where γ is a hyper-parameter for trade-off between loss1 and loss2 and controls the weight of the explicit supervision.

4 Experiments

4.1 Datasets

We conduct experiments on two benchmark datasets: WebNLG [Gardent et al., 2017] and Baidu SKE 1 . The WebNLG dataset is driven from DBPedia and contains 25,298 triple-text pairs, with 18,102 samples for training, 2,268 for validating and 4,928 for testing. The Baidu SKE

1 http://ai.baidu.com/broad/introduction

is a large-scale human annotated dataset based on real-world Chinese corpora. We follow Zhu et al. [2019] who only used samples related to the film domain. The number of samples for training, validating and testing are 28,597, 3,177 and 3,972 respectively.

4.2 Implementation Details

We implement the triple anchor based on a bidirectional recurrent neural network with 512 LSTM units. The trade-off γ between loss1 and loss2 is 0.9. We use the Adam optimizer and the learning rate of it is 1e-5. For the encoder and decoder, the number of multi-head attention layer L is 4 and the rest parameters are followed by the setting of the basic Transformer [Vaswani et al., 2017].

For better comparison, we use the similar strategy as Wu et al. [2019] to retrieve prototypes for each sample. Specifically, we first employ Lucene 2 to construct a pre-defined index based on description similarity 3 . For each training pair (x, y), top 20 similar descriptions with their associated triples are retrieved, denoted as {(x′i , y′i )}20 i=1 . Furthermore, to avoid the input and the prototype being too similar or too different, only the pairs that satisfy 0.25 ⩽ Jaccard(y, y′i ) ⩽ 0.75 can be reserved, where Jaccard measures the Jaccard similarity. If no prototype is found or no pair is reserved, a special prototype [DEFAULT] will be provided.

4.3 ComparedMethods

We compare our model with several approaches:

NMT [Bahdanau et al., 2015] is the standard attentionbased Seq2Seq baseline.

Transformer [Vaswani et al., 2017] puts forward an encoder-decoder model based on multi-head attention layers. GTR-LSTM [Distiawan et al., 2018] improves the LSTM encoder by considering relationships between the triples.

Step-by-Step [Moryossef et al., 2019] explicitly models content selection and order planning for triple-to-text generation by manually designing features.

GraphWriter [Koncel-Kedziorski et al., 2019] is a transformer model with graph encoding mechanism to encode both the entities and relations from the input.

EditVec [Wu et al., 2019] is a prototype-based model proposed for response generation.

2 https://lucene.apache.org/core/

3 For test data, the pre-defined index is based on triples similarity

4.4 EvaluationMetrics

We calculate scores on both automatic and manual evaluation. For automatic evaluation, we adopt BLEU [Papineni et al., 2002]. For human evaluation, we evaluate the quality of a generated description on three criteria: Grammar (whether the output is Grammatically correct), Expression (whether the output is fluent and uses appropriate words to express) and Correctness (whether the output correctly represents the information in the input). The score for each criterion takes an integer from 0 to 5 (the higher the better). We randomly select 200 sets of input triples in test set from both the WebNLG and Baidu SKE datasets, along with corresponding descriptions produced by our model and other baselines. Five welleducated graduate student are asked to score the descriptions. Notice that method identifiers are masked during the human evaluation.

4.5 Overall Results

The experimental results on WebNLG and Baidu SKE are summarized in Table 1. There are multiple observations as follows:

(1) Generally, our model substantially and consistently outperforms the existing methods by a noticeable margin on both automatic and human evaluation metrics. The main strength of our model comes from its capability of leveraging anchor to prototype that are complementary to each other for tripleto-text generation.

(2) Our model outperforms prototype-based model (EditVec) in term of Expression metric, verifying the benefits of incorporating triple anchor to better match the prototypes. The advantage of Step-by-Step is that it employs manual design rules to make use of the standard Seq2Seq framework. Nevertheless, the performance of our model performs better with less labor intensity and time costs.

(3) The performance of baseline models in terms of Grammar metric cannot perform as well as our model, which shows that our model can produce descriptions close to human writing. With the combined exploration of guidance extractor and triple anchor, our model effectively bridges the gap between structured input and unstructured output, and overcomes the issue of complex and verbose prototype, thereby providing fluent and informative description.

4.6 Ablation Study

In this section, we conduct ablation study to systematically evaluate the effect of our guidance extractor and triple anchor, which is reported in Table 2. In order to perform more comprehensive evaluation, we split the test set based on the number of input triples: single triple and multiple triples. In addition to the BLEU score, we additionally introduce two extra metrics: (1) Coverage (Cov.) measures the overlap of words (entities and relations) between input triples and output descriptions. (2) Originality (Ori.) calculates the proportion of words that are not in the input but both in the output and references.

The results from Table 2 show that equipped with both the guidance extractor and triple anchor, the complete version achieves almost the best performance on both data splits. In the case of input with single triple, the coverage and originality drop without the guidance extractor, which suggests that the guidance extractor could maintain the provided words from the input and create new words to add the diversity. However, since the input is too simple, the triple anchor has little influence in this situation.

In the situation of multiple triples with more information, the coverage and originality drop dramatically in comparison with the single one. In this case, although the guidance extractor still have impact, it has become smaller than the single input case. Meanwhile, thanks to the triple anchor which groups triples to better match the prototypes, the model with the triple anchor performs better in terms of all metrics.

4.7 Case Study

An example with multiple triples in WebNLG along with generated results from different models are shown in Fig 3, from which we can observe that: although the Step-by-Step outputs acceptable description, there are some grammatical errors and the sentences appear mechanically and not fluently. Due to the complexity of the input, the EditVec mixes all the triples into a single sentence with logical confusion ("is now retired before retiring") and groundless information ("credit dimitia mission"). By contrast, our model precisely learns Table 2: Ablation study of our model on WebNLG datasets. "Single" and "Multiple" correspond to data splits with single triples input and multiple triples input respectively writing patterns (e.g., "was a member of 's crew") from the retrieved prototype to describe the input triples and the generated result is fluent, expressive and faithful to the input.

5 Conclusion

In this paper, we present a novel framework, Anchor-toPrototype, to facilitate triple-to-text generation. To bridge the information gap between the structured input and a natural text, the model retrieves sets of human-written descriptions as prototypes from the training data to guide the text generation from the input triples. What's more, we employ a triple anchor that aligns the input triples into groups to better match the prototypes, so that the model can make a more precise use of the retrieved prototypes.

Acknowledgments

This research is supported by National Natural Science Foundation of China (Grant No. 61773229 and 61972219), Shenzhen Giiso Information Technology Co. Ltd., the Basic Research Fund of Shenzhen City (Grand No. JCYJ20190813165003837), Tencent AI Lab Rhino-Bird Focused Research Program (No. JR202032), Overseas Cooperation Research Fund of Graduate School at Shenzhen, Tsinghua University (Grant No. HW2018002) and the Shenzhen General Research Project (No. JCYJ20190808182805919).