Infobox-to-text Generation with Tree-like Planning based Attention Network

1 Introduction

Generating textual descriptions from structured data is a significant and challenging task, which can help people understand the key information from the complex non-linguistic data better. In this paper, we focus on generating fluent, faithful and logically coherent description from an infobox with a set of attributes, each of which can be regarded as a key-value pair. Figure 1 shows an example of writing a description from a given infobox which is in the form of a key-value table.

Previous works using neural models based on encoderdecoder architecture treat the task as an end-to-end learning problem [Lebret et al., 2016; Mei et al., 2016; Wiseman et al., 2017; Liu et al., 2019b]. Some works also exploit the structure information for better input representation [Jain et al., 2018; Liu et al., 2018; Gong et al., 2019], or model content selection for generation [Konstas and Lapata, 2012; Kim and Mooney, 2010; Mei et al., 2016; Shao et al., 2019]. Since all these neural methods represent the input data as a sequence, the order of the input data has a great effect on guiding the generation of descriptions. However, these methods pay little attention to the order-planning and thus have difficulty capturing the order information of the input. In our preliminary experiments, we find that models without explicit order-planning have an unstable performance on disordered input.

∗ indicates equal contribution.

†Corresponding author: zheng.haitao@sz.tsinghua.edu.cn.

In a real-world scenario, disordered input data are common and more in line with the practical application. Therefore, explicitly modeling the order-planning to guide the generation is of great necessity. There have been some methods taking the order-planning into account. Specifically, Sha et al. [2018] use a link matrix to determine the local order between different attributes while Puduppully et al. [2019] employ a pointer network to order the records of the input table. Based on Puduppully's work, Trisedya et al. [2020] further design a plan-based bag of tokens attention to handle disordered input. However, these planning methods still face the following problems. (1) Due to the complexity of the input data, the planning results cannot be guaranteed to be perfect, which may cause error propagation between planning and generation such as wrong order and missing information. (2) Worse still, the planning stage only makes a static order-plan which is unchangeable once determined. Consequently, the problems of error propagation may get severer because their generation is directly guided by the static order-plan.

Intuitively, if we take the obtained order-plan as an initial plan and utilize the dependency relation among different attributes to dynamically tune it, we can obtain a more reliable and coherent plan to guide the generation. An example is shown in Figure 1 which aims to generate a description of a person from a set of attributes {i.e. Origin, Instrument, Genre, Occupation, etc.}. We first order the attributes to an approximately proper order as the initial order-plan. Then considering the relation between different attributes, we reorganize them and merge different attributes layer by layer like a tree, to tune the initial order-plan. At each layer, the most relevant attributes are merged together. Finally, a wellorganized plan is obtained and can be used to guide the generation of a more logically coherent description (... an English folk guitarist and singer).

To this end, we propose a Tree-like PLanning based Attention Network (Tree-PLAN) which leverages both order-planning and dynamic tuning to facilitate infobox-totext generation. After representing the attributes in the input infobox with an attention based encoder, we first apply a pointer network to preliminarily order the input attributes and obtain an initial order-plan. As mentioned above, the initial order-plan is not enough to directly guide the generation, a tree-like tuning encoder is thus utilized to dynamically tune the initial order-plan for better planning. Specifically, we implement a merging attention mechanism to capture the dependency relations among different attributes to merge the most relevant attributes together layer by layer. At each layer, we design a hierarchical tuning mechanism that tunes the initial order-plan on both word-level and attribute-level. Finally a dual attention based decoder is employed to leverage both attribute-level attention and word-level attention to generate textual descriptions guided by the determined plan.

We conduct sets of experiments on two real-world datasets, which aim to generate a textual description of a person (or restaurant) from a given infobox. The experimental results show that our model outperforms state-of-the-art methods on automatic evaluation metrics and has better adaptability to disordered input. We also implement qualitative human evaluation to further estimate the quality of our model. The results indicate that our model can generate fluent, faithful and logically coherent descriptions.

2 RelatedWork

Traditional methods for data-to-text generation follow a pipeline of modules including content selection, sentence planning and surface realization [Barzilay and Lapata, 2005; Barzilay and Lapata, 2006; Liang et al., 2009]. Most recent works use end-to-end neural networks to generate textual descriptions directly from the input data or focus on exploiting the structure of data for better representation [Mahapatra et al., 2016; Wiseman et al., 2017; Kaffee et al., 2018;

Nie et al., 2018; Liu et al., 2018]. Some works also implement hierarchical attention mechanism to model the structure of tables on multiple levels [Jain et al., 2018; Liu et al., 2019a] or different dimensions [Gong et al., 2019]. To solve the problem of information loss, Liu et al. [2019b] propose a force attention method to force the generator to focus on more attributes of the input infobox.

Various studies have been conducted to model content planning explicitly to guide generation including rule-based planning [Konstas and Lapata, 2013] or content selection [Konstas and Lapata, 2012; Kim and Mooney, 2010; Mei et al., 2016; Shao et al., 2019]. Despite generating fluent and grammatically correct descriptions on an ordered input, these models have an unstable performance on disordered input without explicit order-planning. To model the order planning, Sha et al. [2018] design a link-based attention to capture the order information of input items while Puduppully et al. [2019] implement a pointer network to order the input data to guide the generation. Further, Trisedya et al. [2020] design a plan-based bag of tokens attention to handle the disordered input. Differing from these approaches, we propose a tree-like planning method to model both order-planning and dynamic tuning for better order representation.

For dynamic tuning, we employ a tree-like attention encoder that integrates tree structure to multi-head attention. Similar strategy is applied to grammar induction with Tree Transformer [Wang et al., 2019]. However, the Tree Transformer only calculates constituent attention between two neighboring words while our model computes a constrained attention over all the input attributes.

3 Methodology

The input of our model is a set of attributes x = {a1, a2, ...an}, each of which can be regarded as a key-value pair. The output of our model is the textual description of the input data with a sequence of words y={y1, y2, ..., yT }.

As shown in Figure 2, the architecture of Tree-PLAN is composed of the follow three parts: (1) Representation, where a set of multi-head attention encoders are utilized to represent the values of attributes in the input infobox, then each attribute is represented as a weighted sum of the associated values. (2) Planning, where a pointer network is applied for preliminary order-planning and further a tree-like tuning encoder is designed to dynamically tune the initial order-plan. (3) Generation, which implements a dual attention to decode and generate textual descriptions from the determined plans.

3.1 Representation

The input infobox is a set of attributes represented as keyvalue pairs ai =〈ki, vi〉. For each attribute ai, its value vi is flatten as a phrase vi ={wi,1, wi,2, ..., wi,m} where wi,j is the j-th word and m is the length of vi. We further represent each single word as a key-value pair vi,j = 〈ki,j,wi,j〉 according to the position of the word in the phrase where ki,j =[ki; pi,j ] and pi,j =j.

We first embed all the keys, positions and words into vectors (denoted as ke , pe and we ) then utilize a set of multihead self-attention [Vaswani et al., 2017] encoders to encode

3.2 Planning

After representing the input infobox, we implement a planning-based encoder to model order planning of the input. The planning process can be divided into two stages: attribute-aware order-planning and tree-like tuning.

Attribute-aware Order-planning

An attribute-aware order-planning is designed to organize the attributes in the input infobox into a reasonable order. A plan z={z1, z2, ..., z|z|} is a sequence of the input attributes which are in a new order. Since the output of the orderplanning stage corresponds to positions in the input sequence, we apply an attention based pointer network similar to [Wang and Wan, 2019] which is also an encoder-decoder architecture to point to the input attributes.

First a multi-head attention encoder is used to encode the input attributes then a multi-head attention decoder is applied to decode. At each decoding step t, we get the hidden state ht, then the probability P(zt|z<t, x) is computed as an attention over the input attributes:

where Wp ∈ Rdw×dw are parameters

According to the calculated probability distribution, we get the pointer index over the input attributes:

where ˆ I represents the index of the ordered attributes. Therefore we can then reorder all attributes (including all keys and values) to a proper order following ˆ I.

We assume that the order in which the attributes appear in the description is the golden order of input attributes. Following this rule, we automatically annotate the order of each

the value of each attribute:

where we i ∈Rm×dw are the words of the i-th attribute and dw is the dimension of word vectors. Multihead takes queries Q, keys K and values V as input and calculate as follows:

where Wo ∈ Rdw×dw , W h q ,W h k ,W h v ∈Rdk×dk are parameters, dk = dw/H, H is the number of attention heads, [; ] represents the concatenation operation.

Considering that different word in the same attribute may not contribute the same, an attention mechanism is implemented to estimate the contribution of each word in the same attribute. The contribution weight is calculated as follows:

where uv ∈ Rdw is the word context vector which is randomly initialized and learned during the training process. Wv ∈ Rdw×dw and bv ∈ Rdw are parameters, αi,j represents the contribution of vi,j . Then we represent the attribute ai as the sum of va i weighted by αi:

Finally we get the representation of the words {ve i }ni=1 and attributes ae .

attribute in the input data to get the gold plan and train this stage with supervision, which aims to minimize the negative log-likelihood of the gold plan:

where D represents all the training examples including input data, golden plans and target sentences.

The ordering stage ends when a special symbol eos is pointed. Noticing that the ordering stage could end before all items are pointed, which may cause information missing, we append the missing attributes in the back of the ordered attributes and obtain an initial order-plan z = {a, 〈k, p, v〉} including the ordered attributes with the corresponding keys, positions and values.

Tree-like Tuning Encoder

Since the initial order-plans could not be perfect, a dynamic tuning encoder is further designed to dynamically tune the order-plan for better planning, which aims to capture the dependency relations among different attributes and merge the most relevant attributes layer by layer. As shown in Figure 2, at each layer we employ a hierarchical tuning mechanism that tunes the order-plan on both attribute-level and word-level.

To this end, an extra constraint is added to attention heads and encourages the heads to follow the tree structure. Here we refer to the constraint as merging attention. To make each attribute attend to its most relevant one, the attention probability matrix A is modified with the merging attention:

where Q=Wqa, K =Wkk, Wq and Wk ∈ Rdw×dw are parameters, k are the keys of attributes. C is the merging attention and Ci,j = Cj,i represent the dependency relation between ai and aj. Higher value of Ci,j means ai and aj are encouraged to merged at this layer.

The merging attention is calculated with attributes a and keys k and is updated layer by layer:

whereWc ∈ Rdw×dw are parameters, l represents the current layer, Eq.(13) is to make sure C l i,j and C lj,i are the same.

The tuning encoder is an N-layer tree structure where C 0 is initialized to 0 and Eq.(14) guarantees that C l is larger than C l-1 . Furthermore, considering the special key-value structure of the attributes, each layer is designed as a hierarchical architecture for better planning, where two attention blocks are employed to update the representation of the order-plans on both word-level and attribute-level.

For attribute-level, we utilize the computed attention matrix A to encode the represented attribute ˆa=Aa. Then ˆa is used to update the representation of each word to ˆ v via a gate mechanism.

For word-level, the updated word ˆ v is encoded by a multihead attention layer. Then a content selection mechanism same as Eq.(5-7) is applied to update the representation of the corresponding attribute.

Similar as the transformer encoder, each layer also contains a feedforward sub-layer and a layer normalization. After an N-layer merging and update, the input infobox is encoded on both attribute-level ˆa and word-level ˆ v.

3.3 Generation

As the input infobox has been encoded on both attribute-level and word-level, a dual attention based decoder is then applied to decode and generate textual descriptions with the guidance of the determined plans.

The structure of the decoder is similar to the Transformer decoder. At each decoding step t, a multi-head self-attention is first utilized to capture the dependency from the generated words y<t and obtain the hidden state ht. Then, to leverage the information of both attribute-level and word-level of the input data, we employ the dual attention [Gong et al., 2019] to first choose the most relevant attribute based on the key of the attribute then attend to the words in the attribute and obtain the modified attention weights γ. Then we employ another multi-head attention layer to update the hidden states:

where Multiheadγ is calculated based on the modified attention as mentioned above, kv are the keys of the words, kv i,j =ki+pi,j . After a linear layer with a softmax activation, the probability of generated results yt is obtained as:

where Wy ∈ Rdw×V is the word embedding matrix and V is the vocabulary size.

The goal of the generation stage is to minimize the negative log-likelihood of the sentences in the training set:

where T is the length of the target sentence.

3.4 Training

We train our model end-to-end with a joint learning of both planning and sentence generation by aggregating the losses over the two stages:

where L1 and L2 are calculated by Eq. (10) and Eq. (17), λ is hyper parameter.

4 Experiment

4.1 Experimental Setups

Dataset and Metrics

We conduct experiments on two datasets: (1) WIKIBIO aims to generate the first sentence of a biography from a given Wikipedia infobox. (2) E2E [Novikova et al., 2017] aims to generate descriptions of restaurants from dialogue act-based meaning representations. We follow the previous works [Lebret et al., 2016; Liu et al., 2018; Liu et al., 2019a; Liu et al., 2019b] that use BLEU and ROUGE as the automatic evaluation metrics.

Training Configuration

During experiments we set the dimension of word vectors and hidden state dw = 300 and the number of heads H = 6. We choose the most frequent 30, 000 words in the training set as the vocabulary of WIKIBIO. A copy mechanism is applied to replace the unknown words UNK with the most likely word in the input data according to the attention distribution. We train the two stages separately for 10 epochs then jointly for 40 epochs. During joint training we set λ = 0.4.

4.2 Baselines

We compare Tree-PLAN with some strong baseline models as follows, where the former three do not perform planning and the latter three are planning based:

TableNLM [Lebret et al., 2016] is a neural language model which integrates field and position embedding into the data representation.

Struct-aware [Liu et al., 2018] is a seq2seq architecture with a gate mechanism to introduce field information and a dual attention to incorporate the attribute information.

FA+RL [Liu et al., 2019b] is a neural model with force attention and reinforcement learning to force the generator to attend to more attributes of the input infobox.

Order-plan [Sha et al., 2018] is a seq2seq model where a link matrix is designed to model the order of the attributes for infobox-to-text generation.

PHVM [Shao et al., 2019] is a planning-based hierarchical variational model with a high-level planning and a low-level realization for long and diverse text generation.

NCP+BTA [Trisedya et al., 2020] is a neural contentplanning based model with bag of tokens attention which uses a joint learning of order-planning and sentence generation.

4.3 Overall Results

Table 1 shows the results of automatic evaluation. TreePLAN outperforms all the baselines on both BLEU and ROUGE scores (about 1.62/1.28 increase on BLEU/ROUGE compared to the state-of-the-art method), indicating that our proposed model can truly facilitate the infobox-to-text generation. The improvements (about 2.42/2.32 increase on BLEU/ROUGE) compared to the model without planning demonstrate that the proposed order-planning method ia able guide the model to generate higher quality descriptions.

Without attribute-level ordering, our model gets a 1.14/1.24 decrease on BLEU/ROUGE, which proves that explicit order-planning is able to improve the performance of our model. Moreover, the decline of performance without tree-like dynamic tuning (about 0.66/0.45 decrease on BLUE/ROUGE) illustrates that the tuning mechanism can tune the initial order-plan for better planning, thus improving the performance of our model.

4.4 Human Evaluation

Since BLEU and ROUGE are calculated based on n-gram matching, automatic evaluation is not enough to evaluate the quality of our model. Therefore we implement human evaluation to evaluate on three aspects: Grammar (whether a generated sentence is fluent without grammatical error), Faithful (whether the output is faithful to input), and Coherent (whether a sentence is logically coherent and the order of expression is in line with human writing habits). We randomly select 300 samples with the descriptions generated by three well-performing baselines and our proposed model. We invite five annotators with sufficient background knowledge to score the given generated descriptions.

The results are reported in Table 2, showing that TreePLAN outperforms other models on the three metrics. We find that all the reported models get high scores on Grammar, which illustrates that neural encoder-decoder models are able to generate fluent and grammatically correct descriptions. Compared to other reported models, Tree-PLAN gets a remarkable improvement on Faithful and Coherent (about 0.12 and 0.15 increase), indicating that the proposed planning method can guide to generate more faithful and logically coherent descriptions. The results of human evaluation demonstrate that Tree-PLAN is able to generate more fluent, faithful and logically coherent descriptions.

4.5 Evaluation of Planning

To further demonstrate the effectiveness of the proposed order-planning, we conduct disorder experiments to evaluate the adaptability to disordered input of Tree-PLAN. We disorder the input attributes and get ten different disordered datasets (including a gold order). We test some baselines and Tree-PLAN on the ten datasets and report the maximum, minimum, mean and deviation of the BLEU score in Table 3. The results show that the order of the input attributes has a significant influence on the performance of neural models.

Tree-PLAN outperforms all other models with higher maximum, higher minimum, higher mean and lower deviation. Higher maximum, minimum and mean indicate the highperformance of Tree-PLAN while lower deviation demonstrate the adaptability of Tree-PLAN to disordered data, which further proves the effectiveness of the proposed planning methods. Furthermore, models with planning have better adaptability to disordered data by having lower deviations, which proves that planning can not only facilitate infoboxto-text generation but also improve the adaptability of neural models to disordered input.

4.6 Case Study

Figure 3 shows the descriptions generated by different models of the same input infobox from the test set of WIKIBIO.

Struct-aware misses some important attributes (Birthplace, Position, etc.) while FA+RL gets a better coverage thanks to its force attention mechanism. However, FA+RL still misses some attributes and suffers from the problems of groundless information. Compared to these two models, Tree-PLAN can generate a more faithful and coherent description.

To further show the effect of our tree-like planning, we visualize the results of planning stage in Figure 3 including the initial order-plan and merging attention heat maps. The result of order-plan shows that the multi-head attention based pointer network can generate a approximately proper orderplan. The attention heat maps illustrate how the tuning stage works. The relevant attributes are merged to larger ones layer by layer (e.g. Birthdate and Birthplace are merged at the first layer where the attention weight between the two attributes is high). And finally the input attributes are all merged together in the last layer and a logically coherent plan is obtained.

5 Conclusion

In this paper, we propose a tree-like planning-based attention network (Tree-PLAN) which leverages both static orderplanning and dynamic tuning to guide infobox-to-text generation, where a novel tree-like dynamic tuning mechanism is proposed to dynamically tune the static order-plan for better planning. Experiments on two datasets show that TreePLAN achieves the state-of-the-art performance and can generate more fluent, faithful and logically coherent descriptions.

Acknowledgments

This research is supported by National Natural Science Foundation of China (Grant No. 61773229 and 61972219), Shenzhen Giiso Information Technology Co. Ltd., the Basic Research Fund of Shenzhen City (Grand No. JCYJ20190813165003837), Tencent AI Lab Rhino-Bird Focused Research Program (No. JR202032) and Overseas Cooperation Research Fund of Graduate School at Shenzhen, Tsinghua University (Grant No. HW2018002).