Better AMR-To-Text Generation with Graph Structure Reconstruction

1 Introduction

Abstract Meaning Representation (AMR) is a popular semantic formalism in representing the meaning of natural language text. AMR abstracts away from the surface form of a sentence and encodes its meaning as a rooted and direct graph, where nodes denote the concepts and edges denote the relations between the concepts. AMR-to-text generation is the task of recovering a text representing the same meaning as a given graph and it has attracted lots of attention in recent years. Because the function words and syntactic realizations are abstracted away and numerous details including tense, number, and definiteness in AMR graph are underspecified, this task is very challenging.

Off-the-shelf approaches for neural machine translation have been explored for AMR-to-text generation. Konstas et al. [2017] first transform the graph into sequence and apply sequence-to-sequence model to solve the task. Such a method may lose the information of reentrant structure. Recent works regard this task as a graph-to-sequence learning problem [Beck et al., 2018; Song et al., 2018; Damonte and Cohen, 2019]. These studies propose various graph neural networks to encode the graph, which focus on aggregating information from one-hop neighbors. The current state-of-the-art methods propose using relation encoders to model relation of indirectly connected concepts by encoding the shortest path between nodes [Zhu et al., 2019; Cai and Lam, 2019]. They extend the encoder in the Transformer and use relative position encoding to utilize the information captured by the relation encoder. However, these graph-to-sequence models bring errors like missing information from the input graph and mistranslation of the semantic relations between the concepts [Konstas et al., 2017] , which indicates that some graph structure information is not captured in the node representations. An example is shown in Figure 1.

To enhance graph structure learning, we propose a novel approach that generates natural language texts from AMR graphs while reconstructing the input graph structure. We adopt the Transformer encoder-decoder architecture and propose a variant of Transformer, which employs graph attention mechanism to aggregate information from one-hop neighborhoods. To learn better node representations, we propose to optimize two simple but effective auxiliary reconstruction objectives, i.e. link prediction objective and distance prediction objective. Link prediction requires the model to predict the relation between two given nodes or predict the target node (or source node) given a source node (or target node) and a labeled edge. Distance prediction requires the model to predict the distance between two given nodes. All the information to be predicted is explicitly or implicitly provided in the input graph and we require the model to reconstruct the graph structure based on the learned node representations. The former encourages the model to encode the neighbor relation information as much as possible in the node representations, and the latter helps the model to distinguish information from neighboring nodes and distant nodes. The learned node representations with the aid of the two objectives can reduce the errors of mistranslation of semantic relations and missing information in the output texts.

Experimental results on two benchmark datasets show that our model substantially outperforms the prior methods and achieves a new state-of-the-art performance. Our model improves the BLEU scores by 2.4 points on LDC2015E86 and 2.1 points on LDC2017T10, respectively. In all, our contributions can be summarized as follows:

• We propose a novel approach that generates texts from AMR graphs while reconstructing graph structures. Two simple but effective reconstruction objectives are optimized during training, which help better capture the information provided in the graph. 1
• Our proposed variant of Transformer shows its effectiveness on AMR-to-text generation.
• Empirical studies on two benchmark datasets exhibit that our model advances the state of the art for the AMRto-text generation task.

2 Our Approach

Our model adopt the Transformer encoder-decoder architecture, which can generate natural language texts from AMR graphs while reconstructing the input structure. In this section, we begin by providing the notations we use, followed by describing our variant of the Transformer model, including a graph encoder which employs graph attention mechanism to aggregating information of incoming neighbors and outgoing neighbors, respectively, and a sentence decoder which generates sentence with copy mechanism. Then we introduce the proposed graph structure reconstruction objectives. Finally, the objective functions will be detailed. The overall architecture of our model is shown in Figure 2.

2.1 Notations

Let G = {V, E, R} denote an AMR graph, where V is a set of N nodes, E is a set of M edges, and R is a set of L edge label types. N, M, and L are the number of nodes, edges and label types, respectively. Each edge e in E is denoted as (i, j, ri,j), where i and j are the indices of source node and target node, respectively, and ri,j ∈ R is the edge label. In addition, we denote the neighbor nodes reached by incoming edges of node vi as N in i and the neighbor nodes reached by outgoing edges as N out i . The distance di,j of a given node pair is defined as the length of the shortest path from vi to vj (regardless of the direction of the edges).

1 The code is available at https://github.com/sodawater/ graph-reconstruction-amr2text.

For convenience, we denote queries, keys and values for attention in Transformer as Q, K, and V . Let MHAtt(Q, K, V ) denote the multi-head attention, FFN(x) denote the feedforward network, and LN(x) denote the layer normalization.

2.2 Graph Encoder

The purpose of the graph encoder is to directly encode the input graph and learn the representation for each node. It is composed of a stack of L1 identical graph layers, where different parameters are used from layer to layer. Each layer has two sub-layers: a graph attention mechanism and a feedforward network.

At each layer, we first update the node representations by aggregating a weighted average from neighbors by using learned attention weights. Let hl = (hl1 , hl2 , ..., hlN ) ∈ Rdmodel×N be the node representations learned at layer l, where dmodel is the dimension size of the model. In particular, h0 are the linearly transformed input embeddings.

where We ∈ Rdmodel×demb is the transformation matrix and xni ∈ Rdemb is the word embedding of node vi, and demb is the dimension size of the embedding. Considering that AMR is a directed graph, neighbor nodes reached by incoming edges and nodes reached by outgoing edges play different roles and contribute different information to the central node. We perform graph attention mechanisms over incoming neighborhoods and outgoing neighborhoods, respectively. We use an additive form of attention in our model. The attention score of outgoing edge (i, j, ri,j) for node vi is computed as follows:

where [; ; ] is the concatenation operator, xri,j ∈ Rdmodel is the embedding of edge label ri,j, and Wou1 ∈ Rdmodel×3dmodel and Wou2 ∈ R1×dmodel are the parameters. Similarly, the attention score of incoming edge (j, i, rj,i) is computed as follows:

Then the attention probabilities -→α li,j and ←- α li,j are calculated as a softmax over the scores -→e li,j and ←-e li,j , respectively. We further aggregate the information of neighbor nodes and corresponding edges using these attention probabilities.

where Win3 , Wou3 ∈ Rdmodel×2dmodel .

Following Veliˇckovi´c et al. [2017], we also extend the graph attention mechanism to employ multi-head attention, which is beneficial for stabilizing the learning process. We split the attention into K heads and perform K independent attention mechanisms to execute the computation of the weighted average, which are further concatenated to get the final representation.

where f is the concatenation operator. ←-g li is computed in the same way.

Since the above graph attention mechanism does not aggregate the information of the central node, we use a linear transformation layer to combine the information of the central node and neighborhoods.

where Wc ∈ Rdmodel×3dmodel and bc ∈ Rdmodel are the parameters.

The second sub-layer is a fully connected feed-forward network. Residual connection and layer normalization are employed for connecting the adjacent layers.

A linear transformation layer is employed upon the encoder stack for aggregating the outputs of different layers.

where the final representation for node vi is denoted as hi for convenience, and Wl ∈ Rdmodel×(L1+1)dmodel and bl ∈ Rdmodel are the parameters.

2.3 Sentence Decoder

The sentence decoder in our model has an architecture similar to the decoder in Transformer. It is composed of L2 identical layers, where parameters are different from layer to layer. Each layer has three sub-layers: a multi-head self-attention mechanism, a multi-head attention mechanism over the node representations, and a feed-forward network.

At each layer, we first update the token representations by a self-attention mechanism. Let ˆh = (ˆhl1 , ˆhl2 , ..., ˆhlT ) ∈ Rdmodel×T represent the token representations at layer l. In particular, ˆh0 are sum of the linearly transformed input embeddings of tokens xt and position encoding pe.

Note that a masking is used for ensuring that the attention and prediction for position i depend only on the known words at position preceding i.

Following the self-attention, we employ a multi-head attention over the output of the encoder and a feed-forward network.

For convenience, we denote the final representations of the tokens in the decoder as ˆh. The final output is transformed and passed through a softmax layer to generate the probability pgi of next word over the vocabulary

where Wg ∈ Rdvocab×dmodel , bg ∈ Rdvocab , and dvocab is the size of vocabulary.

We apply a copy mechanism to tackle the data sparseness problem. A gate for controlling generating words from vocabularies or copying words from inputs is used.

where Wη1 ∈ Rdmodel×2dmodel , Wη2 ∈ R1×dmodel , bη1 ∈ Rdmodel and bη2 ∈ R1 are the parameters. We use the average of the K independent attention probabilities αl,k i,j of the last multi-head attention sub-layer as the copy probabilities pci .

where zj is the one-hot indicator vector for the node vj. The final distribution pi is the weighted average of the two probabilities with gate θi.

2.4 Graph Structure Reconstruction

To learn better node representations from the graph structure, we propose to optimize two simple but effective auxiliary reconstruction objectives.

The first objective is based on link prediction, which requires the model to predict the semantic relations for the given node pairs. For a given pair of nodes (vi, vj), we concatenate the representations of these two nodes and employ a multi-layer perceptron to predict the corresponding semantic relation.

where Wr ∈ R(L+1)×dmodel , br ∈ RL+1 , and L is the number of semantic label types in the AMR graph. For the pair of nodes that are adjacent in the graph, i.e., connected with an labeled edge, the gold relation label is exactly the given semantic relation ri,j. For the pair of nodes that are not adjacent, the gold label is non-adjacent.

The link prediction can be expressed in another way, which requires the model to predict the target node (or source node) given a source node (or target node) and a semantic relation type. In other words, we aim to predict the node vj in relation (vi, vj, ri,j) when vi and ri,j is given. We employ a pointer module to predict the node.

Obviously j is the gold answer for k∗ . Either of the above two forms of link prediction encourages the model to exactly encode the neighbor relations as much as possible in the node representations, so less information will be missing and less semantic relation will be mistranslated during decoding.

The second objective is based on distance prediction, which requires the model to predict the distance between a pair of nodes in the graph. As mentioned before, the distance di,j of a pair of nodes is defined as the length of the shortest path from vi to vj regardless of the edge direction. In the graph encoder, non-local information is captured by using multiple aggregation layers. This objective helps the model to distinguish nodes whether they are adjacent or distant, so direct relations of neighbor nodes and indirect relations of distant nodes can be encoded with distinction and mixing of semantic relation during generation can be reduced. We employ a multi-layer perceptron to predict the distance between two nodes.ˆ

where Wd ∈ R(D+1)×dmodel , bd ∈ RD+1 , and D is the maximum diameter of the AMR graphs in the dataset. Obviously di,j is the gold answer.

2.5 Objective Function

We aim to optimize the negative log-likelihood of each goldstandard output sentence, S, given the input graph G.

where si is the gold answer for i-th token, θ represents the model parameters, and P(si|s1:i-1,G, θ) is computed in Eq.(15).

To learn better node representations and generate texts of better quality, we also optimize the two proposed graph reconstruction objectives. As described before, link prediction objective has two forms. The first form is to predict the relation given the node pair, which aims to optimize the following negative log-likelihood:

where ri,j is the gold answer for the relation of nodes vi and vj (note that the gold label is set to non-adjacent when the two nodes are not adjacent), λn and 1N are used for balancing the weight of negative samples and P(ri,j|i, j,G, θ) can be computed from the predicted probability ˆ ri,j in Eq.(16).

The second form of link prediction objective is to predict the target node given a source node and a labeled edge.

where P(j|i, ri,j,G, θ) can be computed from the results in Eq.(17).

Distance prediction objective is defined as follows

where P(di,j|G, θ) can be computed from the results in Eq.(18).

Our model is trained by optimizing the weighted sum of the generation objective and graph reconstruction objectives.

where L1/2 l represents one form of the link prediction objective and λl and λd are the hyper-parameters.

3 Experiment

3.1 Data

Two standard English AMR corpora (LDC2015E86 and LDC2017T10) are used as our evaluation datasets. The LDC2015E86 dataset contains 16833 training instances, 1368 development instances, and 1371 test instances. The LDC2017T10 contains 36521 training instances and the same instances for the development and test as LDC2015E86.

3.2 Setup

We set the model parameters based on preliminary experiments on the development set. dmodel is set to 512. The numbers L1, L2 of layers of the encoder and decoder are both set to 6. The head number K is set to 2. The batch size is set to 64. λn is set to 0.1, λl is set to 0.4 and λd is set to 0.1. We share the vocabulary of the encoder and decoder, and use Glove vectors [Pennington et al., 2014] to initialize the word embeddings and demb is set to 300. We apply dropout and use a rate of 0.2. Label smoothing is employed and the rate is set to 0.1. We use the Adam optimizer [Kingma and Ba, 2015] with β1 = 0.9, β2 = 0.98 and ϵ = 10-9 . The same learning rate schedule of Vaswani et al. [2017] is adopted and the maximum learning rate is set to 0.0005. During training, we filter out instances with more than 50 nodes in graph or 50 words in sentence for speeding up. During inference, beam search with size 5 is used.

Following prior works, we use BLEU [Papineni et al., 2002], Meteor [Banerjee and Lavie, 2005], and CHRF++ [Popovi´c, 2017] as automatic metrics for evaluation.

3.3 Comparison Results

We compare our model with several strong baselines, including sequence-to-sequence models and graph-to-sequence models. We use the first form of link prediction objective in this experiment. Models trained with the second form of link prediction objective or other combination of objectives will be detailed in the ablation study. Our base model without graph reconstruction objectives is also compared. Note that some compared baselines are ensemble model but our models are both single model.

Table 1 summarizes the results of these models on the benchmarks. Our model substantially outperforms previous models and achieves the new state-of-the-art performances. Our base model (i.e., without graph reconstruction) also outperforms baselines, which shows the effectiveness of our proposed variant of Transformer. With graph reconstruction objectives, the performance of our model improves by 1.6

BLEU points and 1.2 BLEU points on two datasets, respectively. Among all the baselines, StructuralTransformer-SA achieves the best score on LDC2015E86. Our model improves the BLEU score by 2.4 points, Meteor score by 0.6 points, and CHRF++ score by 1.0 points. On LDC2017T10, our proposed model also outperforms StructuralTransformerCNN by more than 2 BLEU points. Comparing the two sequence-to-sequence neural models, Transformer is much better than the RNN-based model S2S. Similar phenomenon is observed in the graph-to-sequence models. This is the reason we adopt the Transformer encoder-decoder architecture in our model. We can also see that ensemble models achieve better performance than single models with the same architecture, which indicates that ensemble learning is beneficial in this task. However, our single model still strongly outperforms these ensemble models.

3.4 Ablation Study

We further perform an ablation study on the LDC2015E86 development dataset to investigate the influence of the proposed auxiliary objectives. We vary the overall objective function in the following ways: only use the generation objective (Lg); use auxiliary link prediction objective in the first form (Lg + λl ∗ L1l ); use link prediction objective in the second form (Lg + λl ∗ L2l ); use distance prediction objective (Lg + λd ∗ Ld); use link prediction objective in the first form and distance prediction objective (Lg + λl ∗ L1l + λd ∗ Ld); use link prediction objective in the second form and distance prediction objective (Lg + λl ∗ L2l + λd ∗ Ld). Note that we only report the BLEU score in this experiment.

Table 2 presents the results. We can see that using either the link prediction objective or the distance prediction objective could improve the performance. Both two forms of the link prediction objective result in an improvement of about 1 BLEU points. With the distance prediction objective, the result is 0.7 BLEU points higher. Our model achieves the best performance by optimizing the generation objective and two graph reconstruction objectives at the same time. These results verify the usefulness of graph structure reconstruction.

3.5 Semantic Error Analysis

We further analyze the semantic error types in the outputs of different AMR-to-text models. Similar to Konstas et al. [2017], we observe that there are three types of common semantic errors: 1) missing information; 2) generating words inconsistent with the given concept; 3) mixing the given semantic relations. To quantitatively compare different models, suitable metrics are needed. Considering that human evaluation is very time-consuming and requires expertise, we use an SOTA AMR parser [Zhang et al., 2019] to automatically parse the outputs into graphs and compare them with the given input graphs. SMATCH [Cai and Knight, 2013] and a set of fine-grained metrics [Damonte et al., 2016] are used for evaluation. These scores directly measure the degree of semantic overlap between two semantic structures, and indirectly reflect the semantic consistency between the generated sentence and the input graph. We compare the baseline DenselyGCN (DGCN), GraphTransformer (GT), our base model without graph reconstruction, our proposed model and the gold answer.

The results are listed in Table 3. Because the text and the AMR graph are not strictly one-to-one mapping and the parser is not perfect, the scores of gold answer are not 100 points. The SMATCH score reflects the overall semantic overlap and our model outperforms baselines. The Concepts score reflects the translation quality of the concepts and higher Concept score means that less concept information might be missing or mistranslated. Our model performs closely to our base model and outperforms GT and DGCN. The Unlabeled and SRL scores reflect the translation quality of semantic relations, i.e., the edges in the AMR graph. Our model achieves higher scores than baselines on these metrics but is still much lower than the gold answer, which indicates that our proposed reconstruction objectives can reduce the mixing of semantic relations to some extent. Higher Reentrancies score shows that our model can learn better representations for reentrant structure. In general, the above semantic errors occur in outputs of all compared models and our proposed model have less errors in the outputs than baselines.

4 RelatedWorks

Most studies on AMR-to-text generation regard it as a distinct machine translation task. Early works focus on statistical methods. Flanigan et al. [2016] apply tree-to-str transducers to generate texts after transforming AMR graphs to trees. Pourdamghani et al. [2016] use a phrase-based machine translation model on the input of linearized graphs. Song et al. [2017] adopt a synchronous node replacement grammar to generate texts. Moving to neural machine translation methods, Konstas et al. [2017] achieve promising results by named entity anonymization and applying sequence-tosequence model on the linearized graphs. Zhu et al. [2019] adopt the Transformer architecture and introduce Byte Pair Encoding to solve the data sparseness problem.

Recent works treat the task as a graph-to-sequence learning problem and propose various graph neural networks to tackle it. Beck et al. [2018] use Graph Gated Neural Network (GGNN) to directly encode the graph and an attentive decoder to generate texts. Song et al. propose a graph state LSTM as the encoder. Damonte and Cohen [2019] develop a hybrid neural model by stacking a BiLSTM on the output of a Graph Convolution Network (GCN) encoder. Guo et al. [2019] also use GCN and propose densely connected GCN to capture both local and non-local semantic relations in the graph. These methods all focus on aggregating information from one-hop neighborhoods and propagate information from distant nodes by stacking aggregation layers.

The current state-of-the-art approaches propose using relation encoders to model relation of indirectly connected concepts by encoding the shortest path between these nodes. Zhu et al. [2019] propose five different methods to encode the relation path and further extend the conventional self-attention architecture to explicitly utilize the encoded relation between concept pairs. Cai and Lam [2019] use bi-directional GRU encoder to get the representation for the shortest relation path.

5 Conclusion

In this paper, we propose a novel approach that utilize graph structure reconstruction for the AMR-to-text generation problem. Two simple but effective reconstruction objectives, i.e, link prediction objective and distance prediction objective, are proposed for enhancing the capturing of structure information and semantic relation in the node representations. We perform experiments on two English benchmarks and the results show that our model achieves the new state-of-the-art performance. The result of ablation study indicates the effectiveness of our base model and the graph reconstruction objectives. In addition, we analyze the semantic errors in the outputs by using automatic metrics.

In future work, we will apply our model to other graphto-sequence problems. We will also incorporate more welldesigned and effective graph reconstruction objectives for better node representation learning.

Acknowledgments

This work was supported by National Natural Science Foundation of China (61772036) and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). Xiaojun Wan is the corresponding author.