Effective Graph Context Representation for Document-level Machine Translation

1 Introduction

Recently, document-level neural machine translation (DocNMT) has attracted much attention in the machine translation community [Zhang et al., 2018; Yang et al., 2019; Ma et al., 2020]. Typically, DocNMT considers the documentlevel contextual information, for example, several previous sentences or all sentences in the entire document. All words within the document-level context are then encoded as an additional source representation one-by-one for enhancing the translation of the input document. As a result, the DocNMT models greatly alleviated translation errors in the documentlevel translation scenario [Tan et al., 2019; Voita et al., 2019; Kang et al., 2020; Bao et al., 2021].

Different from a single sentence, multiple sentences within one input document are often associated with effective items like content words, logical order, co-occurrence, and cohesion/coherence relations, to introduce this document-level context. As illustrated by the English input document in Figure 1, there are obvious multiple content words (e.g., "owns"

∗ Corresponding author

and "FBI agents") in green, including co-occurrence words (e.g., "Austin" and "Florid") in blue, related to the documentlevel context. Furthermore, the three sentences describe people and things related to "a scuba diving shop", and introduced the matter progressively and coherently, for example, "Austin owns a scuba diving shop, FBI agent will visit his shop, but he did not receive any notification". Meanwhile, there are also many functional or auxiliary contexts (e.g., not highlighted words) to assist in the expression of this matter. Most existing DocNMT methods universally encode all words within the input document regardless of whether these words are related to this effective document-level context. Thus, those functional and auxiliary context words disperse the attention of DocNMT for this effective document-level context. Particularly, the advanced DocNMT systems [Tu et al., 2018; Maruf et al., 2019] often use several previous sentences instead of the entire document, which further hinders the representation of the global document-level context. Therefore, learning a representation for the effective documentlevel context is more crucial for the advanced DocNMT.

In this paper, we propose an effective graph context representation (EGCR) method to learn the representation of the contextual information from the input document. To this end, we first recognize content words from the input document by term frequency-inverse document frequency and design four heuristic relations to connect the recognized content words as a graph structure without relying on external syntactic knowledge. Graph attention networks (GATs) [Guo et al., 2019] are then used to learn the feature representation of this graph structure. As a result, the proposed EGCR allows DocNMT to effectively make use of the documentlevel context to enhance the translation of the input document. Experimental results on several widely-used document-level translation benchmarks demonstrated that our DocNMT model with EGCR gained significant improvement over strong baselines. Further quantitative and qualitative analysis verified the effectiveness of the proposed EGCR.

2 Methodology

2.1 Heuristic Graph Structure

To effectively represent the input document, we first recognize content words from an input document Mm by term frequencyinverse document frequency (TF-IDF). Formally, the TF-IDF

score of each word w in Mm is:

where Jw represents the number of occurrences of word w in Mm and J is the total number of word w in all source documents M; |M| is the total number of source documents; and Mw is the number of document including word w. 1 Note that multiple consecutive content words are as a phrase.

Then, we propose four heuristic relation edges to connect the recognized content words as a graph structure: Intersentence edge (ISEdg) denotes the relation edges between all sentences in the input document; Sentence-word edge (SWEdg) denotes the relationship between one sentence and its content words; Intra-sentence edge (IASEdg) used to retain order between content words within the same sentence; Co-occurrence edge (CoEdg) denotes the co-occurrence of the same word in different sentences from the input document. Formally, these recognized content words from the input document are denoted as a set of nodes V={v1, v2, · · · , vU}, where U is the number of nodes and the representation of each node vu is the average of the word embeddings in the entity or verb. Additionally, an adjacency matrix E∈RU×U denotes connections between two nodes using the aforementioned four kinds of relation edges:

1 This paper empirically select 25% words with higher TF-IDF scores in the input document as content words.

Finally, the constructed graph, which is undirected, is formally denoted as G={V, E} as shown in Figure 1.

2.2 Representation of Effective Context

Formally, GATs are applied to the constructed graph G={V, E} to learn the feature representation of each node using the multi-hop mechanism [Xu et al., 2021b]. Formally, given the outputs of all previous hop iterations {s1m, s2m, · · · , sn-1 m }, we concatenate them, then transform them into a fixed dimensional vector as the input of the n-th hop operation to learn an intermediate feature representation znm:

where sn-1 m ∈Rdmodel and Wnd ∈Rdmodel×(n×dmodel) , and dmodel is the word embedding dimension. According to edge matrix E[n][bi] (0≤bi<I), all I direct adjacent nodes of vn are {znb1 , · · · , znbI } when E[n][bi] is equal to one according to Eq.(2). Particularly, Wnd aims to summarize direct connections introduced from all its preceding layers. We use the selfattention module [Vaswani et al., 2017] to learn the n-th feature representation snm using znm and {znb1 , · · · , znbI }:

where the direct adjacent feature representations of nodes {znb1 , znb2 , · · · , znbI } are packed into key matrix Kd and value matrix Vd. After we perform N hop iterations, there is a sequence of annotations {s1m, s2m, · · · , sNm} to encode the feature information of the m-th node. Finally, we apply another nonlinear layer to integrate the feature information {s1m, s2m, · · · , sNm} and the node vector vm in V:

As a result, the proposed EGCR is represented as G={g1 , g2 , · · · , gU } (see Figure 1).

3 EGCR for DocNMT

3.1 Background: Transformer-based NMT

In Transformer-based NMT [Vaswani et al., 2017], given an input sentence of length J, X={x1, x2, · · · , xJ}, The positional embedding pej of each word is added to the corresponding word embedding xj as a combined embedding hj. Thus, there is an input representation H0e ={h1, h2, · · · , hJ} which will be fed into the stacked encoder to learn the source representation HLe :

where ATTl (·), LN(·), and FNNl (·) are the self-attention module, layer normalization, and feed-forward neural network for the l-th identical layer, respectively. [· · · ]L denotes the stack of L identical layers. Compared with the stacked encoder, there is an additional encoder-decoder attention sublayer that performs the dependent-time attention module for the top output of the stacked encoder HLe .

where Ql-1 i ,Kl-1 i , and Vl-1 i are the query, key, and value matrices that are transformed from the (l-1)-th layer Sl-1 in the current time step i and {KLe , VLe } are transformed from the top output HLe of the stacked encoder in Eq.(6). Tli represents the generated target fragment and is therefore used to compute the dependent-time context vector Bli .

Finally, the top output of the stacked decoder SLi is used to generate the next target word yi using a linear, potentially multi-layered function:

where Wto and Wtw are projection matrices. To obtain the NMT model θ, the training objection maximizes the conditional translation probability over the training dataset {[X, Y]}:

3.2 DocNMT with EGCR

This section will introduce the proposed DocNMT with EGCR, as shown in Figure 2. Formally, we feed the learned EGCR A into Eq.(7) together with the original representation of source sentence HLe to learn a dependent-time context vector:

where KA=VA is the matrix representation of the proposed EGCR G={g1 , g2 , · · · , gU }. Note that both the original sentence-level context vector Bli and the proposed documentlevel context vector Dli are learned by a shared crossing attention module ATTlc .

We use the top output of decoder S Li , which is enhanced by the extracted effective document-level context, as input to Eq.(8) to compute the conditional probability of next target word yi :

Finally, the training objective maximizes the probability over the training dataset {[X, Y]}:

where θ is a parameter set related to the processing of the SentNMT model and ϕ is another parameter set related to the learning of EGCR.

4 Experiments

4.1 Datasets and Setup

We evaluated our approach on several benchmarks for document-level machine translation. For TED Talks in IWSLT17, we used dev-2010 as the development set, and test-2010/2011/2012/2013 as the test sets for both ChineseEnglish (Zh-En) and English-German (En-De) language pairs. For News-Commentary v14 (News), we use the newstest2017 for development and newstest2018 to test both Zh-En and EnDe language pairs. Our approach was also evaluated on a large scale corpus Euro extracted from Europarl v7 [Maruf et al., 2019]. Table 1 showed the corpora statistics. We adopted the

BPE algorithm [Sennrich et al., 2016], and set the vocabulary size to 32K. We set the dimension of all input and output layers to 512, the dimension of the inner feedforward neural network layer to 1024, and the total heads of all multi-head modules to 8 in both the encoder and decoder layers. The number of multi-hop reasoning N was set to 2 empirically. Each training batch consisted of a set of sentence pairs that contained approximately 4000×8 source tokens and 4000×8 target tokens. The value of label smoothing was set to 0.1, and the attention dropout and residual dropout were 0.1. We varied the learning rate under a warm-up strategy with warmup steps of 8,000. Following the training of 100,000 batches, we used a single model obtained by averaging the last five checkpoints, which validated the model with an interval of 2,000 batches on the dev set. We trained all models on eight V100 GPUs and evaluated them on a single V100 GPU. The multi-bleu.perl script was used as the evaluation metric.

We re-implemented several advanced document-level NMT systems based on the sentence-level Transformer NMT model (SentNMT) [Vaswani et al., 2017] by using the fairseq toolkit: CTX [Zhang et al., 2018] uses an additional encoder to learn context representations, which are then integrated by cross-attention mechanisms. HAN [Miculicich et al., 2018] uses a hierarchical attention mechanism with two levels

(word and sentence) of abstraction to incorporate context information from both the source and target documents. Selective [Maruf et al., 2019] uses all contexts in the entire document by calculating the sentence-level and word-level weights. HierAtt [Tan et al., 2019] learns a global context representation using a hierarchical attention mechanism. DynS [Tu et al., 2018] introduces a cache to memorize previous hidden states as dynamically updated context during decoding. Additionally, we reported the results of two recent DocNMT models: Unified [Ma et al., 2020] employed the first encoder layer of Transformer to encode the current sentence with context information and fed it into the decoder; DGraph [Xu et al., 2021a] applied syntactic parser to both source and target documents, and incorporated their graphs into the Transformer with graph convolutional networks.

4.2 Main Results

Table 2 shows the overall results on all translation tasks, where the hyperparameter N in Eq.(5) is set to 2 for the Zh-En and En-De tasks (the relation between N and BLEU scores is as shown in Appendix ??). The results demonstrated that all the document-level NMT models performed better than the SentNMT model, which indicates that documentlevel contextual information is beneficial for NMT. +EGCR achieved the best performance among all document-level NMT models on five test sets. This means that the proposed EGCR can leverage the document-level context to improve translation performance. Particularly, +EGCR outperformed the comparison DocGraph by 0.49 BLEU scores on the same TED Zh-En test set. We think that the proposed EGCR summarized more effective context from the input document while DocGraph constructed all words within one document as a graph that contains other redundant contextual information (i.e., functional and auxiliary words). Additionally, +EGCR had higher efficiency than comparison methods in terms of model parameters and training speeds.

4.3 Ablation of EGCR

As mentioned in Section 1, those functional and auxiliary context words within the input document disperse the attention of DocNMT for this document-level context. To evaluate the relevance of our constructed document-level context, Table 3 shows results of our SentNMT+EGCR model when the different percentage of remaining words from the input document is introduced into EGCR (the +EGCR in Table 2 contained about 25% words in the input document). BLEU scores began to decrease as the increase of the remaining words (e.g., functional and auxiliary context words) but the EGCR with the least amount of content and co-occurrence words gained the highest BLEU scores. This indicates that these functional and auxiliary context words hindered the learning of the document-level context.

4.4 Evaluating GATs for Learning EGCR

To evaluate the effectiveness of GATs, we replaced GATs with recurrent neural networks (RNNs), convolutional neural networks (CNNs), and self-attention networks (SANs). Specifically, we flat the constructed graph in Section 2.1 as a long sequence according to word order in the original document and then feed this long sentence into RNNs, CNNs, and SANs to learn an approximation EGCR. Table 4 showed ablation results on the five translation tasks. First, four SentNMT+EGCR models outperformed the baseline SentNMT model in terms of BLEU scores, confirming the effectiveness of the proposed approach. The EGCR learned by GATs gave more improvement than ones learned by three alternative RNNs, CNNs, and SANs. We believe that GATs can better encode structural and hierarchical translation knowledge in the input document.

4.5 Ablation Study for Different Edges

To evaluate different relation edges of the constructed graph, we report ablation test results in Table 5. Among all relationship edges, removing CoEdg yielded the greatest performance degradation on all test sets. This means that the co-occurrence edge contributed the most to the improvement, which is consistent with the effectiveness of coreference information in the advanced DocNMT systems [Maruf and Haffari, 2018; Maruf et al., 2019; Ma et al., 2020]. Similarly, removing IASEdg was in second place for the improvement, which indicates that the sequential edge is an important contribution. The performance drop for our model without SWEdg and ISEdg demonstrated that the sentence-entity edges and inter-sentence edges were also useful for DocNMT. In short, the four types of relation edges effectively simulated order dependency in the input document.

4.6 Ablation Study of Different Nodes

To evaluate the effect of sentence, content, and cooccurrence nodes in the EGCR, we report ablation results in Table 6. Removing any of three types of nodes resulted in the performance degradation but outperformed the baseline SentNMT, which indicates that they were useful for representing the document-level context. Moreover, EGCR without sentence or co-occurrence nodes had a lesser degradation in terms of BLEU scores than EGCR without content nodes. By comparison, content nodes can encode more document-level contextual information than sentence and co-occurrence nodes in EGCR.

4.7 Ablation Study of Sentence Order

Multiple sentences within one document often introduce this matter of the input document gradually according to some logical order. The proposed graph structure encoded the document-level contextual information in a hierarchical manner, which may have a strong capability of capturing logical order. Therefore, we simulated a scenario where the order of some sentences was randomly swapped in one input document. For example, "2" indicates that there were two randomly swapped sentences for each document in the test set. Figure 3 showed the results of TED Zh-En and En-De test sets. As the number of sentences whose word order is changed increases, the performance of DocNMT will decrease significantly. Particularly, when Num is greater than two (ZhEn) or three (En-De), BLEU scores of +DynS were even lower than the sentence-level SentNMT model. This means that the logical order plays a very important role for DocNMT, but the proposed approach does have the strong ability to capture logical order between multiple sentences in one document.

4.8 Analysis of Discourse Phenomena

In this section, we investigate the contribution of documentlevel context information in improving the translation of discourse phenomena. The experiment is designed according to [Voita et al., 2019] to evaluate four types of discourse phenomena (i.e., deixis, lexical cohesion, inflection, and VP ellipses). Each test instance consisted of positive and several negative translations with incorrect phenomena. We evaluated the models using accuracy, which is defined as the proportion of times the generation probability of a positive translation was higher than that of a negative translation. We trained the SentNMT, +DynS, and +EGCR models on applying the OpenSubtitles2018 corpus for English and Russian following the settings of [Voita et al., 2019]. The accuracy of the discourse phenomena is shown in Table 7. Both +DynS and our +EGCR comprehensively improved consistency over the document-level context-agnostic SentNMT. Moreover, our +EGCR outperformed +DynS over four types of discourse phenomena. We attribute this to the fact that our document graph contained structural information within the documentlevel context (i.e., lexical consistency, logical order, and co-reference), thereby directly linking relevant contexts for repeated and deistic words.

4.9 Accuracy of Pronoun/Noun Translations

We followed the [Miculicich Werlen and Popescu-Belis, 2017]'s setting to evaluate coreference and anaphora using the reference-based metric, that is, the accuracy of pronoun translation, which can be extended to nouns. The list of evaluated pronouns is predefined in the metric, while the list of nouns is extracted using POS tagging, as shown in the results of two Zh-En test sets in Table 8. The results demonstrate that both +DynS and +EGCR achieved accuracy with a significant improvement compared with the SentNMT model, which indicates that the document-level context was beneficial to the translation of discourse nouns and pronouns. Furthermore, the proposed +EGCR gained higher accuracy than the baseline +DynS. This indicates that EGCR more effectively leveraged the document-level contextual information than DynS to improve the translation of discourse nouns and pronouns.

4.10 Cohesion and Coherence Evaluation

Following Wong and Kit's setting, we evaluated the lexical cohesion, which is defined as the ratio of the number of repeated and lexical similar content words to the total number of content words in a target document. Meanwhile, for coherence, we used a metric based on latent semantic analysis [Foltz et al., 1998], which is used to obtain sentence representations, then calculated the cosine similarity from one sentence to the next, and averaged the results to obtain a document score. Table 9 shows the average ratio per tested

document for lexical cohesion and the average coherence score of documents. +DynS and +EGCR consistently obtained better scores for two Zh-En tasks, which is in line with the improvements to lexical cohesion and coherence in [Xiong et al., 2013b; Lin et al., 2015]. In particular, the scores of our +EGCR model were higher than those of +DynS, which indicates that the proposed EGCR captured more document-level contextual information related to discourse lexical cohesion and coherence.

5 RelatedWork

Typically, document-level context information was used to improve the lexical cohesion of the input document [Xiong et al., 2013a; Ben et al., 2013; Xiong et al., 2013b], where the sentences are connected via syntactic and lexical devices. Document-level context information has also been used to capture coherence for document translation [Xiong and Zhang, 2013; Lin et al., 2015], sentences of which are tied into a meaningfully connected structure.

Recently, document-level contextual information has begun to be introduced into NMT to enhance translation performance. Part of the document-level context has been encoded to assist the translation of the current sentence, for example, a single previous sentence [Tiedemann and Scherrer, 2017; Voita et al., 2018]; one previous sentence both in the source and target [Bawden et al., 2018]; more than one previous source sentence [Zhang et al., 2018; Voita et al., 2019; Kang et al., 2020]; or a few previous source and target sentences [Miculicich et al., 2018], and the contextual information in a fixed scope [Kuang et al., 2018; Wong et al., 2020]. By contrast, in some studies, the full document context has been used effectively for the source or target-side document-level contextual information in NMT [Maruf and Haffari, 2018; Zhang et al., 2018; Maruf et al., 2019; Tan et al., 2019; Ma et al., 2020].

More recently, [Xu et al., 2021a] proposed a graph-based DocNMT method to simulate both source and target documentlevel context according to inter-sentential and intra-sentential relations. Although the graph-based DocNMT method is closely related to our work, there are two main differences. To begin with, we represent the document-level contextual information as the graph structure constructed by heuristic rules without demanding external syntactic knowledge. This breaks through the constraints of prior syntactic knowledge and provides a general alternative method for researchers to explore the document-level context for DocNMT. Moreover, our graph structure can encode the relevant contextual information from the input document, which allows DocNMT to more effectively make use of the document-level context than the existing universal encoding method.

6 Conclusion

This paper heuristically constructed the graph structure of the input document without external syntactic knowledge to summarize the document-level contextual information. We then applied GATs to the constructed graph to learn its feature representation, and thereby effectively leveraged the document-level context to improve translation performance. The experiment results on several widely-used documentlevel translation benchmarks demonstrated the effectiveness of the proposed approach. Furthermore, the exhaustive quantitative and qualitative analysis demonstrated that the proposed approach helped DocNMT capture the relevant document-level translation knowledge, for example, logical order, discourse phenomena, and coherence information. In the future, we will explore another efficient document-level translation knowledge for advancing the DocNMT.

Acknowledgments

We are grateful to the anonymous reviewers, area chair, senior area chair, and program committee for their insightful comments and suggestions. Min Zhang was partially supported by the National Natural Science Foundation of China (No. 62036004). Rui Wang is with MT-Lab, Department of Computer Science and Engineering, School of Electronic Information and Electrical Engineering, and also with the MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200204, China.