A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model

1 Introduction

Grammatical Error Correction (GEC) is a monolingual textto-text rewriting task where given a sentence containing grammatical errors, one needs to modify it to the corresponding error-free sentence. In recent years, pre-training on synthetic erroneous data and then fine-tuning on annotated sentence pairs has become a prevalent paradigm [Grundkiewicz and Junczys-Dowmunt, 2019; Lichtarge et al., 2019; Zhang et al., 2019; Zhou et al., 2021] in English GEC, advancing the state-of-the-art results [Ge et al., 2018b; Sun et al., 2021; Rothe et al., 2021] with various novel data synthesis approaches [Ge et al., 2018a; Grundkiewicz and JunczysDowmunt, 2019; Kiyono et al., 2019].

∗This work was done during the author's internship at MSR Asia. Contact: Tao Ge (tage@microsoft.com)

As GEC in other languages has drawn increasing attention [Flachs et al., 2021; Rothe et al., 2021], synthetic erroneous data construction has been borrowed to non-English languages for improving the results given a lack of annotated data. For instance, the rule-based approaches obtain promising results [Grundkiewicz and Junczys-Dowmunt, 2019; N´aplava and Straka, 2019; Wang et al., 2020a]. However, these approaches require language-specific rules and confusion sets for word replacement based on expertise to simulate diverse linguistic phenomena across multiple languages, e.g., homophones in Chinese characters and morphological variants in Russian. Moreover, rule-based approaches always produce erroneous data with limited error-corrected patterns [Zhou et al., 2020].

To address the above limitations, we propose a generic strategy for training GEC systems in non-English languages. Our approach is easily adapted to new languages if only provided with two relatively easy-to-obtain resources: 1) a pretrained cross-lingual language model (PXLM); 2) the parallel translation data between English and the language. In this paper, we choose InfoXLM [Chi et al., 2020] as the PXLM in our implementation.

Our approach consists of synthetic data construction and model initialization. Since InfoXLM was pre-trained with translation language modeling objective, which requires the model to recover the masked tokens conditioned on the concatenation of a translation pair, it already possesses the capability of Non-Autoregressive Translation (NAT). That is, when presented with an English sentence, InfoXLM can provide a rough translation in dozens of non-English languages.

Compared with AT, NAT sacrifices translation quality due to the multimodality problem [Gu et al., 2017; Ghazvininejad et al., 2019]. When vanilla NAT performs independent predictions at every position, it tends to consider many possible translations of the sentence at the same time and output inconsistent results, such as token repetitions, missing or mismatch [Ran et al., 2020; Du et al., 2021]. Compared with predefined rules, such error-corrected patterns are more reasonable and diverse with large model capacity and dependency in sentence context. We regard the rough translation generated by InfoXLM as a source sentence and the gold translation as a corrected sentence for pre-training. To further improve the generalization ability of the seq2seq GEC model, we initialize the GEC model with InfoXLM and pre-train it with the synthetic data generated by itself.

We conduct experiments on Chinese, German and Russian GEC benchmarks. Our approach achieves the state-of-theart results on the NLPCC 2018 Task 2 dataset (Chinese) and obtains competitive performance on Falko-Merlin (German) and RULEC-GEC (Russian). The results also demonstrate that our approach can effectively complement rule-based corruption methods.

The contributions of this paper are as follows:

• We propose a unified strategy for GEC in the nonEnglish languages consisting of synthetic data construction and model initialization.
• We propose a novel NAT-based synthetic data construction approach, which generates diverse error-corrected data for pre-training. To the best of our knowledge, it is the first to utilize the non-autoregressive translation ability of a PXLM for GEC erroneous data construction. The generated sentence pairs perform promising results alone and also nicely complement rule-based corruption methods.
• Our approach achieves the state-of-the-art performance on the Chinese benchmark and very competitive results for German and Russian benchmarks as well.

2 Methodology

In this section, we present the unified strategy for non-English languages. At first, we briefly describe Translation Language Modeling (TLM) objective and Non-Autoregressive Translation (NAT) ability of InfoXLM. Then, we introduce two steps in our framework: NAT-based synthetic data construction and model initialization. Figure 2 shows the overview of our data construction approach.

2.1 Background: Translation Language Modeling

The basis of our data construction is the non-autoregressive translation ability of InfoXLM, owing to its Translation Language Modeling (TLM) objective during pre-training. Given a sentence x = x1 · · · x|x| in the source language (e.g., English) and the corresponding translation y = y1 · · · y|y| in another language (e.g., Chinese), the input sequence of TLM is the concatenation of these two parallel sentences S = ⟨s⟩ x ⟨/s⟩ y ⟨/s⟩ and some percentage of tokens are replaced with [MASK] at random. Formally, let M = {m1, · · · ,m|M|} denote the positions of the masks:

where the replace denotes the replacement operation at the certain positions. By leveraging bilingual context, the model is required to predict the original tokens with cross entropy loss. The TLM loss is computed as:

where S\M = {Si}i̸∈M means tokens that are not included in the M and T is the translation corpus.

To the extreme, we can use InfoXLM as a nonautoregressive translator. Specifically, we concatenate an English sentence x with enough placeholders ([MASK]) as the input. InfoXLM is capable of translating it to other languages by predicting tokens at all masked positions in parallel. Formally, M = {|x| + 3, · · · , |x| + |y| + 2} denotes all target tokens are replaced with [MASK] and the predicted translation y∗ is derived by maximizing the following equation:

which infills the words with the highest probability.

In practice, following Mask-predict [Ghazvininejad et al., 2019], we partially mask some percentage of target translation (m ∈ [|x| +3, |x| + |y| +2]) rather than all of them, which ensures the outputs are of appropriate quality.

2.2 NAT-based Data Construction

To generate diverse error-corrected sentences for GEC in a non-English language (e.g., Chinese), our approach utilizes sentence pairs of machine translation (e.g., English-Chinese parallel corpus). Our approach starts by adding noise to the target sentence and then masking sampled tokens. We feed the corrupted target sentence with the original English sentence as the input of InfoXLM. InfoXLM performs TLM predictions at every masked position. To obtain poor sentences containing grammatical errors, we randomly sample the word from the top predictions.

Specifically, given a sentence y in the target language, we select tokens for modification with a certain probability pnoise and perform the following operations:

Mask. Replace the token with [MASK] with a probability of pmask.

Insert. Add a [MASK] after the token with a probability of pinsert.

Delete. Delete the token with a probability of pdelete. Swap. Replace the token with its right token with a probability of pswap.

Synthetic error-corrected corpus (Zh):

We get the noisy text ˜ y = NOISE(y) and the corresponding positions of the masks M. Then, we concatenate the English sentence x with the corrupted sequence ˜ y containing enough masked tokens as the input of InfoXLM. The predicted words are sampled for every [MASK] according to the output distribution:

where we produce erroneous sentence by replacing [MASK] with sampled tokens.

Our artificial corruption by four operations before TLM prediction improves the difficulty of translation. The independent assumption between target tokens brings in more errors and less fluency. The predicted words are sampled based on distribution rather than the best predictions to create more inconsistencies. It resembles the scenario where elementary language learners render a low-quality sentence when completing the cloze task. However, we only mask some percentage of target tokens and the English sentence restricts InfoXLM to recover original information, which ensures that the sampled tokens are plausible.

Since the recovered sentence contains diverse and reasonable word-level grammatical errors, we apply character-level corruption operations to add more spelling errors: 1) insert; 2) substitute; 3) delete; 4) swap-right; 5) change the casing. We call it post edit. Finally, we regard the gold translation as the corrected sentence and the corrupted prediction as the erroneous sentence.

2.3 Model Initialization

To further improve the generalization ability of the GEC model, we use InfoXLM to initialize the seq2seq model. We follow [Ma et al., 2021] and use DeltaLM for multilingual GEC. DeltaLM is an InfoXLM-initialized encoder-decoder model trained in a self-supervised way. We continue pretraining it with synthetic data generated by our NAT-based approach.

Overall, our unified strategy exploits InfoXLM in two ways. We make use of its NAT ability to produce synthetic GEC data and its pre-trained weights to initialize our GEC model.

3 Experiments

3.1 Data

We conduct our experiments on three GEC datasets: NLPCC 2018 Task 2 [Zhao et al., 2018] in Chinese, FalkoMerlin [Boyd, 2018] in German and RULEC-GEC [Rozovskaya and Roth, 2019] in Russian. The statistics of the datasets are listed in Table 1. We use the official MaxMatch [Dahlmeier and Ng, 2012] scripts 1 to evaluate precision, recall and F0.5.

For non-autoregressive translation generation, we use datasets of the WMT20 news translation task 2 - UN Parallel Corpus v1.0 for Chinese and Russian, the combination of Europarl v10, ParaCrawl v5.1 and Common Crawl corpus for German. We construct 10M synthetic sentence pairs in every language for pre-training and then fine-tune the GEC model on respective annotated datasets.

1 https://github.com/nusnlp/m2scorer

2 https://www.statmt.org/wmt20/translation-task.html

3.2 Implementation Details

Unless explicitly stated, we use Transformer (base) model in fairseq 3 as our GEC model. For Chinese, we construct a character-level vocabulary consisting of 7K tokens. We apply Byte Pair Encoding to preprocess German and Russian sentences and obtain the vocabularies with size of 32K tokens, respectively. When using DeltaLM, we utilize its shared vocabulary of 250000 tokens. During pre-training for German and Russian, following [N´aplava and Straka, 2019], we use source and target word dropouts and edit-weighted MLE [Junczys-Dowmunt et al., 2018]. We leave the detailed hyperparameters in the supplementary notes.

3.3 Baselines

Most of the previous studies for Chinese GEC focus on model architecture or training strategy, which are orthogonal with our synthetic data construction method. For example, YouDao [Fu et al., 2018] combines five hybrid correction models and a language model together. AliGM [Zhou et al., 2018] combines NMT-based, SMT-based and rulebased models together. BLCU [Ren et al., 2018] uses multilayer convolutional seq2seq model. BERT-encoder [Wang et al., 2020b] initializes the encoder of seq2seq model with BERT . BERT-fuse [Wang et al., 2020b] incorporates BERT for additional features. As for training strategy, DropoutSrc [Junczys-Dowmunt et al., 2018] sets the full embeddings of randomly selected source words to 0 during the training process. MaskGEC [Zhao and Wang, 2020] performs dynamic masking method by substituting the source word with a padding symbol or other word.

The most comparable approach is [Wang et al., 2020a], which constructs pre-training data using the rule-based corruption method. For our approach, we implement MaskGEC during the fine-tuning stage. To make a fair comparison, we also construct synthetic data with rule-based corruption in the same setting as baseline. It incorporates a character-level confusion set 4 and uses pypinyin 5 to perform homophony replacement.

3 https://github.com/pytorch/fairseq

For German and Russian, the main data construction method is rule-based corruption. [Grundkiewicz and Junczys-Dowmunt, 2019] and [N´aplava and Straka, 2019] build confusion sets with edit distance, word embedding or spell-checker (e.g., Aspell dictionary 6 ). [Flachs et al., 2021] uitlizes Unimorph which provides morphological variants of words for word replacement operations. They also incorporate WikiEdits and Lang8 as additional training resources. [Rothe et al., 2021] only applies language-agnostic operations without any confusion set. They pre-train a unified seq2seq model for 101 languages and fine-tune for respective languages. [Katsumata and Komachi, 2020] proposes to directly use mBART without pre-training.

3.4 Main Results

Table 2 shows the performance of our approach and previous methods on the NLPCC-2018 Chinese benchmark. Our NAT-based synthetic data construction approach is comparable with the rule-based corruption approach. We assume that 0.15 F0.5 descend comes from that rule-based corruption leverages many useful confusion sets. When combined with the confusion sets, our approach obtains 39.42 F0.5 which outperforms the rule-based counterpart. If combining two data sources from the rule-based and NAT-based approaches, we obtain better performance which demonstrates two methods complement each other. Initializing the GEC model with DeltaLM achieves 40.70 F0.5, which is the state-of-the-art result of the dataset.

Table 11 shows the performance for German and Russian datasets. In the same setting, our NAT-based synthetic approach outperforms rule-based corruption methods and most baselines with two exceptions. For instance, [N´aplava and Straka, 2019] leverages more training strategies during finetuning phase such as mixing pre-training data and oversampled fine-tuning data, checkpoint averaging and so on. Although [Rothe et al., 2021] obtains the best performance with the pre-trained T5-XXL (11B parameters), its base-size model lags behind ours significantly under the similar model capacity. Overall, the performance on the German and Russian datasets demonstrates the effectiveness of our unified strategy and NAT-based synthetic approach, which performs competitive results alone and also nicely complements rulebased corruption methods.

To make fair comparison with multiple synthetic construction approaches, we follow the experimental setting and model hyperparameters 7 in [Flachs et al., 2021]. The results on the German dataset are shown in Table 4. Our approach significantly outperforms commonly used synthetic methods such as the rule-based approach with Unimorph, Aspell word replacement and Wikipedia edits extracted from the revision history. Although back-translate is effective for English, it performs poorly with limited annotated sentence pairs to learn diverse error-corrected patterns. Round-trip translation utilizes the same translation corpus as us but achieves inferior performance since it usually produces sentences without grammatical errors.

4 http://nlp.ee.ncu.edu.tw/resource/csc.html

5 https://github.com/mozillazg/python-pinyin

6 http://aspell.net/

7 We use the "transformer clean big tpu" setting.

3.5 Ablation Study

We further conduct an ablation study as shown in Table 5. Overall, we find that all of these variants perform worse than the original strategy. From the last row, PXLM is much better than a regular translation model under the same setup (i.e., training data and sample strategy). Our approach can control the degree of overlap and errors, while the sentences generated by AT have few grammatical errors or little overlap with the original sentences. The removal of NAT-based replacement and bilingual constraint also results in a significant degradation, which indicates substitution with similar semantic meanings plays a crucial role in our strategy.

3.6 Error-type Analysis

We analyze the GEC performance of data construction approaches on different error types. We use the German extension [Boyd, 2018] of the automatic error annotation tool ERRANT [Bryant et al., 2017] for evaluation. Table 6 shows the F0.5 score of top 9 error types on the German dataset. We can observe that our approach improves the model in all error types significantly compared with that trained from the scratch and outperforms that with rule-based corruption in 7 out of 9 error types. For example, the largest improvement comes from the 'Other' type by 6.5 F0.5 score, which is defined as the errors that do not fall into any other specific type, such as paraphrasing (e.g., feel happy → be delighted) [Bryant et al., 2017]. Such error type is beyond pre-defined rules and hard to simulate [Zhou et al., 2020].

Two exceptions are 'Orthography' and 'Pronoun'. 'Orthography' refers to the error related with case or whitespace errors (e.g., Nexttime → next time), which the specific rules are able to simulate very well. 'Pronoun' denotes the substitutes for the noun (e.g. you → yourself ) which also fall into the strength of the rule-based approach with languagespecific confusion set. We also observe the combination of these two sources of synthetic data yields better results, which demonstrates that they are helpful to complement each other to enrich error-corrected patterns.

To verify our explanation, we present the top 5 error types distribution of rule-based corruption and our approach in Figure 3. Our approach yields more 'Other' type errors compared with 'Spell' errors, which may account for the improvement in that category. The large ratio of 'Orthography' and 'Pronoun' errors generated by rule-based corruption is consistent with its better performance on these two types.

3.7 Case Study

To give a qualitative analysis of generated erroneous sentences, we present an example of our approach and existing synthetic methods in Table 7. We can see that backtranslation tends to generate similar modifications such as token deletion and simple paraphrasing. The rule-based corruption approach is hard to simulate human writing since it directly swaps adjacent tokens and performs word replacement without consideration of the sentence context. Roundtrip translation generates an error-free sentence. In contrast, our approach generates the less fluent sentence by paraphrasing the corrupted contents and maintaining the meaning of the corresponding English sentence.

4 RelatedWork

Pre-training a seq2seq model on synthetic data and then finetuning on annotated error-corrected sentence pairs is the common practice for GEC. Available datasets in non-English languages such as German [Boyd, 2018], Russian [Rozovskaya and Roth, 2019] and Czech [N´aplava and Straka, 2019] only contain a lack of annotated data, which requires high-quality synthetic data construction.

Back-translation is the reverse of GEC, which takes corrected sentences as input and error sentences as output. It is popular and effective for English GEC [Kiyono et al., 2019; Zhang et al., 2019] but difficult to adapt to these low-resource scenarios, since it is hard to learn diverse error-corrected patterns with less annotated sentence pairs. Round-trip translation (e.g., translating German to English then back to German) [Lichtarge et al., 2019] has been blamed for errors that it introduced are relatively clean.

The most effective construction method for non-English languages is rule-based corruption [N´aplava and Straka, 2019; Grundkiewicz and Junczys-Dowmunt, 2019]. Most of them rely on word substitutions with ASpell or languagespecific confusion sets. It requires well-designed rules to simulate diverse linguistic phenomena across different languages. [Rothe et al., 2021] only performs language-agnostic operations without any confusion set to construct corrupted sentences but achieves inferior performance with moderate model size. Wikipedia edits extracted from the revision history of each page are also useful GEC pre-training resources [Boyd, 2018; Flachs et al., 2021]. Most studies for Chinese GEC focus on model architecture [Fu et al., 2018; Zhou et al., 2018; Ren et al., 2018] and training strategy [Zhao and Wang, 2020], which are orthogonal with our approach.

The most similar approach to ours is [Zhou et al., 2020] which trains two autoregressive translation models with poor and good qualities, respectively. With the same sentence in the source language, they regard two translations of two models as error-corrected sentence pairs. In comparison, our approach directly utilizes the non-autoregressive translation ability of the PXLMwithout training translators additionally, which is easier to adapt to new languages. Utilizing a pretrained language model to propose candidate words for replacement and insertion has also been applied to lexical substitution [Zhou et al., 2019], text generation [Li et al., 2020], etc. By contrast, we adopt bilingual constraints to avoid generating candidate words that are inconsistent with the original meaning.

Leveraging pre-trained language models in GEC seq2seq models has been extensively explored [Katsumata and Komachi, 2020; Wang et al., 2020b; Kaneko et al., 2020]. We initialize the model with DeltaLM [Ma et al., 2021], which adjusts the InfoXLM-initialized encoder-decoder model to generation mode by self-supervised pre-training.

5 Conclusion and FutureWork

We propose a unified and generic strategy for training GEC systems in non-English languages given a PXLM and the parallel translation data. Our approach obtains state-of-theart results on the NLPCC 2018 Task 2 dataset (Chinese) and competitive results on German and Russian benchmarks. The synthetic sentence pairs also complement rule-based corruption to yield further improvement. Compared with a regular translator, NAT by the PXLM can control the degree of overlap between the generated sentence and the original sentence. The bilingual constraint also ensures that the sampled tokens will not deviate from the original meaning, which plays an important role in our strategy.

We plan to investigate whether utilizing back-translated English sentences rather than gold English sentences leads to similar performance, which could get rid of quantitative restriction by the size of the translation corpus to generate an unlimited number of error-corrected sentence pairs.

A Hyper-parameters

The parameters for NAT-based data construction and post edit are presented in Table 8 and Table 9. The hyper-parameters of training the Transformer on NLPCC 2018 Task 2 (Chinese) are listed in Table 10. The hyper-parameters for German and Russian are shown in Table 11. For Russian, we use rulebased corruption without any language-specific operation and confusion set with 50% probability to assist in synthetic data construction.

Acknowledgments

We thank all the reviewers for their valuable comments to improve our paper. The work is supported by National Natural Science Foundation of China under Grant No.62036001 and PKU-Baidu Fund (No.2020BD021). The corresponding author of this paper is Houfeng Wang.