UM4: Unified Multilingual Multiple Teacher-Student Model for Zero-Resource Neural Machine Translation

1 Introduction

The encoder-decoder framework [Vaswani et al., 2017; Gheini et al., 2021] has gained outstanding performance on rich-resource machine translation tasks, such as EnglishGerman, English-French, and Chinese-English [Koehn et al., 2019; Zhou et al., 2021; Johnson et al., 2017], where largescale parallel corpora are available. However, it is incapable of directly modeling the zero-resource translation task when the parallel training data does not exist.

A straightforward solution for the zero-resource machine translation problem is the pivot translation approach [Bertoldi et al., 2008; Wu and Wang, 2009; Zahabi et al., 2013; Cheng et al., 2017]. Bilingual pivot-based models perform the twopass translation, which increases the computation cost and potentially suffers from the error propagation problem [Zhu et al., 2013]. There are also some works [Chen et al., 2017;

∗ Equal contribution. Work done during internship at Microsoft.

† Corresponding author.

Currey and Heafield, 2019; Kim et al., 2019a] directly building the source→target model but limited by the bilingual translation task. Beyond pivot-based methods, the multilingual model trained over multiple pivot corpora with shared parameters utilizes the language symbol to infer the desired translation direction [Firat et al., 2016; Johnson et al., 2017; Lakew et al., 2019; Currey and Heafield, 2019]. The multilingual model benefits from different language pairs and only requires one-step translation, which avoids error propagation and saves inference time. But the performance of this approach [Kim et al., 2019b] is worse than pivot-based models.

Along the line of leveraging the multilingual model to address the zero-resource translation problem, we propose a novel method called Unified Multilingual Multiple teacherstudent Model for NMT (UM4). Given the available corpora of the pivot and other languages, we directly build the source→target student translation model guided by the multilingual multiple teachers as shown in Figure 1. The multiple teacher models can be decomposed into a source-teacher model, a target-teacher model, and a pivot-teacher model. The source-teacher model transfer the knowledge from the pivot to the source sentence. The target teacher distills pivot knowledge to the target side and boosts the capability of the target generation. The pivot-teacher model further enhances the student model by mining the potential of monolingual pivot corpora. The overall distilled corpora from unified teachers with normalized scores are used for the student.

Specifically, we first train a single multilingual model on all pivot corpora as unified multiple teachers sharing all model parameters. Then, we construct the distilled multilingual corpora of all zero-resource directions using multiple teacher models. The distilled corpora with the normalized scores generated by the unified multiple teachers are used to guide a source→target student. We conduct experiments on the multilingual corpora from the WMT benchmark of 9 languages with 72 translation directions. The experimental results show that our method can significantly outperform multilingual baselines and pivot-based methods. Furthermore, we verify the effectiveness of our method by perturbation experiments and visualization of the multilingual sentence representations. Analytic results demonstrate that our UM4 student model with better crosslingual ability enhances zero-resource translations and avoids error propagation.

2 Our Approach

In this section, we introduce the unified multiple teacherstudent model for zero-resource machine translation. As illustrated in Figure 2, our method simultaneously uses multiple teacher models to train a multilingual end-to-end translation model when the direct parallel data is unavailable.

2.1 Overview of UM4

Given the bilingual corpora DB = {DBn }Nn=1 of N languages, where one side is the pivot language Lz and the other side is the language Ln ∈ {Ln}Nn=1 , the multilingual model is trained on the available pivot corpora DB to solve the zeroresource translation without direct parallel data between the zero-resource pair Li and Lj (1 ≤ i, j ≤ N and i ̸= j):

where x, zx denote the source and pivot language sentence in the bilingual corpus DB. y, zy denote the pivot and target sentence. LD is the combined objective of the multilingual model. The multilingual model over the source-pivot and pivot-target corpora with shared parameters prepends the language symbol to indicate the zero-resource translation direction from language Li to language Lj .

Without parallel training data for zero-resource language pairs, the multilingual model can easily translate into a wrong language and result in poor translation quality. Therefore, we introduce the synthetic multilingual multiple corpora of zeroresource language pairs DS = {DSm }Mm=1 .

where x and y denote the source and target language sentence in the distilled multilingual corpora DS. wx,y is the weight of the training sample frommultilingual multiple teachers.

Our multilingual student model is trained on both original corpora DB and distilled corpora DS, which improves the translation quality under the supervised manner among zeroresource directions:

where LT is the total objective of our multilingual student model. LD and LS denote training objective over the original pivot corpora DB and distilled corpora DS respectively.

2.2 Multiple Teacher Models

Formally, given the source-pivot and target-pivot parallel corpus DBi = {x(k) , z(k) x }|DBi | k=1 and DBj = {y(k) , z(k) y }|DBj | k=1 , we aim to build a source→target translation model θx→y for the zero-resource translation task. x and y denote the source and target sentence respectively, zx and zy denote the pivot sentence from the source-pivot corpus DBi and pivot-target corpus DBj separately. |DBi | and |DBj | are the size of corpora DBi and DBj . θ denote model parameters.

Source-teacher Model If the target sentence y and pivot sentence zy are parallel from the dataset DBj . The source teacher θzy→x is trained on the source-pivot corpus DBi . The source-teacher student training objective can be written as:

where P(x|zy; θzy→x) is the weight generated by the sourceteacher model θzy→x.

Target-teacher Model If the source sentence x and pivot sentence zx are parallel from the dataset DBi . The target teacher θzx→y on the pivot-target corpus DBj . The targetteacher student training objective can described as:

where P(y|zx; θzx→y) is the weight generated by the targetteacher model θzx→y.

Pivot-teacher Model Given the monolingual pivot corpus DM, the pivot-teacher is used to guide the student model. The pivot-teacher model θzm→x ∪ θzm→y is trained on the pivot corpora DBi and DBj . The pivot-teacher student training objective can described as:

where wx,y = P(y|zm; θzm→y)P(x|zm; θzm→x) is the weight generated by the pivot-teacher model.

All teachers are based on the multilingual training on the available corpora DB and shares the same semantic space for all languages. Therefore, the unified teacher is comprised of different teachers with respective functions simultaneously.

Combining the source-teacher model, the target-teacher model, and the pivot-teacher model, the training objective of our teacher-student training can be described as:

where the parameters of multiple teachers remain unchanged during the training process.

We adopt sequence-level knowledge distillation [Chen et al., 2017] to distill the knowledge from teacher models to the student model in practice. Specifically, we use multiple teacher models to construct the distilled corpora of zeroresource language pairs DS = {DS1 , . . . , DSM } with corresponding normalized scores, combined with the original pivot corpora DB = {DB1 , . . . , DBN } to train the student model.

As shown in Figure 2, our method can utilize the sourceteacher model, target-teacher model, and pivot-teacher model simultaneously to guide the source→target student model, resulting in a more powerful student model.

2.3 Teacher-Student Transfer

This section will introduce the knowledge distillation details of multilingual multiple teacher-student. Limited by the exponential search space of the source sentences x and y, we employ the beam search strategy to generate N-best translation candidates 1 and renormalize the probabilities to teach the student model to approximate the distribution of the teacher model as below:

where S is the beam size of the fixed teacher model. wsx,y is the probability of the s-th sentence generated by the teacher model. τ is the temperature. The temperature τ → 0 increases the weight for the top-selected distilled sentences. We set τ < 1.0 to force the model to focus more on the bestdistilled sentence pairs when training.

1 Sampling from synthetic data generated by the teacher model according to the probabilities is an easy way to force the student to approximate the teacher [Kim and Rush, 2016].

We first train a single multilingual model with all available pivot corpora DB as the multiple teacher models for all languages instead of training different bilingual teacher models.

Source-teacher Transfer For the source-teacher model, we use the pivot→source model to translate the monolingual pivot sentences of the pivot-target corpus DB into the distilled source sentences. In this way, we get a distilled corpus Dsrc S . According to the Equation 4, the distilled corporaDsrc S with the score wx,y = P(x|zy; θzy→x) is used to teach the student model.

Target-teacher Transfer We adopt the beam search strategies and translate the monolingual pivot language part in the source-pivot corpora DB into target language sentences. Another distilled corpora Dtgt S is obtained with the score wx,y = P(y|zx; θzx→y) for knowledge transfer.

Pivot-teacher Transfer Given the additional monolingual pivot corpora, the pivot sentences are separately translated to the distilled source and target sentences by the pivot-teacher model. We obtain the distilled corpora Dpivot S with the score wx,y = P (x|zm; θzm→x)P (y|zm; θzm→y) from the monolingual corpus DM.

Eventually, with the parameters of the multilingual teacher model fixed, we generate the distilled knowledge and combine them into a whole training dataset DS = Dsrc S ⋃ Dtgt S ⋃ Dpivot S to train the multilingual source→target student model.

3 Experiments

We evaluate our method on the multilingual dataset including 9 languages and 56 zero-resource translation directions. English is the most popular language and there are extensive English-centric data in the real world compared to other languages. Therefore, English (En) is treated as the pivot language in all experiments.

3.1 Dataset

All experiments are conducted on the multilingual dataset of 9 languages extracted from the previous work [Wang et al., 2020], including English (En), French (Fr), Czech (Cs), German (De), Finnish (Fi), Estonian (Et), Romanian (Ro), Hindi (Hi), and Turkish (Tr).

Bitext Data We collect the training data from the latest available year of each language between English and other languages on the WMT benchmark and exclude WikiTiles. The duplicated samples are removed and the number of parallel data of each language pair is limited to 10 million by randomly sampling from the whole corpus. For 72 translation directions of 9 languages, we use the same valid and test sets from TED Talks as the previous work 2 for evaluation.

2 http://phontron.com/data/ted talks.tar.gz

Monolingual Data The English monolingual data is collected from NewsCrawl 3 and randomly sample 1 million English sentences. We use a multilingual NMT model to translate these English monolingual data to sentences of other languages as the augmented parallel data, which is utilized as the back-translation data for all baselines. Our method uses the pivot-teacher model to guide the training of the source→target student model with the monolingual data.

3.2 Evaluation

During the inference, the beam search strategy is performed with a beam size of 5 for the target sentence generation. We set the length penalty as 1.0. The last 5 checkpoints are averaged for evaluation. We report the case-sensitive detokenized BLEU using sacreBLEU 4 .

3.3 Baselines

Our method is compared with pivot-based and multilingual baselines. Bilingual Pivot-based [Cheng et al., 2017] translate source to target via pivot language using two single-pair NMT models trained on each pair. Multilingual Pivot-based [Lakew et al., 2019] leverages a single multilingual NMT model trained in all available directions for pivot translation. The details of the multilingual baselines are described as follows. Multilingual [Johnson et al., 2017] shares the same vocabulary of all languages and prepends the language symbol to the source sentence to indicate the translation directions. Monolingual Adapter [Philip et al., 2020] tunes adapter of each language for zero-shot translation based on a pretrained multilingual model. Teacher-Student [Chen et al., 2017] uses the pivot-target translation model to teach the sourcetarget translation model. MTL [Wang et al., 2020] proposes a multi-task learning (MTL) framework including the translation task and two denoising tasks.

3.4 Implementation Details

All experiments are conducted based on the Transformer big architecture [Vaswani et al., 2017]. Both the encoder and decoder contain 6 layers with 16 heads per layer. The word embedding size dmodel is set to 1024 and the FFN (feed-forward network) size is 4096. The learning rate is set to 3e-4 with 4000 warmup steps on the multilingual dataset. Adam [Kingma and Ba, 2014] is used for updating the parameters. The model with a mini-batch size of 4096 tokens is trained on 64 Tesla V100 GPUs.

3.5 Experiment Results

The evaluation results over the test sets against the baselines are listed in Table 1 and Table 2. The source-pivot corpus is high-resource compared to the low-resource pivot-target corpus in Table 1, and the source-pivot corpus is low-resource compared to the high-resource pivot-target corpus in Table 2.

As observed in Table 1, pivot-based methods including Multilingual Pivot and Bilingual Pivot significantly outperforms the multilingual methods including Multilingual, MTL, and Monolingual Adapter. But pivot-based methods still suffer from error propagation and are computationally expensive by the two-pass translation. On the contrary, our UM4 method is a unified multilingual source→target model and alleviates this problem. Compared with strong baselines Teacher-Student, our model achieves consistent improvements with ≥ 0.6 BLEU points gains (≥ 0.6 with "MonoData") on Avg>28 . It shows that our student model learns a high-quality representation space across multiple languages with guides of multiple teachers to enforce the capability of zero-resource translation directions. Given the monolingual pivot corpus, our method also beats the pivot-based and multilingual methods by the distilled knowledge from the pivotteacher model.

3 http://data.statmt.org/news-crawl

4 BLEU+case.mixed+lang.{src}-

{tgt}+numrefs.1+smooth.exp+tok.13a+version.1.3.1

In Table 2, the size of the source-pivot corpus is smaller than the pivot-target corpus, so the source teacher transfers more knowledge from the larger pivot-target corpus to guide the student model with more distilled sentences. Our UM4 also beats all previous methods and gains ≥ 0.9 BLEU points gains (≥ 0.5 with "MonoData") on Avg<28 . It demonstrates the effectiveness and significance of the introduction of source-teacher model. Our UM4 method without monolingual data (Avg>28=11.8 and Avg<28=14.7) performs even better than all baselines with back-translation data.

3.6 Analysis

Effect of Different Teachers To investigate the effect of the different teachers, we train 7 student guided by all possible combinations of the source-teacher, target-teacher, and pivot-teacher model. Our method combines multiple teachers to direct the source-target student model simultaneously so that our method can improve performance. Table 3 shows ablation results guided by different teachers. Consistently, more teachers can lead to better results, which demonstrates that our proposed model can comprehensively manipulate the advantages of different teachers.

Size of Distilled Training Data Given the multilingual pivot corpora of the pivot language and other N languages, N(N - 1) distilled training sets of zero-resource directions are used to guide the multilingual student model. The overall scale of corpora contains TN(N - 1) sentence pairs, where T is the average size of the distilled corpora. To reduce complexity O(TN 2 ) to O(TN), we adopt a downsampling strategy with 1N downsampling ratio as formulated below:

where T ′ is the size of the downsampled corpus. Tm = 1M is the threshold to avoid undersampling the low-resource pairs.

In our work, 16 parallel corpora and 56 (8×7) distilled corpora are used for training. The results with different sampling ratios are listed in Figure 3, which indicates the proper downsampling ratio ( 1N = 18 = 12.5%) can simultaneously reduce computation cost and get comparable performance. The training time cost of our model is acceptable in a practical scenario. Therefore, our proposed method has been evaluated on the 72 translation directions and can be easily extended to more languages.

Sample Space of Possible Sequences We employ the sequence-level knowledge distillation as formulated in Equation 8 and examine our method with different sample space (beam size) settings, where S denotes beam size. Limited by the exponential search space, we use the beam search strategy with a beam size of S (S ∈ [1, 4]) to guide the student model. Figure 4 shows that the multilingual student model gains the best performance when S = 3 or S = 4 on the zero-resource translation tasks. Considering the computation cost and model performance, we set S = 4 in our work.

Robustness against Input Errors To further test the robustness of different methods, we add perturbations with different ratios to the source sentence of the test set in Figure 5. The input sentences are randomly corrupted with four types of perturbations including (1) deletion (drop words), (2) masking (replace words with "[unk]"), (3) swap (swap words), and (4) substitution (replace words with random words in the vocabulary). For the test set, we randomly perturb source sentences by a fixed corrupting probability.

Given the perturbed input sentence with different corrupting probabilities in Figure 5, the multilingual model [Johnson et al., 2017] (green line) is easily influenced by the noisy input and degrades to the worst performance. It indicates the multilingual model delivers unstable translation performance for the zero-resource directions that are unseen at training time. The performance of the multilingual pivotbased method [Lakew et al., 2019] (red line) also consistently drops more than our method due to error propagation introduced by the two-pass translation procedure. The results demonstrate that the multilingual student guided by multiple teachers performs better and avoids error propagation.

Number of Training Language Pairs Our student model is trained on the original parallel corpora DB and the distilled training corpora DS generated by multiple teachers described in Equation 3. The zero-resource translation ability of our student model benefits from the shared semantic space. In Table 4, "Supervised" denotes the multilingual model trained only with the original corpora of 16 directions, "Zero-resource" denotes the student multilingual model trained only with distilled corpora of 56 directions, and "Both" denotes our method trained on both the original corpora and distilled corpora. Our UM4 model trained jointly with 72 directions gets the best performance by transferring the knowledge among different languages.

4 RelatedWork

Zero-Resource NMT Zero-resource neural machine translation (NMT) is a challenging task since the source-target parallel corpus is not available. A feasible solution is the pivotbased NMT [Zhu et al., 2013; Firat et al., 2016; Cheng et al., 2017; Zheng et al., 2017; Currey and Heafield, 2019], where the source language is translated to the pivot language followed by translating the pivot language into the target language. This two-pass translation procedure both increases the complexity and potentially suffers from the error propagation problem because the errors made by the source→pivot model will be introduced to the pivot→target model [Lakew et al., 2019]. Recent works [Chen et al., 2017; Zheng et al., 2017; Currey and Heafield, 2019] apply explorations into using the available parallel corpus and the additional monolingual corpus to improve zero-resource performance but limited by the bilingual setting.

Multilingual NMT Multilingual neural machine translation (MNMT) [Firat et al., 2016; Johnson et al., 2017; Lakew et al., 2019; Tan et al., 2019; Garcia et al., 2020; Yang et al., 2021] provides an alternative manner for zero-resource translation without any source-target parallel data but the performance is worse than pivot-based models. The multilingual models with language-aware module [Bapna and Firat, 2019; Zhang et al., 2020; Philip et al., 2020] are used to translate in zero-resource directions which are unseen at training time. However, the multilingual models often underperform the pivot-based models and deliver poor zero-resource translations. Multilingual pretraining method [Kim et al., 2019a] are used to obtain the crosslingual encoder and then finetunes on the pseudo data. Inspired by previous works [Chen et al., 2017; Zheng et al., 2017], we employ multilingual multiple teachers to guide the multilingual source→target student to enhance the zero-resource translation.

5 Conclusion

In this paper, we propose a novel method called Unified Multilingual Multiple teacher-student Model for NMT (UM4) to ameliorate the translation of zero-resource directions. Our method unifies the source-teacher model, targetteacher model, and pivot-teacher model to guide the multilingual source→target student model, alleviating the error propagation problem caused by two-pass translation. Experimental results on the multilingual dataset of the WMT benchmark corroborate the effectiveness of our method in leveraging the distilled knowledge from the unified teachers.