High-resource Language-specific Training for Multilingual Neural Machine Translation

1 Introduction

Recent advances in multilingual neural machine translation (MNMT) aim to build and deploy a single universal model in real industrial scenarios, which supports multiple translation directions by sharing model parameters [Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019; Fan et al., 2020; Lin et al., 2020]. Furthermore, parameter sharing across various languages encourages knowledge transfer, especially from the high-resource language (HRL) to low-resource language (LRL) and even enables zero-shot translation [Aharoni et al., 2019; Zhang et al., 2020].

∗ Contribution during internship at Microsoft Research.

† Corresponding author.

While having attracted many interests, MNMT is still beleaguered by the negative language interference residing in the multilingual parameter sharing [Conneau et al., 2020; Wang et al., 2020b; Gong et al., 2021], where the multiple translation directions degrade performance on high-resource languages. In Figure 1, the multilingual model outperforms the bilingual model on low-resource translations profited by knowledge transferability. But the multilingual model performs worse than the bilingual counterpart when it comes to high-resource translations, which is exactly the manifestation of the negative interference caused by the severe competition in multiple training directions among HRLs and LRLs.

To mitigate the negative interference, the previous works [Wang et al., 2019; Philip et al., 2020] propose the language adapter owing to its simplicity. The adapter modules are usually added to the encoder and decoder layers to indicate the source and target language. With the whole multilingual model frozen, only the parameters of the adapter continue to be tuned. But when the number of languages is large, the adapter methods suffer from the sharp increasing parameters and extra inference time. Another line of research is to extend the depth and width of the MNMT model and construct large-scale corpora to increase model capacity and include more languages [Zhang et al., 2020; Kong et al., 2021]. It still rises in substantial training and inference costs caused by numerous extra parameters.

In this work, we propose a novel multilingual translation model with the high-resource language-specific training (HLT-MT) for one-to-many and many-to-many translation directions in a two-stage training framework. In the first stage, our model is trained only on high-resource languages with the language-specific modules to avoid the negative impact caused by LRLs. To address the negative interference problem among HRLs, we introduce a language-specific pool containing a sequence of independent modules for HRLs. Considering the increasing number of the languages, we apply the selection mechanism to the language-specific pool with the constrained size, denoted as selective language-specific pool (SLP), which enables different groups of certain languages to share the same module from SLP. After pretraining with the high-resource languages, we extract the shared representations of the decoder to transfer the fine-trained knowledge to the low-resource languages.

Our method aims to enhance translation quality of highresource directions with the selective language-specific pool compared to the bilingual counterpart and then transfer the knowledge to the low-resource directions based on the bottom shared features. We conduct experiments on the WMT10 benchmark of 11 languages and OPUS-100 benchmark of 95 languages. Experimental results demonstrate that our method significantly outperforms previous bilingual and multilingual baselines. Besides, extensive probing experiments are performed for the multilingual baseline and HLT-MT, helping further analyze how our method can benefit the multilingual machine translation. Empirical studies show that HLT-MT maintains a balance between language-agnostic and language-distinct features and thus helps to alleviate the negative language interference among various languages.

2 Negative Language Interference

Multilingual translation model aims at transferring knowledge across languages to boost performance on low-resource languages, where the multilingual model is trained in multiple translation directions simultaneously to enable crosslingual transfer through parameter sharing. However, different groups of languages have heterogeneous characteristics, such as different dictionaries and grammars. The previous works have shown [Wang et al., 2020b; Yu et al., 2020] that knowledge transfer is not beneficial for all languages by sharing all parameters. To analyze the effect of mutual influence among different languages, we calculate the cosine similarities between gradients of two translation directions. In Figure 3, we observe that the certain high-resource language sustains negative interference from other HRLs and LRLs. For example, En→Cs acutely conflicts with En→De and En→Hi in the second row of Figure 3. This shows that HRLs are conducive to LRLs but may be hindered by other HRLs and LRLs in turn. To prevent the HRLs from negative interference introduced by LRLs, we focus on sufficiently training the highresource directions and then continue tuning on all directions. To further address the conflicts among HRLs, we propose the selective language-specific pool for different high-resource languages. Our method effectively mitigates the negative interference in the analytic experiments compared to baselines.

3 Our Method

In this section, we introduce HLT-MT for multilingual translation. We propose the two-stage training framework, where the selective languages-specific pool (SLP) and the universal layer are applied for HRLs and LRLs respectively.

3.1 Overview of HLT-MT

Given different translation directions of K languages Lall = {Lk}Kk=1 , we employ SLP for high-resource directions and the universal layer for low-resource directions. Our method is illustrated in Figure 2, where the selective language-specific pool (SLP) for the high-resource languages is inserted into the top of the decoder denoted by θp = {θt}Tt=1 . T is the constrained size of the selective language-specific pool and T ≤ K since the number of languages K can be numerous. Thus, SLP contains T individual modules with the same architecture, and each θt is a sub-network. If the translation direction Li → Lk is the high-resource direction (from the source language Li to target language Lk), we only activate the relevant language-specific module from SLP for Lk. Otherwise, the universal layer is triggered for Lk.

3.2 Multilingual Machine Translation

Given the high-resource bilingual corpora Dh = {Dhm}Mm=1 and low-resource corpora Dl = {Dln }Nn=1 , where M and N separately represent the number of the high-resource and lowresource training corpora of K languages Lall = {Lk}Kk=1 . The multilingual model is jointly trained on the union of the high- and low-resource training corpora Dh ∪ Dl :

where the first and second term denote objective of the highand low-resource training corpora respectively. Θ are shared parameters for all languages. x and y are the sentence pair.

3.3 High-resource Language-specific Training

To prevent the high-resource languages from the negative interference caused by low-resource languages, we only train the model with SLP on high-resource directions, which effectively ameliorates translation quality of high-resource translation directions with slight extra parameters.

To take advantages of the cross-lingual pretrained encoder to boost model performance, our multilingual model is initialized by XLM-R [Conneau et al., 2020]. Besides, we verify the effectiveness of our method on the Transformer model [Vaswani et al., 2017] without any pretrained model. Given the source sentence xLi = {xLi 1 , . . . , xLi u } with u words and target sentence xLk = {xLk 1 , . . . , xLk v } with v words, the shared features hLk s of the target language Lk at the top of the decoder are obtained by the Transformer model:

where hLk s = {hLk s1 , . . . , hLk sj , . . . , hLk sv } and hLk sj is the j-th representation of the target token hLk j generated by the single shared Transformer encoder and decoder.

After obtaining a sequence of decoder representations hLk s of the high-resource language Lk, we project the languageagnostic representations to the language-distinct ones via the language-specific pool with the selective mechanism. Given the translation direction Li → Lk (1 ≤ i, k ≤ K ∧ i ̸= k) and the selective language-specific pool θp = {θt}Tt=1 , the corresponding module θg(Li,Lk) is used to generate the language-distinct representations. g(·) is a map function that maps the language index to the corresponding module index: Lk ∈ {1, . . . ,K} ↦-→ t ∈ {1, . . . , T}. g(Li, Lk) is the map function only depending on the target language and thus can be simplified into g(Lk). Therefore, we project the languageagnostic representations hLk s to the language-specific features hLk b using function Fθg(Lk ) as below:

where hLk s is the representations generated by the shared parameters. Fθg(Lk) is a function defined as below:

where f(·) is the layer normalization and σ(·) is the ReLU activation function. W u g(Lk) ∈ Rde×dh is the up-projection matrix and W d g(Lk) ∈ Rdh×de is the down-projection matrix as shown in the right part of Figure 2, where de and dh are the embedding size and hidden size of SLP (de < dh).

Another issue is how to design a proper map function g(·) with an appropriate selection mechanism for the translation direction Li → Lk. In our work, each source sequence is prefixed with a special target language symbol to indicate the translation direction, which enables the decoder to correctly generate the target sentence with the shared decoder parameters. Therefore, the embedding of the target language symbol is used to select the language-specific module from SLP. The selection function g(·) is defined as:

where eLk = WgE[Lk] of T dimensions. E[Lk] denotes the embedding of the target language Lk symbol. Wg ∈ Rde×T maps the target embedding to the vector eLk , where eLk i is the i-th element of eLk and SLP is comprised of T sub-networks. The sub-network with the highest probability will be selected to produce the language-specific features.

Equation 3 and 5 are only related to θg(Lk) and thus can not propagate gradients to all language-specific parameters. The selective language-specific pool θp = {θt}Tt=1 contains a set of modules described in Equation 4. To tackle the undifferentiable problem of SLP, we use the weighted average to ensure gradients to be propagated to all language-specific modules:

where αLk t is calculated by the target embedding and softmax function. We project the target embedding E[Lk] the probability vector eLk = WgE[Lk] with the learned matrix Wg.

where eLk = WgE[Lk] of T dimensions. Wg ∈ Rde×T and E[Lk] denote the language embedding of Lk. de is the embedding size. αLk t is the t-th element of the vector αLk .

In the practice training, we alternately leverage the Equation 3 and 6 with equal probabilities to learn the map function and generate the language-distinct representations hLk b . Finally, the representations are used to generate the target sentence xLk :

where xLk is the target sentence and W o ∈ de × V is the output matrix, where V is the vocabulary size.

3.4 Low-resource Transfer

After training the multilingual model on the high-resource language pairs, the bottom features generated by the shared parameters are utilized for the low-resource target sentence generation. Then, our multilingual model is jointly trained on the high-resource bilingual corpora Dh and low-resource bilingual corpora Dl . Given the low-resource translation direction Li → Lj, the shared representations hLj s generated by the shared parameters Θ are fed into a universal layer θu:

where hLj b are features generated by the shared parameters in Equation 2. θu is a sub-network same as the θt (1 ≤ t ≤ T) of SLP θp = {θt}Tt=1 . All low-resource languages share the same universal layer to project the shared features hLj s to the last representations hLj b . Then the target sentence is produced by hLj b and output matrixW o similar to Equation 8.

3.5 Training Strategy

Our model first accumulates the cross-entropy loss on the high-resource pairs and the disparity loss. Then, the multilingual model trained on multilingual corpora to maintain the performance of high-resource languages and meanwhile transfer the knowledge to the low-resource languages.

Disparity Loss To encourage different languages to select different language-specific modules of SLP, we minimize the disparity loss Ld, which measures the similarity of languagespecific layer selection among languages.

where αLi , αLk ∈ RT denote the selection probabilities generated by Equation 7, where SLP contains T modules.

High-resource Language-specific Training The objective is to minimize the cross-entropy loss of high-resource training corpora and the auxiliary disparity loss jointly as below:

where Θ denotes all shared parameters and θp are parameters of SLP. x and y are sentence pair.

Multilingual Training After trained on the high-resource directions, the multilingual model is continued to be tuned on the union of the high- and low-resource corpora Dh∪Dl with the extra SLP for high-resource languages and the universal layer for low-resource languages:

where Θ are shared parameters for all languages. SLP contains a list of language-specific layers for HRLs and θu is a universal layer for LRLs.

4 Experiments

4.1 Datasets

To evaluate our method, we conduct experiments on the WMT-10 and the OPUS-100 dataset.

WMT-10 We use a collection of parallel data in different languages from the WMT datasets to evaluate the models [Wang et al., 2020a], The parallel data is between English and other 10 languages, including French (Fr), Czech (Cs), German (De), Finnish (Fi), Latvian (Lv), Estonian (Et), Romanian (Ro), Hindi (Hi), Turkish (Tr) and Gujarati (Gu).

OPUS-100 We use the OPUS-100 corpus [Zhang et al., 2020] for massively multilingual machine translation. OPUS100 is an English-centric multilingual corpus covering 100 languages, which is randomly sampled from the OPUS collection. We obtain 94 English-centric language pairs after dropping out 5 languages, which lack corresponding test sets.

4.2 Baselines

Our method is compared to the bilingual and multilingual methods. For a fair comparison, XLM-R and LS-MNMT are initialized by XLM-R [Conneau et al., 2020]. BiNMT [Vaswani et al., 2017] is the bilingual Transformer model. MNMT [Johnson et al., 2017] is jointly trained on all directions, where the target language symbol is prefixed to the input sentence. mBART [Liu et al., 2020] is an encoderdecoder pretrained model and then is finetuned on all corpora. XLM-R [Conneau et al., 2020] is initialized by the pretrained model XLM-R [Ma et al., 2020]. LS-MNMT [Fan et al., 2020] integrates the language-specific layers of all languages into the end of the decoder.

4.3 Training and Evaluation

We adopt Transformer as the backbone model for all experiments. We train multilingual models with Adam (β1 = 0.9, β2 = 0.98). The learning rate is set as 5e-4 with a warm-up step of 4,000. The models are trained with the label smoothing cross-entropy with a smoothing ratio of 0.1. The batch size is 4096 tokens on 64 Tesla V100 GPUs. For WMT-10, we first train the multilingual model with 6 languages and then finetunes on all languages. For OPUS-100, the model is trained in the languages where the number of pairs exceeds 10K. The evaluation metric is the case-sensitive detokenized sacreBLEU [Post, 2018].

4.4 Main Results

WMT-10 As shown in Table 1, our method clearly improves multilingual baselines by a large margin in 10 translation directions. Previously, multilingual machine translation underperforms the bilingual translation model in richresource scenarios. It is worth noting that our multilingual machine translation baseline XLM-R is already very competitive initialized by the cross-lingual pretrained model. Interestingly, our method outperforms the bilingual baseline in the high-resource translation direction, such as En→De translation direction (+1.5 BLEU points). Our method consistently outperforms the multilingual baseline on all language pairs, confirming that using HLT-MT to alleviate negative interference can help boost performance.

OPUS-100 In Table 2, we observe that HLT-MT achieves reasonable results on 94 translation directions. The improvement can be attributed to the high-resource training with the selective language-specific pool, which avoids the competition between high-resource and low-resource training directions. Another benefit of our approach is light and convenient to be applied to different backbone models since the parameters of the selective language-specific pool on the top of the decoder are tiny compared to all parameters.

5 Analysis

Size of Language-specific Parameters The size of the selective language-specific pool depends on the two key factors, namely the hidden size dh and the number of modules T described in Equation 4 and 3. We tune the different values of dh and T in Figure 4(a) and 4(b) on the WMT-10 dataset. Naturally, the selective language-specific pool with a larger capacity leads to better performance. Increasing the number of the selective pool brings more improvement than the improvement of hidden size. Our method can efficiently reduce the language-specific parameters (T = 3) using the selection mechanism and get comparable results compared to the baseline, where each high-resource language has the independent language-specific layer (T = 6).

Ablation Study In Table 3, we empirically validate our approach on the different backbone models including Transformer [Vaswani et al., 2017] without any pretrained model and XLM-T [Ma et al., 2020] initialized by the cross-lingual pretrained model XLM-R [Conneau et al., 2020]. Highresource training significantly helps improve the model performance but has trouble in effectively handling the lowresource translation directions merely by sharing all parameters, which is caused by the competition in the shared parameters between high-resource and low-resource languages. By introducing the selective language-specific pool (SLP) and extracting the bottom representations for low-resource languages, our approach ameliorates all translation directions.

Conflicting Gradient To delve into the function of the language-specific module for multilingual training [Yu et al., 2020], we define Φ(La,Lb) = gLa ·gLb ∥gLa ∥∥gLb ∥ as the cosine similarity between two task gradients gLa and gLb , where gLa and gLb separately denote the gradients of the En→ La and En→ Lb translation direction. Φ(La, Lb) determines whether gLa conflicts with gLb by computing the cosine similarity between vectors gLa and gLb , where the small value indicate conflicting gradients. Figure 5(b) and 5(a) show the similarities of the baseline and our method between different training directions. Different training tasks of our method have similar optimization, where Φ(La, Lb) has larger value compared to the baseline scores. It corroborates that our language-specific training can effectively mitigate the conflicting gradients.

Language-specific Selection Mechanism The selective language-specific pool (SLP) of high-resource languages significantly contributes to translation quality. Equation 7 describes the selection of the module with the highest probabilities generated by the embedding of the target language symbol for the given translation direction. In Figure 6(a), θ1 is used for En, Fr, Cs, and Fi. θ2 is for De and Lv. θ3 is for Et. The universal layer is used for all low-resource languages.

In Figure 6(b), we calculate the overlaps of dictionaries among multiple languages to measure the relationship of different languages. The language similarity between La and Lb is calculated by Sim(La,Lb) = ∥DLa ∩DLb ∥ ∥DLa ∪DLb ∥ , where DLa and DLb are the dictionary of language La and Lb. Figure 6(b) shows that the language Et has the minimum overlap between other high-resource languages, where θ3 is only used for Et. Therefore, we conclude that similar languages tend to select the same language-specific module from SLP.

Decoder Representation Visualization We randomly select 500 English sentences and visualize their representations [Maaten and Hinton, 2008] of the bottom decoder layers and the language-specific layer in Figure 7. The first hidden state of the decoder is regarded as the sentence representation. Compared to Figure 7(a), 7(b), and 7(c), different languages become more distinct and less likely to overlap with each other in Figure 7(d), proving that the selective languagespecific pool (SLP) effectively projects the language-shared representations into language-distinct ones for better target generation of different target languages.

6 RelatedWork

Multilingual neural machine translation (MNMT) [Johnson et al., 2017; Aharoni et al., 2019; Fan et al., 2020; Kong et al., 2021; Tang et al., 2021] enables numerous translation directions by shared encoder and decoder for all languages. The MNMT system can be categorized into one-to-many [Wang et al., 2018], many-to-one [Tan et al., 2019], and many-tomany [Pan et al., 2021] translation. Previous studies utilize assisting high-resource languages to improve low-resource or even zero-shot translation.

While MNMT is promising, it often underperforms bilingual baselines due to the interference in shared parameters, especially on high-resource pairs [Wang et al., 2020b]. To address this issue, language-specific modules are proposed to both enhance the low-resource translation and maintain the high-resource performance. Recent works mainly focus on designing language-specific components to boost the rich-resource translation quality [V´azquez et al., 2019; Philip et al., 2020; Gong et al., 2021]. Further works discuss when and where language-specific capacity matters most in MNMT [Escolano et al., 2021]. Our method finds a better balance between language-specific and language-agnostic models to mitigate negative interference.

7 Conclusion

In this work, we propose a novel multilingual translation model with the high-resource language-specific training called HLT-MT. The multilingual model is trained on multiple high-resource corpora with the selective language-specific pool, followed by continuing training on both high- and lowresource languages. Experimental results evaluated onWMT10 and OPUS-100 benchmarks demonstrate that HLT-MT significantly outperforms all previous baselines.