Improving Zero-Shot Cross-Lingual Transfer via Progressive Code-Switching

1 Introduction

Zero-shot cross-lingual transfer learning aims to train an adaptable model on a source language that can effectively perform on others without labelled data in the target languages. This study is particularly valuable in circumstances where there are limited or no annotations available for the target languages. In recent years, the multilingual pre-trained language models, such as mBERT [Devlin et al., 2019], XLM [Conneau and Lample, 2019] and XLM-R [Conneau et al., 2020] have achieved significant performance improvements through fine-tuning on source language data and direct application to target language data (as shown in Figure 1(a)). Furthermore, it has been discovered that a further multilingual contextualized representation alignment improves the zero-shot cross-lingual transfer performance by exploiting a bilingual dictionary to replace some tokens in the source text with target-lingual translated words. This strategy, named Code-Switching (CS), usually randomly chooses substitution words and has been shown improvements in many zeroshot cross-lingual tasks [Liu et al., 2020; Qin et al., 2021; Zheng et al., 2021; Ma et al., 2022; Wang et al., 2023b] (as shown in Figure 1(b)).

∗ Corresponding author

Code-Switching, as a data augmentation technique, on one hand, inevitably leads to losing original contextual information when over-replacing words with other lingual synonyms within a sentence. On the other hand, under-replaced code-switching results in insufficient cross-lingual alignment and limited data variation may limit the model's ability to learn and transfer knowledge across languages. Existing studies have indicated that such uncontrolled samples might not necessarily benefit model learning [Qin et al., 2021; Yang et al., 2021]. For example, an original English sentence is "All the services were great." and its corresponding multilingual code-switching sentence is "todas (ES) les (FR) services waren (DE) great.", wherein capital letters in parentheses represent target language abbreviations (e.g. ES indicates Spanish). Because of very different contextual sentence expressions, such code-switched sentences may not contribute to multilingual word representation alignment, but in some extreme cases, hurt the trained models' cross-lingual generalization ability. Therefore, we should devise some kind of switching scheme to govern the switching extent subtly, e.g., the number of substitutions or the model discrimination ability impact of code-switched sentences.

In this study, we assume that easy code-switching samples could act as pre-training knowledge, which guides the model optimization on harder code-switching data. For instance, we first consider the easy code-switching sentence "All les (FR) services were great.", which shares substantial contextual overlap with the original sentence. In this case, the model can correctly align the word pair "the (EN) - les (FR)". Then for a harder code-switching sentence, "All les (FR) services waren (DE) great.", the previously aligned "the (EN) - les (FR)" can serve as a pivot for aligning the new word pair "were (EN) - waren (DE)". By adopting this progressive strategy, new word pairs can be aligned based on the previously identified pairs. Consequently, even the hard code-switching data such as "todas (ES) les (FR) services waren (DE) great." would become easier to be handled and thus progressively improving multilingual alignment.

To pursue both effective utilization of code-switching data and model generalization, we borrow the idea of curriculum learning [Bengio et al., 2009] and propose a progressive code-switching framework termed PCS (as shown in Figure 1(c)). However, determining the difficulty of code-switching data is challenging, as the importance of each word varies for different tasks. Drawing inspiration from explanation learning methods, we develop a difficulty measurer to estimate the difficulty of code-switching sentences based on the contribution of substitution words toward the prediction. We then introduce a code-switcher with an adjustable temperature parameter to generate appropriate code-switching sentences that align with the current curriculum difficulty. Furthermore, to mitigate the problem of catastrophic forgetting in curriculum learning, we design a scheduler that dynamically adapts the difficulty level to revisit the previously acquired knowledge.

In summary, we make the following key contributions:

• We propose a progressive code-switching method for zero-shot cross-lingual transfer, which mitigates the negative impacts of uncontrolled code-switching data and improves the multilingual representation alignment.
• We introduce a word relevance score-guided difficulty measurer, a temperature-adjustable code-switcher, and a dynamic scheduler. They collaboratively regulate the switched samples' difficulty and gradually generate code-switching samples in a controlled manner.
• We comprehensively evaluate our proposed approach on three different cross-lingual tasks covering ten different languages. The results demonstrate that PCS substantially enhances performance compared to some strong code-switching baselines.

2 RelatedWork

Zero-shot cross-lingual transfer aims to learn a model with labelled source language data and perform well on other target languages. In recent years, there have been some pretrained multilingual language models for cross-lingual transfer, such as mBERT [Devlin et al., 2019], XLM [Conneau and Lample, 2019] and XLM-R [Conneau et al., 2020; Goyal et al., 2021]. Some studies further improve the alignment of multiple different languages by parallel corpora [Artetxe and Schwenk, 2019; Cao et al., 2020; Pan et al., 2021; Chi et al., 2021; Wei et al., 2021b]. Recently, code-switching leverages low-resource bilingual dictionary to align multilingual contextual representations and achieve the state-of-theart performance in many cross-lingual tasks, such as text classification [Lee et al., 2021], dialogue system [Liu et al., 2020; Ma et al., 2022], sequence tagging tasks [Feng et al., 2022], and question answering [Nooralahzadeh and Sennrich, 2023]. There are additional attempts to avoid the original signal loss in code-switching [Lee et al., 2021; Feng et al., 2022]. Most of the code-switching work in word substitution is random and our work considers the negative impacts of excessive code-switching data. To the best of our knowledge, only a few studies have focused on word selection in code-switching, and none of them investigates progressive code-switching.

Curriculum learning is proposed as a machine learning strategy by feeding training examples to the model by a meaning order, which is inspired by the learning process of humans and animals [Bengio et al., 2009]. In general, curriculum learning contains a difficulty evaluator used to evaluate the difficulty score of instances, and a scheduler used to decide how examples should be fed to the model. Curriculum learning has been successfully applied to many areas in natural language processing, such as question answering [Sachan and Xing, 2016], reading comprehension [Tay et al., 2019], dialogue system [Shen and Feng, 2020; Zhu et al., 2021] and text classification [Lalor and Yu, 2020; Xu et al., 2020]. Another line of research aims at providing a theoretical guarantee of curriculum learning, including transfer learning method [Xu et al., 2020] and optimization methods [Kumar et al., 2010; Graves et al., 2017]. In this work, we adopt curriculum learning into code-switching and address the negative impacts of code-switching in cross-lingual transfer.

3 Progressive Code-Switching

We introduce Progressive Code-Switching (PCS) method in this section, which can be applied to various zero-shot crosslingual transfer learning downstream tasks. Figure 2 depicts an overview of the PCS framework. We will describe the details of our approach from the following four components: difficulty measurer, code-switcher, scheduler, and model trainer.

Problem Formulation. Formally in a zero-shot crosslingual transfer task, given a source language sentence x = {x1, x2, ..., xL} with L words, our code-switcher module generates the augmented code-switching sentence xa = {xa1 , xa2 , ..., xaL} by replacing or not xi with xai from the predefined bilingual dictionary according to word relevance and difficulty temperature. The label y is kept the same as the original sentence. With the word relevance score of each word xi measured in a sentence, the code-switcher gradually generates harder code-switching sentences. A pre-trained difficulty measurer with the source language data determines the word relevance. Our goal is to learn a model with source language data Dtrain S = {(x, y)} and augmented codeswitching data Dtrain CS = {(xa , y)} to perform zero-shot prediction on target languages Dtest T = {x}.

3.1 DifficultyMeasurer

The idea of our PCS lies in the strategy of "training from easier code-switching data to harder code-switching data". Due to the code-switching data being augmented from the original data without a predefined difficulty score, we first need to measure what kinds of data are harder than others. Existing popular difficulty measurers in natural language tasks include sentence length [Spitkovsky et al., 2010], word rarity [Platanios et al., 2019], and replacement ratio [Wei et al., 2021a]. However, these measures ignore that replacing the same words produces different degrees of change under different tasks. Therefore, this cannot guarantee the 'easy-tohard' order, because a code-switching sentence with a few important words being replaced is more difficult than one with a large of irrelevant words being replaced for the model. Here, we argue that code-switching sentences having more important words being replaced have bigger semantic distortion with the original sentences than others, and are treated as more difficult instances. Inspired by explanation methods [Arras et al., 2019], we use Layer-Wise Relevance Propagation (LRP) [Bach et al., 2015] to assign each word a relevance score indicating to which extent it contributed to a particular prediction, as the basis for estimating the difficulty level of CS. In other words, LRP can quantify whether a token is important in the model's decisions to the prediction we are interested in.

We suppose given a trained model f, which has learned a scalar-valued prediction function, e.g. fc(x) means prediction probability of an input sequence x being class c in the classification task. We adopt a BERT-based LRP [Wu and Ong, 2021] consisting of a standard forward pass, followed by a specific backward pass. For a linear layer of the form as Eq. (1), and given the relevance of the output neurons rj, the relevance of the input neurons ri are computed through the following Eq. (2), where ϵ is a stabilizer.

In practice, starting from the output neuron whose relevance is set to the value of the prediction function, i.e. fc(x), LRP uses Eq. (2) to iteratively redistribute the relevance from the last layer fc(x) down to the input layer x, layer by layer, and verifies a relevance conservation property. We denote r(d)(x) the relevance of the d-th dimension (d ∈ {1, 2, ...,D}) of the token x and we can derive it as follow Eq. (3):

where z(l) is column matrix of hidden states in layer l, and derivatives of non-linear activation functions a′ (·) are ignored as proposed in [Arras et al., 2019; Wu and Ong, 2021]. For non-linear layers such as the self-attention layer and the residual layer, zl is approximated by the first term in the Taylor expansion formally as Eq. (4) as proved in [Bach et al., 2015], where fψ is an arbitrary differentiable function, and ˆx is the Taylor base point where fψ(ˆx) = 0. We derive the relevance score of the token r(x) w.r.t. the class c using absolute sum of r(d)(x), i.e. r(x) = ∑d r(d)(x).

Via this backward pass, we can observe which words really contributed to the output. An example is shown in Figure 2, "services" and "great" significantly contribute to the prediction of "positive".

3.2 Code-Switcher

To generate code-switching sentences that match the difficulty level of the current curriculum, we introduce a codeswitcher that incorporates a variable temperature denoted as τ. This temperature parameter represents the proportion of words to be replaced, and it increases linearly as the curriculum stage advances. Given the original source-language sentence and the word relevance score, the code-switcher selects the words in ascending order of word relevance. After that, a target language is chosen randomly based on a bilingual dictionary. It's important to note that source-language words can have multiple translations in the target language. In this case, one of the multiple translations is randomly selected as the target word. While this selection might not guarantee an exact word-to-word translation within the context, we have observed that this scenario is infrequent within most bilingual dictionaries. Consequently, the randomness introduced by this process has a minimal impact on our code-switching.

3.3 Scheduler

The scheduler aims to sample the data and send it to the model trainer for training. The scheduler decides when to sample the harder training data with the training progress. For our PCS, we begin with a temperature of τ = 0 equivalent to sampling the source language data. Then, the temperature linearly increases by the increment δ (e.g. δ = 0.1) every time the validation loss convergence, up to a final temperature of τ = 1. As the temperature increases, we will generate harder code-switching data. To encourage the model to pay more attention to harder data, we set larger early stopping patience for harder curricula than easier ones. However, we found that training the model on a sequence of CS datasets faces the problem of catastrophic forgetting [Kirkpatrick et al., 2017]. As the curriculum stage progresses, code-switching training datasets with varying augmentation levels are sequentially inputted into the model. This results in the modification of weights acquired during the initial curriculum once the model encounters the target of the new curriculum, causing the occurrence of catastrophic forgetting. To mitigate this problem, we design a dynamic curriculum scheduler for the model to revisit previous curricula. Specifically, at the k-th curriculum stage, the scheduler selects the code-switching data D(i) CS for the model training on the following probability:

3.4 Model Trainer

The model trainer progressively trains downstream taskspecific models with the training data given by the scheduler. And the model has the same network architecture as the pretrained model in difficulty measurer. We use the conventional fine-tuning method as proposed in the [Devlin et al., 2019]. Specifically, we use a pre-trained multilingual model as an encoder to obtain the representation. Then model f predicts task-specific probability distributions, and we define the loss of cross-lingual fine-tuning as

where G(x) denotes the ground-truth label of example x, l(·, ·) is the loss function depending on the downstream task.

4 Experiments

4.1 Setup

Tasks and Datasets. To comprehensively evaluate our proposed method, we conduct experiments on three types of cross-lingual transfer tasks with three widely used datasets. (1) For paraphrase identification, we employ PAWS-X dataset [Yang et al., 2019] containing seven languages. The label has two possible values: 0 indicates the pair has a different meaning, while 1 indicates the pair is a paraphrase. The evaluation is the classification accuracy (ACC). (2) For document classification, we employ MLDoc [Schwenk and Li, 2018] as our document classification dataset, including seven different target languages. The evaluation is the classification accuracy (ACC). (3) For spoken language understanding, we use the cross-lingual task-oriented dialogue dataset (XTOD) [Schuster et al., 2019] including English, Spanish, and Thai across three domains. The corpus includes 12 intent types and 11 slot types, and the model has to detect the intent of the user utterance and conduct slot filling for each word of the utterance. The performance of intent detection is evaluated using classification accuracy (ACC), while slot filling can be stated as a sequence labelling task evaluated on the F1 score. The statistics of datasets are summarized in Table 1.

Implementation Details. We implement our proposed method based on mBERT and XLM-R-large of HuggingFace Transformer 1 as the backbone model. We set our hyperparameters empirically following previous approaches [Liu et al., 2020; Qin et al., 2021; Lee et al., 2021] with some modifications. We set the batch size to 16 or 64, the maximum sequence length to 128, and the dropout rate to 0.1, and we use AdamW as the optimizer. We select the best learning rate from {5e-6, 1e-5} for the encoder and {1e-3, 1e-5} for the task-specific network layer. As for the scheduler, we initialize τ = 0, which linearly increases as the stage increases. We conduct each experiment 3 times with different random seeds and report the average results of 3 run experiments. To imitate the zero-shot cross-lingual setting, we consider English as the source language and others as the target languages. The bilingual dictionaries we use for code-switching are from MUSE [Conneau et al., 2017]. All models are trained on a single Tesla V100 32GB GPU.

1 https://github.com/huggingface/transformers.

4.2 Performance Comparison

We compare our method with the following competitive codeswitching enhancement models as our baselines:

mBERT [Devlin et al., 2019] is a 12-layer transformer model pre-trained on the Wikipedias of 104 languages and is fine-tuned only on the labelled source language training data.

XLM-R [Conneau et al., 2020] is a 24-layer transformerbased multilingual masked language model pre-trained on a text in 100 languages around 2.5TB unlabeled text data extracted from CommonCrawl datasets.

MLT [Liu et al., 2020] chooses source keywords based on the attention scores computed by a trained source language task-related model to generate code-switching sentences.

CoSDA [Qin et al., 2021] generates code-switching data with an empirically constant token replacement ratio to enhance the multilingual representations.

WS [Lee et al., 2021] simply substitutes words in sentences in every batch during training.

SCOPA [Lee et al., 2021] softly mixups the source word embeddings and the switched target word embeddings with an auxiliary pairwise alignment objective.

M-BoE [Nishikawa et al., 2022] only mixups the embeddings of Wikipedia entities to boost the performance of crosslingual text classification.

HCLD [Ma et al., 2022] classifies pre-defined intent with code-switching augmentation and then fills the slots under the guidance of intent.

SALT [Wang et al., 2023a] incorporates masked language modelling-based offline code-switching and online embedding mixup to enhance the cross-lingual transferability.

IECC [Ji et al., 2023] proposes an isotropy enhancement and constrained code-switching method for cross-lingual transfer to alleviate the problem of misalignment.

TCS [Lu et al., 2023] encourages cross-lingual interactions via performing token-level code-switched masked language modelling.

SCS [Lu et al., 2023] further proposes a semantic-level code-switched masked language modelling based on multiple semantically similar switched tokens in different languages.

Macular [Wang et al., 2023b] incorporates code-switching augmentation into multi-task learning to capture the common knowledge across tasks and languages.

As shown in Tables 2, 3 and 4, compared with strong codeswitching baselines, PCS shows its superiority and generality across different backbones and tasks at the zero-shot setting. In MLDoc and XTOD, we implement the XLM-R baseline following the reported settings of XLM-R [Conneau et al., 2020]. In Table 2, PCS outperforms SCOPA by 0.5% based on mBERT, and outperforms IECC by 0.5% based on XLMR. For Table 3, PCS outperforms SCOPA by 1.1% based on mBERT, and outperforms our reproduced XLM-R by 0.7%. In Table 4, compared to random selection (CoSDA) or solely choosing keywords (MLT) to construct code-switching sentences, our approach demonstrates superior performance.

4.3 Ablation Study

To better understand PCS, we conduct ablation studies to analyse the contributions of each component. Table 5 presents the ablation study results for our PCS on PAWSX. Comparing the full model, we can draw several conclusions: (1) We remove our dynamic scheduler in PCS for the variant w/o scheduler. Results show that our dynamic curriculum selection effectively alleviates the problem of catastrophic forgetting. (2) We remove the curriculum learning strategy for the variant w/o CL, which degrades into the random code-switching model. Results demonstrate that the usage of PCS pushes code-switching by an absolute gain of 1.5% on average. For (3), we use the word replacement ratio as the difficulty measure in PCS. Results show that the performance drops about 0.9% on average because the word replacement ratio cannot flexibly measure the difficulty of a code-switching sentence for different tasks. For (4), as an alternative difficulty measurer in our study, we employ a gradient-based explanation model. Results show that the performance slightly drops about 0.3% on average. This means our LRP-based difficulty measure is superior to the gradientbased method. For (5), we test anti-curriculum learning, which progressively generates code-switching data from hard to easy. Results show that the performance drops about 0.6% on average. This indicates that introducing hard data in the early curriculum negatively impacts learning performance. For (6), we generate code-switching sentences that only mix one target language with the source language. The performance of each language is more degraded than the PCS. This indicates that mixing multiple languages in code-switching can enhance the model's cross-lingual transfer ability.

4.4 Case Study

By analyzing the typical cases bettered by PCS, we seek to shed light on the underlying reasons behind its success.

Firstly, PCS can improve the quality of multilingual word representations. Specifically, by leveraging the high-quality word alignment obtained from easier code-switching sentences and their original counterparts, the model gains valuable pivots to comprehend words within over-replaced codeswitching sentences. As illustrated in Figure 3, the representations of "specific" and "also" exhibit a relatively low similarity score with their corresponding code-switched words in the first two rows. This is because the model without code-switching (w/o CS) and the code-switching model (CS) simultaneously consider all words within the over-replaced code-switching sentence. On the other hand, our Progressive Code-Switching (PCS) provides a higher similarity score for these word pairs. This is because PCS incorporates other word pairs aligned in the early curricula, allowing it to understand the over-replaced code-switching sentence.

Moreover, PCS helps the model focus on task-relevant keywords, enabling accurate predictions. We calculate the relevance scores of each word for the prediction result, and we notice that PCS makes correct predictions and provides understandable justifications. As illustrated in Table 6, PCS correctly predicts that the sentence pair is semantically different by focusing on the discriminative words ("este", "Haryana", "oeste" and "Haryana"). In contrast, the model without code-switching (w/o CS) and the code-switching model (CS) fail to do so. We conjecture that our PCS tends to better understand task-related keywords due to its learning process in the later stages of the curriculum, during which the model has already acquired some multilingual knowledge. This further confirms the effectiveness and generalization of the proposed PCS for different tasks.

4.5 Learning Curve Analysis

To assess the effectiveness of the progressive code-switching method, we compare the learning curves of our LRP-based PCS with the ratio-based progressive code-switching, the random code-switching and the vanilla fine-tuned mBERT (without code-switching). In Figure 4, we draw the average accuracy and the standard deviation of all target languages based on three experimental runs during the model training updates on PAWS-X. Firstly, all three code-switching models demonstrate significantly improved performance compared to the vanilla mBERT baseline. This indicates that introducing code-switching enhances the model's cross-lingual capabilities. Secondly, both progressive CS models converge to a better solution than the traditional random CS model using the same number of updates. This means that progressive codeswitching facilitates the model to converge rapidly to a better minimum. Thirdly, the ratio-based PCS exhibits instability at 15-th and 35-th updates, unlike our LRP-based PCS. This instability arises from the sub-optimal static word replacement rate difficulty measurer, which cannot guarantee that the generated code-switching data satisfies the nature from easy to difficult for various tasks. In contrast, our LRP-based model solves this problem by dynamically generating codeswitching data guided by the token-level relevance scores derived from the trained task-related model.

4.6 Multilingual Representation Visualization

To examine the alignment of multiple languages, we compare the alignment results of vanilla fine-tuned mBERT and our PCS in terms of sentence representation. We select sentences represented by seven different languages from the PAWS-X datasets respectively to visualize the embedding space. As shown in Figure 5 (a), different languages are distributed in different positions in the embedding space, which indicates that fine-tuned mBERT can distinguish them easily. In contrast, Figure 5 (b) shows that the data distributions of different languages are mixed and overlap each other, which indicates that our model induces language-independent features and boosts the multilingual representation alignment.

5 Conclusion

This paper proposes progressive code-switching, which fully mines multilingual knowledge to enhance zero-shot crosslingual performance. We first adopt a word relevance score calculation method to measure the difficulty of the codeswitching data. Then we generate suitable code-switching data controlled by the adoptable temperature. Finally, we introduce a scheduler to decide when to sample harder data for model training. Experimental results on the three zero-shot cross-lingual tasks covering ten languages exhibit the effectiveness and potential of our proposed method.

Acknowledgements

We thank Beijing Advanced Innovation Center for Big Data and Brain Computing for providing computation resources. We also thank the anonymous reviewers and the area chair for their insightful comments. This research was supported by the Shanxi Province Special Support for Science and Technology Cooperation and Exchange (202204041101020), Key Laboratory of Key Technologies of Major Comprehensive Guarantee of Food Safety for State Market Regulation (BJSJYKFKT202302), and the Zhejiang Provincial Natural Science Foundation of China under Grant (LGG22F020043).