GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost

1 Introduction

In recent years, NLP has welcomed great advances and significant progress in deep learning [Vaswani et al., 2017; Devlin et al., 2019; Brown et al., 2020]. Pre-trained language models (PLMs) trained on extensive text have significantly improved performance on various core NLP tasks, bringing NLP new research paradigms such as "pre-training and fine-tuning" and "prompt-based learning" [Liu et al., 2022]. However, despite PLMs' extraordinary performance, training these models both incurs substantial computational costs and relies upon the availability of extensive training data. The former has raised public concern about substantial energy consumption and environmental damage [Strubell et al., 2020], while the latter inevitably creates unequal opportunities for NLP research across languages with differing access to resources [Joshi et al., 2020].

Computation costs for training a single, state-of-the-art deep learning model expanded 300,000 times between 2012 and 2018, a trend which has only intensified since; this presents a challenge for researchers interested in upholding green AI principles [Schwartz et al., 2020] as well as broader public policy ambitions like the United Nations Sustainable Development Goals (SDG) [Assembly, 2015]. Figure 1 shows a comparison of a number of representative PLMs and multimodal models from BERT-base [Peters et al., 2018] to GPT-4 [OpenAI, 2023], where we observe trends of ever-increasing model size, computational costs, economic costs, and environmental costs [Strubell et al., 2019]. Pretraining a BERT-base model incurs cloud computing costs of $3,751 to $12,571 [Schwartz et al., 2020], which is unaffordable for personal use and most researchers in developing countries. Training models such as BERT, T5 [Raffel et al., 2020], PaLM, and GPT-3 [Brown et al., 2020] from scratch emit 0.65, 46.7, 271.43, and 552.1 tons of CO2, respectively.

Beyond economic and environmental costs, the pretraining of large language models requires vast amounts of monolingual text data, which is lacking in many low-resource languages, leading to increased disparities in the utilization of language models among different languages. Languages with a larger number of speakers inherently have larger corpora available for PLM training. This leaves only a tiny fraction of the approximately 7000 languages worldwide with sufficient data to train a monolingual PLM [Ebrahimi and Kann, 2021]. Table 1 lists the sizes of training materials for the monolingual BERT of several languages, exhibiting a significant inequality in monolingual materials. For instance, English GPT-3 was trained on 45 TB of monolingual material, equivalent to 4,608 times size of the Hindi BERT pretraining corpus. This basic inequality in data availability is further exacerbated by many social and financial factors, such Date as internet infrastructure, education, and computational resources available in low-resource speech communities. Selfsupervised pre-training thus enlarges the disparity between high- and low-resource languages, with the result that breakthrough technologies in contemporary NLP have proven particularly difficult to share equally among speakers of all the world's languages.

Figure 1: Trends of Complexity and Training Cost in Mainstream PLMs with Significant Performance Boosts. The icons represent the number of parameters, training corpus size, financial cost by dollars, and the emission of CO2, respectively. ? represents numbers that can not be estimated.

To contribute to ongoing efforts to mitigate these issues, we propose a generalizable framework, GreenPLM, to "translate" monolingual PLMs to new languages at almost no additional cost. We hypothesize that the linguistic knowledge learned by PLMs on large monolingual corpora is transferable to low-resource languages, encouraged by two primary pieces of evidence. First, monolingual word embeddings are partially isomorphic across languages [Artetxe et al., 2016]; etymologically close language pairs show good word translation performance [Conneau et al., 2017]. Second, work on the capacities of monolingual PLMs [Conneau et al., 2020b; Blevins and Zettlemoyer, 2022] has shown that they encode cross-linguistically useful information even without training in further languages. Therefore, GreenPLM utilizes bilingual lexicons to bridge cross-lingual semantic spaces while keeping the well-trained parameters of high-resource PLMs fixed as shown in Figure 2. Our experimental results show that GreenPLM's performance is comparable to or better than current heuristics in 18 tested languages, at nearly zero cost. In addition, with around 0.5% computational cost compared to pre-training a BERT model from scratch, continued pretraining of these models (which we call GreenPLM+) outperforms the performance of monolingual PLMs. To sum up, our contributions to AI and social good are the following:

1. We propose a simple, heuristic pipeline utilizing bilingual lexicons to translate a source language PLM to a target language PLM with almost no computational cost, significantly reducing carbon emissions for building foundation PLMs in various languages;
2. We verify the effectiveness and generalizability of this framework with evaluations in 18 languages, showing performance comparable to monolingual and multilingual PLMs;
3. We propose these models can be further enhanced with continued pre-training in 7 languages and show performance exceeding existing monolingual and multilingual PLMs while reducing pre-training costs up to 200x compared to pre-training from scratch;
4. We suggest this approach can be used to reduce language disparities and energy consumption in pre-training LMs, aligned with the SDG and the LNOB principle to promote the application of AI for social good.

2 RelatedWork

2.1 Cost Reduction in PLMTraining

As we've explored, training large PLMs inevitably requires extensive computational resources, which attracted many studies on efficient PLM pre-training. For example, [?] proposed a task-driven language modeling framework to pretrain task-specific PLMs that required only around 2% of the original computational costs. An alternative approach explored by [Chen et al., 2022] and [Qin et al., 2022] leveraged a small "teacher PLM" to pre-train a large "student PLM" at a low computational cost. Reducing data requirements remains less studied compared to the former. The issue of data requirements has been less studied, though recently [Warstadt et al., 2023] put forth a BabyLM challenge to call for sampleefficient pre-training. Our work addresses both these issues, focusing on simultaneously reducing computational costs and data requirements.

2.2 Language Disparity in PLMs

Researchers have proposed numerous mechanisms to equalize performance across human languages in PLMs [Muller et al., 2021; Ebrahimi and Kann, 2021], but the democratization of language technology remains a major challenge [Bird, 2020]. Multilingual BERT (mBERT) represents a preliminary attempt to adapt PLMs to multiple languages and is pretrained on the concatenation of corpora from different languages [Devlin et al., 2019]. Such models can achieve substantial coverage; for example, mBERT and XLM-R [Conneau et al., 2020a] cover 104 and 100 languages respectively. Nevertheless, model performances in specific languages are reported to be highly dependent on pre-training corpus sizes, and cross-lingual PLMs trained in this manner are still expensive to train and ultimately rely upon the availability of monolingual texts in a given target language.

2.3 Cross-Lingual Transfer

After multilingual BERT showed impressive performance in zero-shot cross-language transfer settings [Pires et al., 2019], substantial attempts have been made to adapt monolingual or multilingual PLMs to other languages.

[Wu and Dredze, 2020] examined the performance of mBERT across various languages and reported that the model performs well for high-resource languages but not for lowresource languages. Nonetheless, in low-resource settings, mBERT performs better than monolingual BERT, validating some degree of cross-language generalizability. [Muller et al., 2021] explored mBERT's performance in never-beforeseen languages, finding unstable performance. They found that mBERT has unstable performance across languages, depending on the training data. [Ebrahimi and Kann, 2021] used the New Testament, which is available in most languages, to adapt multilingual PLMs to more than 1600 languages using continued pre-training, vocabulary extension, and adapter transformers. [Wang et al., 2022] explored how bilingual lexicons can enhance the multilingual language coverage of PLMs. Their results had consistent improvement across sequence labeling and dependency parsing tasks in 19 low-resource languages. In contrast, our current study extends bilingual lexicons to monolingual settings as a bridge rather than data augmentation components to design specified monolingual PLMs equally for all languages.

Monolingual adaptation remains relatively under-explored compared to multilingual. It follows two general directions: extending monolingual PLMs to bilingual and adapting a source language PLM to a target language PLM. In the first direction, [Artetxe et al., 2020] continually pre-trained English PLMs on data in target languages with the transformer body frozen and fine-tuned the PLMs on labeled English data. Their results confirmed the generalizability of monolingual PLMs across languages. [Tran, 2020] transferred PLMs by initializing foreign language embeddings in the English vector space and jointly fine-tuning both models. For the second direction, [de Vries and Nissim, 2021], and [Minixhofer et al., 2022] used word embeddings as the bridge to adapt monolingual models to other languages with continued pretraining. Motivated by these works, we leverage more widely available and less computationally expensive resources other than word embeddings to generalize PLMs.

3 Methodology

3.1 Overall Architecture

An overview of the GreenPLM framework is shown in Figure 2. We start from a bilingual lexicon for a given sourcetarget language pair and the selection of a source PLM. Importantly, contemporary PLMs require a pre-established vocabulary embedding layer which operates on the outputs of a specific tokenizer. Tokens in the vocabulary are a mix of both full words and word pieces, traditionally the outputs of an algorithm like WordPiece or byte pair encoding, where each token is mapped to an embedding learned in the pre-training process.

In our approach, we augment the vocabulary and embedding layer of the source PLM to incorporate target-language vocabulary from the bilingual lexicon, using a method to map this lexicon into a shared semantic space explained below in subsection 3.2. This results in what we call a "translated PLM," though notably the embedding layer still contains the entire vocabulary from the source language and the PLM encoder parameters are yet unchanged from the source PLM. Such a model can already be applied directly to targetlanguage data; in experiments, we call this setting simply "GreenPLM".

The translated PLM can also be fine-tuned towards particular tasks in the target language, or pre-trained further on target language data, both of which modify the PLM parameters to be more suited to the characteristics of the target language. We name this setting "GreenPLM+".

3.2 LexiconWalkMapping

The biggest challenge in the approach outlined above is how to use a bilingual lexicon to generate coherent target-language embeddings that can be fed into source-language PLM encoder parameters. To achieve this we propose a heuristic method we call Lexicon Walk Mapping (LWM), in which a bilingual lexicon is used as the bridge to create a mapping between the semantic spaces of the source and target languages.

We define a bilingual lexicon as L = {xi, yi} where xi is a word or phrase in the source language (English, in this work) and yi is xi's translation in the target language. Then given a source language's PLM S with its vocabulary Svocab and token embeddings Sembedding, we walk through the lexicon and operate on each entry separately. For each entry, we first tokenize the source word or phrase xi into a set of tokens T = {t1, t2, ...ti...tn | ti ∈ Svocab}, and then map T to the token embedding layer of S to retrieve a set of embeddings E = {e1, e2, ...ei...en | ei ∈ Sembedding}. We then take the dimension-wise average of E, and the new embedding eaverage serves as yi's token embedding in the GreenPLM. If yi repeats in the lexicon, as is frequently the case in one-to-many translations across languages, the GreenPLM framework accumulates several instances of eaverage and in turn averages them at the end of the procedure to generate a final token embedding.

Our approach leaves the original token embeddings of source language words, tokenized word pieces, and special characters in Svocab untouched in the translated PLM; instead, we merely expand the vocabulary size by adding unseen target language words in the bilingual lexicon. For example in Spanish, the GreenPLM has a vocabulary size of 119,999 words, while the corresponding Spanish monolingual BERT only has 31,002.

3.3 Continued Pre-training

Continued pre-training has long been validated as one of the most effective strategies to enhance PLMs, with several possible strategies including masked language modeling (MLM) [Devlin et al., 2019] and translation language modeling (TLM) [Conneau and Lample, 2019]. In addition, some special techniques, such as contrastive learning, can be combined with pre-training tasks. Here we use MLM as the basic pre-training task in our continued pre-training experiments. To generate a GreenPLM+ model for each language, the translated PLM is further pre-trained with a batch size of 1024 and a sequence length of 128 symbols by utilizing 10% of the OSCAR dataset [Ortiz Su´arez et al., 2019]. Each model was pre-trained for 5 epochs and every 10,000 pre-training steps were saved.

The experiments presented here examine GreenPLM+ models in seven languages: Spanish, Indonesian, Tagalog, Turkish, Italian, Romanian, and Persian because they are the only languages with a moderate amount of language resources. An attempt to further pre-train a GreenPLM for Maltese fails because the available resources (17 MB in the OSCAR corpus) are too scarce to maintain training stability.

3.4 Alternatives to LWM

We found that LWM worked most effectively as a method to bridge the embedding spaces between models, and therefore present experimental results using that method. However, we also tried several other logical alternatives that were not as effective. We describe those methods here for completeness and report detailed results of these methods in Spanish, Turkish, and Tagalog in the supplementary material. Because these methods are strictly worse, we did not further experiment on other languages.

Vocabulary Expansion (VE) refers to expanding the vocabulary based on the vocabulary of the source language PLM. Given a bilingual lexicon lexicon ={xi, yi} and the PLM S of the source language with a vocabulary of Svocab = {w1, w2, ..., wi, ..., wn} and token embeddings Sembedding , VE will walk through Svocab and find their corresponding embeddings by Sembedding and lexicon. If no matching entries exist, this strategy will keep the original tokens. If there is one yi corresponding to multiple xi, VE will keep the first xi's embedding as the translated embedding. Then, VE will merge all Wi as the vocabulary of the new model, where translated embeddings are used for the newly extended vocabulary.

Vocabulary One-on-one Mapping (VOM) adopts a similar methodology to VE but only uses one translation for multiple entries. While both LWM and VE increase the vocabulary sizes, introducing more parameters, VOM yields a GreenPLM with the same vocabulary sizes.

Vocabulary Translation Mapping (VTM) is the most basic strategy being tested. It cuts sentences according to space delimitation and uses bilingual lexicons to directly transfer the tokens into another language. Then, the newly generated sentences are directly fed to the source language PLM for downstream tasks.

4 Materials and Evaluation

4.1 Language andModel Choices

English was chosen as a source language since it has the largest pre-training corpus size and the largest body of existing PLMs available. For target languages, we included 18 languages from five language families (Table 1). According to [Joshi et al., 2020], the chosen languages fall under various language resources' levels ranging from the extremely lowresource level "1 - The Scraping-Bys" to the high-resource level "5 - The Winners".

In these experiments, we used the bert-base-uncased English model as the source PLM, since it has the broadest coverage in academic and industrial usage. Since the model is relatively lightweight it also allows for extensive experiments on various NLP tasks across languages, even with limited computational resources. Nevertheless, the GreenPLM framework is independent of the particular structure of PLMs beyond the embedding layer, and hence can theoretically be applied to larger models to benefit from continuing advances in large-scale PLMs.

4.2 Experimental Settings

Bilingual Lexicons

This study uses bilingual lexicons from four sources. For relatively high-resource languages, our lexicons are fromMUSE [Conneau et al., 2017], PanLex [Kamholz et al., 2014], and OPUS [Tiedemann and Nygaard, 2004]. For Maltese, M¯aori, Wolof, Luganda, Ilokano, and Hausa, we used lexicons released from [?] directly. Detailed statistics of the lexicons could be accessed in the supplementary material.

Evaluation

Most chosen languages have available benchmark datasets for performance comparison. For example, [Ca˜nete et al., 2020] as the Spanish benchmark; IndoLEM in [Koto et al., 2020] for Indonesian; [Schweter, 2020a], [Cruz and Cheng, 2019; Cruz and Cheng, 2020; Cruz et al., 2020], [Schweter, 2020b], [Dumitrescu et al., 2020] and [Farahani et al., 2021] for Turkish, Tagalog, Italian, Romanian, and Persian, respectively. For the 11 low-resource languages with only NER and POS tasks, GreenPLM was tested on MasakhaNER [Adelani et al., 2021] and universal dependencies [de Marneffe et al., 2021]. For benchmark datasets with multiple tasks, we divided the tasks into three main categories: POS, NER, and text classification for a by-category analysis of our GreenPLM framework. Specifically, for continued pre-training, this study uses batch size × sequence length × training steps as a metric to evaluate pre-training efforts.

Our translated PLMs were evaluated by performance comparison with existing models on various NLP tasks, including part-of-speech tagging (POS), named entity recognition (NER), sentiment analysis, sentence classification, and paraphrase identification. We used the F1 score for the NER task, accuracy in POS and most classification tasks, Pearson's coefficient in the tweet ordering task of Indonesian, and hamming loss in the Dengue task of Tagalog. Specifically, We re-scaled rare metrics like hamming loss and Pearson's coefficient to fit a 0 to 100 scale and took the average of model performance for a direct comparison.

Baseline Methods

To compare the performance of GreenPLM with existing models, four deep learning models were chosen as baselines, and fine-tuned towards each evaluation task: (1) Scratch: a randomly initialized BERT model; (2) Pre-BERT SOTA: state-of-the-art models before BERT, i.e. long short-term memory (LSTM) and convolutional neural networks (CNNs) with pre-trained word embeddings, which we implement using NCRF++ [Yang and Zhang, 2018]; (3) mBERT: multilingual BERT [Devlin et al., 2019]; and (4) monolingual BERT models, where available. In addition, GreenPLM+, the advanced version of GreenPLM with a continued pre-training on a very limited dataset, was also compared with the above models. Only uncased models were tested in this study because most bilingual lexicons are uncased.

5 Results

Our experiments test baseline, GreenPLM, and GreenPLM+ models on 18 languages across various categories of tasks listed in Table 2. Detailed results on each specific task could be accessed in the supplementary material.

5.1 Base GreenPLM

In the base setting where models are translated using only LWM, we observe that the GreenPLM performs much better than random BERT in all cases, and better than Pre-BERT SOTA models in 15 out of 18 languages, with the exception of sequence labeling tasks in 3 languages. This suggests that indeed, the bilingual lexicon based approach we take with GreenPLM is not merely successful due to taskspecific fine-tuning, but rather achieves performance gains due to cross-linguistic knowledge transfer from the source PLM. Though GreenPLM generally does not surpass mBERT in performance, our method provides at least comparable performance on the examined languages at nearly no cost, whereas mBERT itself requires large multilingual training costs.

Moreover, in some language settings our method is surprisingly effective. In Tagalog, the GreenPLMwins over all baselines and shows a large improvement over the monolingual BERT. This may imply that the original Tagalog BERT was not pre-trained with sufficient data. In the low-resource settings of Maltese and Ilokano, the GreenPLM also outperforms all existing baselines.

5.2 GreenPLM+: Continued Pre-training

As a further enhancement to the GreenPLM, we applied a lightweight continued pre-training step with very limited training cost and evaluated the results in the seven languages mentioned above with access to sufficiently-sized pretraining corpora. In the last column of Table 2, we evaluate GreenPLM+ models at the first point at which they outperform mBERT or monolingual BERT for the first time during the continued pre-training, and note in parens the percentage of pre-training efforts required to reach this level of performance compared to pre-training a PLM from scratch. We observe that GreenPLM+ outperforms all monolingual BERT at a relatively low cost, except in Spanish. For example, GreenPLM+ outperforms the original PLM with only 0.68% of the pre-training efforts for Indonesian. For Spanish, GreenPLM+ outperforms multilingual BERT and achieves comparable performance over the monolingual BERT when given 0.7% of the pre-training efforts. We further compared continuously pre-training GreenPLM and mBERT under the same settings in Indonesian, Romanian, and Persian. GreenPLM could catch up and outperform mBERT given minimal costs. In Figure 3(b), we show a visualized comparison of Persian.

6 Discussion

Impact on SustainableDevelopment. High computational requirements for pre-training LMs or even pre-BERT deep learning models have been a major obstacle for researchers, where a slight performance boost may come at the expense of dramatically increased model complexity. Our approach generates translated PLMs for medium-level and low-level resource languages that successfully outperform pre-BERT SOTA deep learning models at extremely low pre-training costs, both in terms of economic costs and CO2 emissions. In contrast to prior approaches for low-cost training of PLMs [Yao et al., 2022], the GreenPLM framework is task-agnostic All computation for PLM translation in the basic GreenPLM approach can be executed on a personal computer within 10 seconds. For continued pre-training with the GreenPLM+

model, we demonstrate comparable results to a fully pretrained monolingual model using less than 10% of the dataset size in less than 8 hours on an 8x RTX3090 system.

Impacts on Language Equality. Unequal language resources have limited recent advances in PLMs from being equally shared by speakers of all the world's languages. While developing larger corpora for pre-training is not always feasible, here we leverage a more widely available language resource - bilingual lexicons - to translate PLMs from higher- to lower-resource settings. This framework can serve as a basic starting point to transfer models across languages and reduce both language-based and economic disparities in the availability of PLMs in different linguistic and socioeconomic backgrounds.

Experimental results show that GreenPLM+ performs better than existing monolingual and multilingual models in languages with small but still substantial pre-training corpus sizes, such as Indonesian, Turkish, and Tagalog. These languages are rated as "4 - Underdogs" or "3 - Rising Stars" in the [Joshi et al., 2020] classification of resource availability. We note with an example from Persian in Figure 3 (b) that in these settings, the base GreenPLM model commonly starts off with lower performance than mBERT, but the GreenPLM+ quickly catches up after continued pre-training with the same number of steps.

For languages with fewer resources in the '2 - Hopefuls' and '1 - Scraping-Bys' categories by [Joshi et al., 2020], where substantial pre-training corpora are not available for continued pre-training, we find that mBERT retains better performance than the base GreenPLM. Nevertheless, the simple and nearly zero-cost transfer method presented here achieves performance generally surpassing Pre-BERT SOTA models, and comparable to mBERT, suggesting that base GreenPLM models can be productively used to transfer PLMs to languages where only bilingual lexicons exist.

Applicability for Downstream Tasks. The evaluations in this work examined downstream tasks in the categories of POS tagging, NER, and classification tasks, corresponding to word-level, span-level, and sentence-level understanding. Results in Figure 3(a) show that the GreenPLM, compared to random BERT models, has the most remarkable improvement on POS tasks, followed by NER and classification tasks. However, compared to mBERT, GreenPLM falls short in POS, followed by NER and classification tasks. This suggests that GreenPLM transfers sentence-level understanding the best in downstream tasks, possibly because embeddingbased strategies of our framework focus mainly on semantic information and lack the ability to capture the variability of syntactic information in natural languages.

Language Distance. In addition, we use a language distance calculator from eLinguistics 1 as a reference and quantify the performance by the error reduction rate (ERR) metric because languages are classified according to their diachronic relatedness in linguistic typology. Corresponding results in Figure 3(c) show that the closer the selected language is to English, the better its GreenPLM will perform. This is not surprising since GreenPLMmodels retain the parameters from the original English BERT being translated; nevertheless, it suggests that linguistic distance is likely to be a crucial consideration in cross-lingual knowledge transfer, supporting existing findings [Lin et al., 2019].

Future Work. Though this paper uses English as a source language and starts from an English PLM to generate target language PLMs, a similar approach could be used to merge PLMs frommultiple languages for cross-enhancement to generate a single target language PLM. In addition, GreenPLM could be extended to multi-modal and multi-domain settings where bilingual dictionaries could facilitate information exchange; for example, could a similar approach help integrate biomedical information using a targeted lexicon? Finally, it remains to investigate how the GreenPLM approach performs with more state-of-the-art BERT-class PLMs such as RoBERTa, as well as with PLMs of different architectures such as BART [Lewis et al., 2020].

1 http://www.elinguistics.net/Compare Languages.aspx

Limitations. This work has two main limitations. First, it only covers uncased PLMs at the current stage because most bilingual lexicons are uncased, and capitalization could affect model performance. Second, there is no mechanism to rank the importance of single entries in bilingual lexicons. Therefore, newly generated models generally have a more extensive vocabulary than the original PLM.

7 Conclusion

This paper proposes a bilingual lexicon-based framework GreenPLM that "translates" a monolingual PLM to other languages with no additional cost. We verify its effectiveness in 18 languages and further pre-train the resulting models to validate their performance when given minimal pre-training costs. Our results provide evidence that the GreenPLM+ (GreenPLM with very limited continued pre-training) outperforms monolingual BERT at much lower computational costs than current frameworks, such as the continued pre-training of mBERT. This framework is easy to incorporate into existing PLMs and can be adapted to various languages given the relatively wide availability of bilingual lexicons. We hope that this work can make a positive impact on promoting green AI and reducing language disparities in access to contemporary NLP technologies.

Acknowledgments

This research was supported by the National Natural Science Foundation of China (No. 62206243).

Contribution Statement

Qingcheng Zeng, Lucas Garay, and Peilin Zhou contributed equally to this paper. Correspondence to Jie Yang.