Unsupervised Context Aware Sentence Representation Pretraining for Multi-lingual Dense Retrieval

1 Introduction

Nowadays, cross-lingual pre-training [Devlin et al., 2019; Conneau and Lample, 2019; Huang et al., 2019; Conneau et al., 2020] has achieved great performance on cross-lingual transfer learning tasks. These pre-trained models could fine-tune on one language and directly test on other languages. Cross-lingual sentence representation models like InfoXLM [Chi et al., 2021] and LaBSE [Feng et al., 2020] target to generate good cross-lingual representation without fine-tuning. These models use contrastive loss to make bilingual sentence pairs have similar embeddings and achieve great performance on bilingual sentence retrieval tasks.

However, there are two potential problems for these methods. First, most of them rely on bilingual corpus, which are not always available, especially for low-resource languages and non-English languages pairs. Only using English related bilingual pairs will limit the transferability between non-English pairs. Second, multilingual dense retrieval tasks such as XOR Retrieve and Mr.TYDI require the model to map semantic related query and passage to similar position in embedding space, but existed methods only could map bilingual sentence pairs with same meaning to similar embedding.

Inspired by recent progress of contrastive learning on dense retrieve, we propose a new pretraining task called Contrastive Context Preiction (CCP). CCP targets on constructing isomorphic embedding space by modeling the sentence level contextual relation in long documents. Formally, a document is a sequence of sentences. For each center sentence, we define the sentence in the window centered on it as a context sentence. First, CCP will encode each sentence into a vector. Given embedding of center sentence sc, the model need to select correct context sentence s out of thousands random sampled sentences and vice versa. With contrastive context prediction, we could estimate the mutual information I(s|sc) of contextual relation. Our experiments show that the embeddings of CCP have isomorphic structure across different languages. Furthermore, we do cross-lingual calibration to further improve the alignment. We illustrate our ideas with Figure 1, which visualizes the English and Arabic embedding space from XLM-R and our models. We randomly highlight five points for each languages and the sentences with same meaning have same color. The distribution of two languages shown in Figure 1(a) doesn't have an obvious pattern. With contrastive context prediction, the five points show similar relative position and the shape of all points are similar to each other, but the embeddings of two languages are spread in two regions of latent space. After cross-lingual calibration, the points with same color almost have similar position.

We evaluate our model on bilingual sentence retrieval task Tatoeba, which can test whether our model could generate similar embeddings for two sentences with same meaning but from different languages. Our model achieves SOTA results among methods without bilingual data, and our results are very close to the model with bilingual data. Besides, our model shows better cross-lingual transferability between non-English languages pairs. On two multi-lingual query-passage retrieval tasks, XOR Retrieve and Mr.TYDI, CCP achieves new SOTA results among all pretraining models using bilingual data.

Our contribution can be summarized as:

• We propose a contrastive context prediction pretraining (CCP) task that is capable of learning isomorphic representations for each language without parallel data. Our model can achieve excellent performance on multilingual retrieval, especially between two non Englishcentric languages-pairs.
• We design an effective contrastive pretraining framework for sentence embedding pretraining. It consists of language-specific memory bank and projection head with asymmetric batch normalization. Both of them play essential role in preventing collapsing. And we also observe the offset phenomenon on on bilingual sentence representation pair produced by our model. To align the sentence embedding space between different languages, we propose cross-lingual calibration to align the bilingual sentence pair into the same position in latent space.
• We conduct evaluation experiments upon three types of multi-lingual retrieval tasks. Extensive results on the three datasets have shown superiority of the proposed model in both effectiveness and rubustness.

2 RelatedWork

Cross-lingual Pre-trainedModel Multilingual BERT (MBERT) [Devlin et al., 2019] performs pre-training based on the multilingual corpus with the masked language model task. By sharing the model parameters and the vocabulary across all languages, M-BERT obtains the cross-lingual capability over 102 languages. XLM [Conneau and Lample, 2019] performs crosslingual pre-training based on multilingual corpus and bilingual corpus, by introducing the translation language model task into pre-training. Based on XLM, Unicoder [Huang et al., 2019] uses more cross-lingual pre-training tasks and achieves better results on XNLI. XLM-R [Conneau et al., 2020] is a RoBERTa [Liu et al., 2019]-version XLM without using translation language model in pre-training. It is trained based on a much larger multilingual corpus (i.e. Common Crawl) and becomes the new state-of-the-art. Both these models and our model could achieve good performance after fine-tuning. Our model also could produce good cross-lingual sentence embedding without fine-tuning.

Dense Passage Retrieval [Lee et al., 2019] proposed a simple Inverse Cloze Task (ICT) method to further continue-train BERT. REALM[Guu et al., 2020] is an end-to end co-training framework for reader and retriever. karpukhin-etal-2020-dense is the first to discover that careful fine-tuning can learn effective dense retriever directly from BERT. Later works then started to investigate ways to further improve fine-tuning. ANCE [Xiong et al., 2020] selects hard training negatives globally from the entire corpus, using an asynchronously updated ANN index. [Qu et al., 2021] proposed the RocketQA fine-tuning pipeline which hugely advanced the performance of dense retrievers. coCondenser [Gao and Callan, 2021] is one of the best dense passage retrieval model on MS-MARCO, Natural Question and Trivia QA. It adds an unsupervised corpus-level contrastive loss to warm up the passage embedding space.

Contrastive Learning CPC [Oord et al., 2018] predicts the future in latent space by using powerful auto-regressive models. It uses a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. [Wu et al., 2018] presents an unsupervised feature learning approach by maximizing distinction between instances via a novel non-parametric softmax formulation, which is so-called memory bank mechanism. SimCLR [Chen et al., 2020] simplifies recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. MoCo [He et al., 2020] maintains a dynamic dictionaries by implementing a momentum-based moving average mechanism of the query encoder. InfoXLM [Chi et al., 2021] contains a pre-training task based on contrastive learning. Given a bilingual sentence pair, they regard them as two views of the same meaning, and encourage their encoded representations to be more similar than the negative examples. LaBSE[Feng et al., 2020] combines masked language model (MLM) and translation language model (TLM)[Conneau and Lample, 2019] pretraining with a translation ranking task using bi-directional dual encoders. CLEAR [Wu et al., 2020] performs contrastive learning on for sentence representation. It employs multiple sentence-level augmentation strategies in order to learn a noiseinvariant sentence representation. We follow SimCLR on the detailed implementation of contrastive learning. Different from these works, we leverage contrastive learning to model the sentence-level contextual relation in natural languages.

3 Methodology

Our method contains two steps. First, we train the model with Masked Language Model and Contrastive Context Prediction. This step targets on building isotropic sentence embedding space for each language. The learned sentence embeddings from different languages will have good cross-lingual property after they became isomorphic by CCP task. Just as Figure 1(c) shows, different languages are spread in different region of latent space with similar structure. Thus in second step, we use cross-lingual calibration to further align different languages.

3.1 Contrastive Loss for Context Prediction

Our method targets to model the sentence-level contextual relation. Formally, a document D is a sequence of sentences (s1, s2, ..., sl). For each center sentence sc, we define its contextual set as Context(sc) = {sp|c-w <= p <= c+w, p ̸= c}, w is radius of window which represents the maximum distance between center sentence and context sentence. Our model will model the relation between center sentence and its contextual sentences. In this subsection, we introduce the details of contrastive loss LN(si, sc).

Scoring Function The scoring function f(si, sc) takes two sentences as input and output a score. To begin with, we encode s and sc to a vector separately. In this step, we choose a Transformer-based encoder. We use the corresponding representation of a manually-inserted token [CLS] as the embedding of the whole sentence. Then, we add a non-linear neural network, namely Projection Head, to further map the vector to a new space. The projection head consists of two linear layers and one batch normalization between two layers. We denote the representation of si and sc as zi and zc respectively. Following SimCLR [Chen et al., 2020], we only use it when computing contrastive loss and abandon it after pre-training. Projection Head could help model learn general representations and will not overfit to contrastive loss. Finally, we choose scoring function as f (si, sc) = exp(cosine(zi, zc)/τ ), where τ represents temperature and is a hyper-parameter.

Language-specific Memory Bank In [Oord et al., 2018], the lower bound of contrastive loss becomes tighter as negative samples number N becomes larger. To further increase N while batch size is limited by GPU memory, we use memory bank to store the embeddings from recent batches and use them in the training of current batch. Because we are handling multiple languages at same time, we tried two strategies: plain memory bank and language-specific memory bank, and find later one is better. The "language-specific" means that our memory bank will tag the embeddings by language. For each language, it will only use the embeddings from the same language in training. We tried to use shared memory bank for all languages. Since a shared memory bank provides negative samples from various languages, the model will focus on classifying the language of sentences, rather than learning contextual relation. Hence the loss could be very small because language classification is very easy, but the cross-lingual performance will be very poor. The memory bank is maintained in FIFO (First-In-First-Out) manner. During each iteration, the representation zi as well as the network parameters θ are optimized via Adam. Then zi is added to M, and the oldest representation in the memory bank is deleted.

Asymmetric Batch Normalization Batch normalization is an essential part of projection head. However, traditional batch normalization is easy to be trapped in information leak problem [He et al., 2020], which means that the contrastive loss is very small but the evaluation results on down stream tasks are very low. Hence, we propose asymmetric batch normalization to avoid information leak. It is more efficient than shuffle batch normalization [He et al., 2020], which requires communication between GPUs. In training procedure, the mode of batch normalization will be changed in training mode and testing mode alternately and the mode of batch normalization in two projection heads should be kept different in training. In testing mode, the batch normalization will use running mean and running variance to replace batch mean and batch variance, which is able to prevent information leak. The detail training procedure can be seen in Algorithm 1. Compared with shuffle batch normalization proposed in [He et al., 2020], our method is very easy to implement and has good performance.

3.2 Cross-lingual Calibration

After we acquire isomorphic sentence embeddings by pretraining, in order to better align between languages, we further propose cross-lingual calibration for sentence representation.

Algorithm 1 The training algorithm for the contrastive context prediction task.

• 1: Input: Batch size N, constant τ, structure of f, g. 2: Output: Model parameters Θ. 2: for sampled minibatch {ck}NW do 3: draw center sentence sc from context cw. 4: randomly draw target sentence si from context cw. 5: hc=f(sc); hi=f(si); 6: if training mode e = 0 then 7: gc() = g().train(); gi() = g().eval(); 8: zc=gc(hc); zi=gi(hi); e = 1; 9: else 10: gc() = g().eval(); gi() = g().train(); 11: zc=gc(hc); zi=gi(hi); e = 0; 12: end if 13: lw c,i = - log exp(cos(zc ,zi )/τ ) ∑2N+Mlg(i) k=1 1[k̸=c] exp(cos(zc,zk)/τ) 14: LCL = ∑2N c=1 ∑2N i=1 m(sc, si)lw c,i 15: m(sc, si) = 1 means c and i exists in the same local window, unless m(sc, si) = 0. 16: update networks f and g to minimize LCL. 16: end for
• 17: return encoder network f(·), and throw away g(·) =0

The cross-lingual calibration consists of three operations: shifting, scaling and rotating. We do three operations separately to better understand the properties of latent space. For shifting, we compute the mean of sentence embedding µ from different languages. Then we acquire shifted mean sentence representation by subtracting corresponding language mean vector. For Scaling, we compute the variance of sentence embedding σ for all languages based on Common Crawler corpus. For a shifted sentence, we acquire a scaled sentence representation by dividing corresponding language variance vector. Fianlly, we learn the rotation matrix in unsupervised method proposed by [Conneau et al., 2018]. With orthogonal rotation matrix Wi,j , sentence embedding of lg(i) can be mapped to lg(j). We provide detailed description of this part in the supplementary materials.

4 Experiments

In this section, we first set up the experiments, and then present the performance comparison and result analysis.

Our CCP model has 1024 hidden units, 16 attention heads and 24 layers in encoder. Following [Wenzek et al., 2019], we collect a clean version of Common Crawl as pre-training corpus. It leads to 2,500GB multilingual corpus covering 108 languages. We first initialize the CCP model with XLMR[Conneau et al., 2020], and then run continued pre-training with the accumulated 2,048 batch size with gradient accumulation and a memory bank of 32768. One 2,048 batch consists of many small batches whose size is 32 for CCP. We use Adam Optimizer with a linear warm-up and set the learning rate to 3e-5. We select two pre-training tasks randomly in different batches. This costs 7 days on 16 V100 for CCP model.

4.1 Baselines

Here are the baselines for our experiments.

• ·M-BERT [Devlin et al., 2019] is a multilingual version of BERT.
• ·XLM-R [Conneau et al., 2020] uses a Transformer-based masked language model on one hundred languages.
• ·InfoXLM [Chi et al., 2021] formulates a cross-lingual pretraining as maximizing mutual information between multilingual multi-granularity text.
• ·Unicoder [Liang et al., 2020] uses mask language model and translation language model as pre-training tasks.
• ·CRISS [Tran et al., 2020] utilizes cross-lingual retrieval for iterative training.
• ·LaBSE [Feng et al., 2020] formulates a translation ranking task using bi-directional dual encoders.

We present the comparison between our method and ABSent [Fu et al., 2020] in appendix, since it only reports their performance on 3 language pairs of tatoeba, All transformer models in this paper use Bert-large structure except CRISS and LaBSE. CRISS follows mBART structure with 24 layers transformer, and LaBSE follows Bert-base structure with a 500k size vocabulary, which is twice as large as our model.

4.2 Cross-lingual Sentence Retrieval

To better evaluate the performance on massive languages, we adopt the Tatoeba corpus introduced by [Artetxe and Schwenk, 2019]. It consists of 1,000 English-centric sentence pairs for 112 languages and the task aims to find the nearest neighbor for each sentence in the other language using cosine similarity distance. To compare with previous model, we only report results on 14 language in experiments, and we present all results in Table 2 of supplementary material. Besides the English-centric dataset constructed by [Artetxe and Schwenk, 2019], we choose 14 language-pairs which don't contain English in the Tatoeba raw dataset, and we present results on 50 language-pairs which don't contain English in Table 2 of supplementary material. Following [Artetxe and Schwenk, 2019], we extract 1000 sentence-pairs for each language pair and test the pre-trained model on the Tatoeba dataset without fine-tuning directly. The accuracy for each language pair is computed. We report CCP in Table 1 and Table 2. For large models, we use the the averaged hidden vectors in the 14-th layers as sentence representation for sentence retrieval.

In Table 1, (1) we find CCP performs significantly better than XLM-R and CRISS and achieves new SOTA results among methods without using bilingual data on the Tatoeba dataset. (2) Compared with models using bilingual data, CCP performs better than InfoXLM [Chi et al., 2021], Unicoder [Liang et al., 2020] and LASER [Artetxe and Schwenk, 2019], but it is worse than LaBSE [Feng et al., 2020], which is the SOTA model with bilingual data on Tatoeba. LaBSE uses bidirectional dual encoders with 8192 batch size to learn crosslingual sentence representation. The translation ranking task is similar to contrastive learning task, which is severely affected by the batch size. Limited by hardware, we can only perform contrastive learning with a batch size of 32 , it's hopeful our model will have a better result with larger batch.

In Table 2, we find (1) CCP performs significantly better than XLM-R and CRISS. We observe that the performance of CRISS decreases obviously, because CRISS only mines English-centric bilingual pairs and it is easy to overfit Englishcentric sentence retrieval. (2) For models using bilingual corpus, CCP performs better than Unicoder and its performance is very close to LaBSE. Since LaBSE is trained on Englishcentric bilingual corpus, its performance decreases severely on sentence retrieval between two non-English sentences, which means it is overfitting on English-centric sentence retrieval. Compared with these English-centric models (LABSE, Unicoder, InfoXLM, CRISS) , our model is more general, and it does not rely on English data in down-streaming tasks. So our model has lesser performance loss when it is evaluated on non English-centric sentence retrieval.

These two experiments proved that CCP has good crosslingual sentence retrieval performance both on English-centric and Non-English languages pairs. Our performance is better than all models without using bilingual data and slightly worse than LaBSE which used bilingual data.

4.3 Cross Lingual Query Passage Retrieval

We further evaluate cross lingual transfer-ability on two new zero-shot settings: 1. Give a query from language L, retrieve relevant passages which can answer the query from the English corpus. 2. Give a query from language L, retrieve relevant passages which can answer the query from language L.

Hence, we adopt XOR-QA [Asai et al., 2020] dataset and Mr. TYDI [Zhang et al., 2021] dataset to evaluate our method on the two settings. Both of the two datasets are constructed from TYDI, a question answering dataset covering eleven typologically diverse languages. The XOR-QA dataset consists of three tasks: XOR-Retrieve, XOR-English Span, and XORFull. We take the XOR-Retrieve task to evaluate our method. XOR-Retrieve is a cross-lingual retrieval task where the query is written in a target language (e.g., Japanese) and the model is required to retrieve English passages that can answer the query. Same to the source paper [Asai et al., 2020], we measure the recall by computing the fraction of the questions for which the minimal answer is contained in the top n tokens selected. We evaluate with n = 2k, 5k: R@2kt and R@5kt (kilo-tokens). The Mr. TYDI dataset is a multi-lingual benchmark dataset for mono-lingual query passage retrieval in eleven typologically diverse languages, designed to evaluate ranking with learned dense representations. Same to the source paper [Zhang et al., 2021], we use MRR@100 and Recall@100 as metrics.

In this paper, we adopt a zero-shot setting to evaluate our method. We train the pre-trained model on Natural Question data and directly test the model on XOR-QA [Asai et al., 2020] dataset and Mr. TYDI [Zhang et al., 2021] dataset. We train the model on 8 NVIDIA Tesla V100 GPUs (with 32GB RAM). We use AdamW Optimizer with a learning rate of 1e-5. The model is trained up to 20 epochs with a mini-batch size of 48. The rest hype-parameters are the same as DPR [Karpukhin et al., 2020]. Besides, we also report our supervised result on XOR-Retrieve in appendix. We achieve 2% advantage over the second-placed model. Besides, we also experiment our model on XOR Retrieve in supervised setting with the same hyperparameter settings, and achieve SOTA results with 2% advantage over the second-placed model on leaderboard 1 .

In Table 3, (1) we find CCP performs significantly better than XLM-R , InfoXLM and LaBSE. CCP achieves new SOTA results among methods without using bilingual data on the Mr. Tydi dataset. (2) Especially, we find InfoXLM has good performance on the 15 languages that it has bilingual corpus. However, on the languages that it doesn't have bilingual corpus, such as BN, FI, and TE, we find it's worse than LaBSE. For CCP, we find it has best performance on almost all languages, because our model is able to support 108 languages without collecting any bilingual corpus. In Table 4, we find (1) CCP performs slightly better than InfoXLM and have comparable performance on LaBSE. On these low-resource languages that InfoXLM can't cover, our method still have a better performance. (2) We find our model has especially bad performance on Bengali (BN), which is consistent to results on Mr. TYDI. Our performance is better than all models using bilingual data, which means our context-aware pretraining is very suitable to various retrieval tasks, not only bilingual paraphrase retrieval, also query passage retrieval.

1 https://nlp.cs.washington.edu/xorqa/

5 Ablation Study and Sensitivity Analysis

Impacts of L2 Normalization and Asymmetric Batch Normalization We next study the importance of L2 normalization, batch normalization, and temperature τ in our contrastive loss. We use the pre-training and test setting in the last section. Table 5 shows that without L2 normalization before softmax and batch normalization in projection head, our model will fail in training. In the last row, we present the result of a model with L2 normalization and vanilla batch normalization, and it has a very terrible result. The model appears to "cheat" the pretext task and easily finds a low-loss solution. This is possibly because the intra-batch communication among samples (caused by BN) leaks information. [He et al., 2020]. With asymmetric batch normalization, different mean and variance will be used to calculate zc and zi, respectively.

Impacts of Batch Size andWindowSize Our current model is highly dependent on context window and batch size affects the number of negative samples directly. We study the importance of batch size and window size in the pre-training stage. Since 108 languages pre-training is too time-consuming, we only use English and French as pre-training corpus, and report the sentence retrieval result between English and French. Table 6 shows that without large window and large batch, performance is slightly worse.

6 Conclusion

We propose a new cross-lingual pretrain task called Contrastive Context Prediction (CCP), and conduct comprehensive evaluations with interesting findings observed. We find CCP task is able to make sentence embedding space of different language isomorphic. The proposed approach achieves a excellent performance on multi-lingual dense retrieval.