Towards Robust Dense Retrieval via Local Ranking Alignment

1 Introduction

Dense retrieval (DR) technique has been successfully applied to quite a few language systems, such as web search [Zhan et al., 2020; Luan et al., 2021; Xiong et al., 2021; Qu et al., 2021] and open-domain question answering [Karpukhin et al., 2020; Xiong et al., 2021; Qu et al., 2021]. DR models employ pre-trained language models (PLMs), like BERT [Devlin et al., 2019], to separately encode queries and passages into low-dimensional dense vectors, and adopt a lightweight similarity mechanism (e.g., dot product) for efficient online retrieval. However, recent work has raised the concern that DR models may not perform as well as expected when confronting queries with typos [Zhuang and Zuccon, 2021]. Aside from typos, for the same information need, queries cre-

ations [Bailey et al., 2015]. Meanwhile, Penha et al. [2022]

∗Corresponding author.

• Original Query: percentage of people with security clearance
• Query Variation: pct of people with security clearance

Figure 1: Example of local ranking alignment. In the representation space of DR, when replacing 'percentage' with its abbreviation 'pct', the query vector drifts away from its original position, resulting in effectiveness loss in that the rank of the relevant passage (id = 2) drops from the 2nd to the 4th. Our RoDR learns to maintain the relative positions of query-passage pairs in the DR space, by aligning the local ranking of query variation to that of original query as close as possible, for the improved DR robustness.

have demonstrated that query variations can significantly reduce the performance of a variety of neural re-ranking models. Nevertheless, it remains to be investigated whether and how various query variations (in addition to typos) weaken the DR model, and how to deal with these many types of query variations for more robust dense retrieval.

Different synthetic query variations have been used to assess the robustness of neural ranking models, including inserted typos [Wu et al., 2021; Zhuang and Zuccon, 2021; Penha et al., 2022], additional punctuation [Ma et al., 2021], synonym substitution [Penha et al., 2022], stopword removal [Penha et al., 2022], back-translation [Ma et al., 2021; Penha et al., 2022] and random word swapping [Penha et al., 2022]. In this paper, we summarize eight types of query variations and investigate their effects on the DR model. Given one original query set, the same type of query variation generator is used to modify all queries in this query set, yielding eight synthetic query variation sets for robustness testing. By comparing the performance of DR model on the original query set and synthetic query variation sets, we can examine the individual impacts of these query variation types. Unfortunately, relative to the original queries, significant effectiveness decreases can be observed on the query variations.

To address this issue, we propose RoDR, a novel variationenhanced robust DR model by taking the relative positions of query-passage pairs in the DR space into account. Intuitively, dense retrieval is based on the similarities between the querypassage pairs in the representation space, and a query variation causes the query vector to shift away from its original position, resulting in skewed query-passage similarities, and consequently, degraded retrieval effectiveness.

As illustrated by the example in Figure 1, replacing 'percentage' with its abbreviation 'pct' causes the query vector to drift closer to the negative passages, diverting from the original well-learned representation. Thereby, our RoDR aims to maintain the spatial layout of the DR model by a local ranking alignment mechanism on the query variations. Specifically, the original queries are first used to train a standard DR model (denoted as DRO) under the NLL objective, and then another DR model (denoted as DRN) initialized from DRO is further enhanced using the query variations, by minimizing the KL divergence on the similarity distributions of querypassage pairs between the DRN and DRO space. Additional to the NLL objective that maximizes the distances between positives and negatives to the query variations, our local ranking alignment mechanism aims to provide extra performance boost by functioning as a regularization loss that maintains the in-batch similarity ranking relationship for query-passage pairs with variations or not. Extensive experiments on MS MARCO [Nguyen et al., 2016] and ANTIQUE [Hashemi et al., 2020] datasets demonstrate that RoDR is able to achieve performance gains on both the original and variant queries. Moreover, RoDR provides a general framework for training a query noise-tolerant DR model. As shown by the evaluation results, RoDR leads to enhanced ranking effectiveness when it is applied to a variety of existing DR models, including ANCE [Xiong et al., 2021], TAS-Balanced [Hofst¨atter et al., 2021], and ADORE+STAR [Zhan et al., 2021]. Our contributions are three-fold:

• We systematically investigate the impacts of eight types of query variations on dense retrieval (DR) models, and observe non-negligible performance decreases.
• A novel robust DR model RoDR is proposed, which learns to maintain the relative positions of query-passage pairs in the DR space by a local ranking alignment mechanism.
• Our RoDR provides a general query noise-tolerate learning framework that is applicable to various existing DR models, improving their robustness and effectiveness.

2 Preliminaries

As mentioned above, recent studies have highlighted the nonnegligible negative impact of query variations on text ranking effectiveness [Zhuang and Zuccon, 2021; Penha et al., 2022]. Herein, we first introduce eight types of operations that generate different query variations in Section 2.1, followed by a brief introduction to the dense retrieval in Section 2.2.

2.1 Query Variation Generation

MISSPELL: injected misspellings. Apart from the widely-studied Typos [Wu et al., 2021; Zhuang and Zuccon, 2021], we also study OCR, Keyboard and SpellingError as defined in TextFlint [Gui et al., 2021]. Specifically, out of the above four types of spelling errors, one type is randomly selected and injected into one random query word.

EXTRAPUNC: extra punctuation. Akin to [Ma et al., 2021], one to three identical and successive punctuation marks (including comma, period, question mark, and exclamation mark) are added at the end of the query.

BACKTRANS: back-translation. Akin to [Ma et al., 2021; Penha et al., 2022], the query in English is first translated to German and then is translated back to English, by the default machine translation model in TextFlint [Gui et al., 2021].

SWAPSYN: swapped synonyms. Akin to [Penha et al., 2022], one random word in the query is replaced by its synonym according to the Glove embeddings or WordNet, coined as SWAPSYN-GLOVE and SWAPSYN-WNET, respectively.

TRANSTENSE: transformed verb tense. The tense of one random verb in the query is transformed by TextFlint [Gui et al., 2021]. No prior work has studied this type of query variation as far as we are aware of.

NOSTOPWORD: stopword removal. Akin to [Penha et al., 2022], all stopwords in the query are deleted.

SWAPWORDS: swapped words. Akin to [Penha et al., 2022], two randomly sampled words in the query are swapped by TextAttack [Morris et al., 2020].

The examples for each type of query variations are listed in Table 1. Note that not all queries can obtain all eight types of variations. For example, if one query does not contain any stopword, the NOSTOPWORD variation is not applicable. Similarly, the BACKTRANS, SWAPSYN, TRANSTENSE and NOSTOPWORD variations could also be inapplicable to some certain queries. Additional to the synthetic query variations, the human validated query variations from [Penha et al., 2022] are also included in our experiments.

2.2 Dense Retrieval

Dense retrieval (DR) usually employs the dual-encoder architecture, and optimizes the negative log likelihood (NLL) of the positive (relevant) passage among a set of negative (irrelevant) passages during its standard training procedure [Karpukhin et al., 2020]. After that, all passages in corpus are encoded and indexed offline, and only a single query encoding inference is needed during online retrieval. Besides, as mentioned, previous work [Zhuang and Zuccon, 2021] has studied the impact of typos in query on the DR model, and found that exposing query with typos (50% chances) directly to the DR model during training using NLL loss could reduce the negative impact of typos. In our work, additional to NLL, a local ranking alignment mechanism is also used to maintain the relative positions of query-passage pairs in the DR space, which is crucial to the subsequent DR retrieval.

3 Method

Let DO = {(qi, p+i , p-i,1 , · · · , p-i,n )}mi=1 be the original training data, wherein each original query qi is coupled with one positive passage p+i and n negative passages p-i,j . The negative passages are usually collected from the corpus or topranked BM25 candidates [Karpukhin et al., 2020]. As mentioned, we first use DO through standard DR training procedure to obtain a DR model, denoted as DRO. Then, another DR model (denoted as DRN) initialized from DRO is further optimized to improve the robustness against query variations under the guidance of DRO, whose parameters are frozen during training. Let DN = {(qi, qi, p+i , p-i,1 , · · · , p-i,n )}mi=1 be the training data used for local ranking alignment, wherein each original query qi is coupled with one uniformly sampled query variation qi from Section 2.1, and the negatives p-i,j are all re-collected from the top-ranked candidates returned by DRO, which act as the nearer neighbors as described later.

3.1 Local Ranking Alignment

As mentioned, in the representation space of DRO, different query variations cause the query vector to drift away from its original position, resulting in the change of its relative positions among the neighboring passages and further a performance decrease. Thereby, it is necessary to maintain the relative positions of query-passage pairs in the DR space, which could be scrambled by the query variations. Herein, we propose to align the ordering of neighboring passages (queries) from DRN to DRO according to the in-batch local ranking of queries (passages). Specifically, in a batch of training samples B = {(qi, qi, p+i , p-i,1 , · · · , p-i,n )}|B| i=1 , an original query set Q = {qi}|B| i=1 , a query variation set Q = {qi}|B| i=1 , and a passage set P = {p+i , p-i,1 , · · · , p-i,n }|B| i=1 = {pj}|P| j=1 , wherein |P| = |B| · (n + 1), can be all collected.

Query centering alignment. For each original query qi ∈ Q and its variation qi ∈ Q, all in-batch positive and negative passages are collected as their shared neighbors (namely, P), which also constitute the local ranking of both qi and qi. In this local ranking, the rank of each passage for query variation qi is calibrated to that for original query qi. In order to be in line with the scoring mechanism of DR model, we also use dot product as the similarity measure for the local ranking. Besides, we use the similarity distribution to align the local ranking between qi and qi, as the relative order of passages in the local ranking matters more than the exact similarity values when training a robust DR model. Consequently, for each qi ∈ Q and pj ∈ P, we compute the conditional probability of passage pj being close to the original query qi in the representation space of DRO, as defined in Eq. 1.

wherein sim(q, p) is the dot product similarity between the representations of query q and passage p.

Similarly, in the representation space of DRN, the conditional probability of passage pj ∈ P being close to the query variation qi ∈ Q can be defined as Eq. 2.

To mitigate the impacts of the different query variations to the local ranking, conditional probabilities po(pj|qi) and pn(pj|qi) should be as close as possible. This alignment target is estimated by the Kullback-Leibler (KL) divergence between these two conditional probabilities:

Passage centering alignment. In the query centering alignment, we calibrate the relative positions of queries using the in-batch passages. Similarly, we can also calibrate the relative positions of passages using the in-batch queries. Specifically, for each passage pj ∈ P, all original queries in Q are served as the local ranking candidates in DRO, and all query variations in Q are served as the local ranking candidates in DRN. Similar to Eq. 1-3, we align conditional probability pn(qi|pj) to po(qi|pj) by KL divergence as in Eq. 6.

Nearer neighbors. To achieve better alignment, we use the top-1k passages returned by DRO excluding those judged relevant to update the initial negatives in DO to obtain the final DN. As the utilized local ranking involves all in-batch passages (especially negatives), these new negatives from DRO act as the nearer neighbors for queries, which is beneficial to correctly calibrate the relative positions of query-passage pairs in the DRN space. Besides, these new negatives also act as hard negatives in NLL objective, which have been shown useful to train an effective DR model [Xiong et al., 2021].

3.2 Model Training

During training, along with the query centering and passage centering alignment described in Section 3.1, the NLL which teaches the DR model to pick out the positive passage from a set of negative passages for query variations is also used:

wherein |B| is the effective batch size, n is the number of effective negative passages, which is usually larger than n (in DN) due to the use of in-batch negatives [Karpukhin et al., 2020] or cross-batch negatives [Qu et al., 2021].

Different to the standard DR training, the original query qi in NLL loss is replaced by its variation qi. Besides, different to [Zhuang and Zuccon, 2021], a local ranking alignment mechanism (LRA) is used as a regularization item to maintain the relative positions of query-passage pairs in the DR space (namely, DRN). As summarized in Eq. 8, we train DRN using the weighted sum of NLL and LRA losses, wherein w1, w2 and w3 are hyper-parameters to control the loss weights.

After the above query noise-tolerate training, the resulting DRN model is coined as RoDR, and used for robust retrieval.

4 Experiments

4.1 Experimental Setup

Datasets and metrics. We employ both MS MARCO passage and document datasets for our experiments. The MS MARCO passage (document) corpus contains 8.8 (3.2) million passages (documents), from which we construct about 0.40 (0.37) million training samples. We use 6,980 (5,193) Dev queries in MS MARCO passage (document) dataset for evaluation, along with the official metric, namely, MRR@10 (MRR@100). Additionally, Recall@1000 (Recall@100) is also used to evaluate the recall quality for passage (document) retrieval. Statistical significance with paired two-tailed t-test is reported. Akin to [Wu et al., 2021; Zhuang and Zuccon, 2021], the drop rate (loss ratio in performance) on one query variation set relative to the original query set (denoted as ORIGINAL) is also reported.

Query variations. We use eight types of query transformation as defined in Section 2.1 for query variation generation on MS MARCO Dev sets. Specifically, for each type of query transformation, we apply it to all queries in the Dev set to obtain a variation set, which is named by the acronym of the query variation (e.g., MISSPELL). As mentioned in Section 2.1, if one variation is not applicable to the given query, this query is kept original in this variation set. The statistics of query variations on MS MARCO passage and document Dev sets are summarized in Tables 2 and 3, respectively. Besides, we report the average results on these eight query variation sets, denoted as VARIATIONS (AVG.).

Manually validated query variations. Additional to our synthetic query variations, which could be occasionally inapplicable or incomprehensible, we also experiment with the manually validated query variations released in [Penha et al., 2022]. This human validated query variation data is based on the 200 original queries from ANTIQUE dataset [Hashemi et al., 2020] (namely, the 'antique/train/split200-valid' set available in ir datasets [MacAvaney et al., 2021]), denoted as ANTIQUE-ORIGINAL. There are five released query variation sets (e.g., T5DescToTitle, BackTransl as in [Penha et al., 2022]), which are used to test the DR models trained on MS MARCO passage dataset in a zero-shot setting. The average results on these five validated query variation sets are reported, denoted as ANTIQUE-VARIATIONS (AVG.).

Training details. Our model training is based on the Tevatron toolkit [Gao et al., 2022], with parameter-shared BERTBase model as the query and passage (document) encoders. The max query length is 32, and the max passage (document) length is 128 (512). For MS MARCO passage (document) retrieval, the DR model is trained with the learning rate of 5e-6 and per-device batch size of 16 (2) for 4 epochs on one (four) GeForce RTX 3090 GPU(s). The negative passages (documents) in DO are sampled from the provided official triples (the top-1k BM25 candidates), and there are 7 negatives in each training sample, namely, n = 7 in DO and DN. The loss weights w1, w2, w3 in Eq. 8 are set as 1, 1, 0.2 (1, 0.1, 1) for MS MARCO passage (document) retrieval.

Models in comparison. As for baselines, we compare the following models: DROQ is a standard DR model trained using original queries (namely, DO), without using any query variation. DRQV is trained using query variations rather than original queries (namely, all original queries in DO are replaced with their variations, instead of 50% used in [Zhuang and Zuccon, 2021], since our setting yields better results). DROQ→QV is first trained using original queries before using query variations, in two training stages and for the same training steps with RoDR for fair comparison.

Plugging RoDR into existing DR models. As described, DROQ is the basic model of our vanilla RoDR, which is denoted as RoDR w/ DROQ. In addition to the standard DR training used in DROQ, quite a few recent works have proposed more effective DR models with more advanced training methods. Thereby, we also apply RoDR to a few existing DR models, including ANCE [Xiong et al., 2021], TASBalanced [Hofst¨atter et al., 2021] and ADORE+STAR [Zhan et al., 2021]. Specifically, we use the released checkpoints of these DR models trained on MS MARCO passage dataset to initialize DRO and DRN, and then conduct query noisetolerate training in the same way as described in Section 3. Herein, equal loss weights in Eq. 8 are used, and the learning rate is 3e-5, 5e-6 and 5e-7 for RoDR w/ ANCE, TASBalanced and ADORE+STAR, respectively. Other training configurations are the same as in RoDR w/ DROQ.

4.2 Results

The performance degradation on query variations. As seen in Tables 2 and 3, non-negligible degradation of DROQ to varying degrees is observed on all query variation sets relative to ORIGINAL set. On average, comparing ORIGINAL with VARIATIONS (AVG.), the performance of DROQ drops 19.4% and 7.26% in terms of MRR@10 and R@1000 on MS MARCO passage Dev set, 19.8% and 14.9% in terms of MRR@100 and R@100 on MS MARCO document Dev set. Among eight types of query variations, DROQ suffers from misspellings (MISSPELL) the most (over 50% MRR drops on both Dev sets), and suffers from extra punctuation (EXTRAPUNC) and transformed verb tense (TRANSTENSE) the least. Overall, the standard DROQ indeed suffers from effectiveness losses on various query variations.

Exposure of query variations to the DRmodel helps. As seen in Tables 2 and 3, when using various query variations during training, DR models including DRQV and DROQ→QV

are much more tolerant to all types of query variations. On average, when facing with query variation sets, DRQV gains 1.7 (2.5) points over DROQ in terms of MRR@10 (MRR@100) on passage (document) Dev set. Besides, compared to DROQ, the average drop rates of DRQV and DROQ→QV are reduced on both Dev sets (e.g., from 19.4% to 14.4% for DRQV on passage Dev set in terms of MRR@10). On the whole, simply adding query variations into the training data is helpful.

RoDR produces significantly better retrieval results and smaller drop rates. The retrieval results of vanilla RoDR w/ DROQ are summarized in the last column of Tables 2 and 3. It can be seen that RoDR w/ DROQ not only performs significantly better on all query variation sets, but also on the original query set, which indicates the super effectiveness of RoDR framework. Besides, compared to all three baseline models, RoDR w/ DROQ can also reach smaller drop rate (which reflects the robustness of DR model) from original query set to various query variation sets. Thereby, we demonstrate that RoDR boosts not only the retrieval effectiveness, but also the retrieval robustness on query variations.

RoDR still performs better on manually validated query variations. The zero-shot retrieval results on ANTIQUE are listed in Table 4. We can see that DRQV and DROQ→QV perform on par with DROQ on both ANTIQUE-ORIGINAL and ANTIQUE-VARIATIONS (AVG.). It indicates that simple exposure of query variations provides no benefit to DR model on ANTIQUE. In contrast, RoDR w/ DROQ produces significantly better retrieval quality on both ANTIQUE-ORIGINAL and ANTIQUE-VARIATIONS (AVG.). Meanwhile, RoDR w/ DROQ still reaches smaller average drop rate on validated query variations (e.g., 5.67% in terms of nDCG@10).

RoDR provides extra performance boosts over existing DR models. As seen in Table 5, although ANCE, TASBalanced and ADORE+STAR with more advanced training strategies can outperform DROQ, they still suffer from the query variations (e.g., 15%-17% average drop rate in MRR@10 on MS MARCO, and 6%-12% average drop rate in nDCG@10 on ANTIQUE). Meanwhile, when RoDR is applied to these three DR models, their effectiveness and robustness can be further improved (e.g., about one point gains in MRR@10 on MS MARCO ORIGINAL set, and smaller average drop rates on both MS MARCO and ANTIQUE).

4.3 Analysis

Ablation study on loss function of RoDR. As seen in Table 6, on ORIGINAL set, the contribution of passage centering alignment (namely, Lq|p) is not strikingly high, but it is still beneficial to the query variation sets. Meanwhile, without both query and passage centering alignments (namely, Lp|q and Lq|p), the performance of RoDR on both ORIGINAL and VARIATIONS (AVG.) sets is greatly affected. Moreover, NLL on query variations (namely, Lnll) is also important in RoDR since its removal leads to marked effectiveness loss.

The learning curve of RoDR. Furthermore, we present the learning curve of RoDR w/ DROQ in Figure 2 to show the effect of the local ranking alignment mechanism (abbreviated as LRA). When used with the NLL loss, LRA acts as a positional regularizer (in the DR space) over Lnll to prevent overfitting on the query variations. Besides, the performance on MS MARCO passage Dev set (both ORIGINAL and VARIATIONS (AVG.)) is always better when LRA is enabled. Thus, our LRA is complementary to NLL, and using both for training can achieve more robust DR retrieval.

The training efficiency. Although using augmented query variations for training can improve the robustness performance, it indeed increases the training time of DR models. In our experiments, for MS MARCO passage retrieval, it takes about 10hrs for both DROQ and DRQV , about 20hrs for DROQ→QV , and about 22.7hrs for RoDR w/ DROQ. In other words, the training time from scratch of RoDR is normally double when compared to the standard DR model. However, when based on the existing DR models, it is generally costefficient to apply RoDR for robustness enhancement.

5 RelatedWork

Dense retrieval systems have been widely studied and developed. Reimers and Gurevych [2019] present SentenceBERT in a siamese architecture to derive semantically meaningful sentence embeddings for textual similarity matching. Karpukhin et al. [2020] use BM25 negatives to better finetune the DR model and successfully apply it to the open question answering task. Luan et al. [2021] find that dense retrieval is not effective when the text sequences are too long.

Xiong et al. [2021] use global negatives from the DR model to enhance itself. Then, more and more studies [Qu et al., 2021; Hofst¨atter et al., 2021; Ren et al., 2021a; Zhan et al., 2021; Ren et al., 2021b; Li et al., 2021] work toward more powerful DR models by a series of sophisticated training procedures.

The robustness issue of DR model has also been raised. Zhuang and Zuccon et al. [2021] have shown that DR model degrades dramatically when facing with the query with typos. In fact, the typos in query not only affect the performance of DR, but also have serious impacts on different neural ranking models [Wu et al., 2021]. Meanwhile, apart from typos, other various query variations are also found to have negative influence on the performance of neural ranker [Ma et al., 2021; Penha et al., 2022]. In our work, additional to typos, a variety of query variations are included to study the robustness of DR model. Moreover, this work proposes a novel RoDR model, which provides a general learning framework that boosts the robustness of various existing DR models.

6 Conclusion

In this work, we first summarize eight types of query variations, which are used for the systematic investigation on the robustness of DR model. Evaluation results show that DR model degrades significantly under various query variations. Then, to address this, we propose a novel robust DR model, named RoDR, to boost the performance by maintaining the relative positions of query-passage pairs in the DR space. In future work, we plan to extend our study to other related tasks, like open question answering [Karpukhin et al., 2020].

Acknowledgments

This work is supported by the National Natural Science Foundation of China under Grant no. U1936207 and 62076233, and the University of Chinese Academy of Sciences.