Re-Weighting Cross-Modal Pairs via Rank Consistency for Noise-Robust Retrieval

1 Introduction

Cross-modal alignment learns a unified space where semantically related data lie close together [Girdhar et al., 2023], yet its success heavily relies on large-scale, correctly aligned multi-modal datasets that are expensive to curate. Crawling multi-modal pairs from the Internet [Jia et al., 2021] scales better but inevitably introduces mismatched or partially relevant pairs (e.g., an image unrelated to its alt text), a problem known as Noisy Correspondence [Huang et al., 2021]. Training standard models (e.g., CLIP [Radford et al., 2021]) on such noisy data forces them to fit spurious correlations, severely degrading retrieval performance [Zhang et al., 2024]. Therefore, developing robust learning algorithms that

∗Corresponding Author.

Well-aligned: Jockeys racing horses down a grass track.

Partial-aligned: A group of horses and their riders are racing each other.

Mismatched: Several people working outside to construct a wood frame.

Figure 1: Examples of data pairs with different matching degrees.

can resist noisy correspondence is of critical practical importance.

This paper studies image-text matching under noisy correspondence. A widely adopted strategy is to estimate a soft matching degree for each pair and adjust its training weight accordingly [Huang et al., 2021; Zhang et al., 2024; Li et al., 2025]. As shown in Figure 1, pairs range from well-aligned (accurate description) to partially aligned (missing details such as the background grass track) to fully mismatched, and should receive high, medium, and zero weight, respectively. Accurately estimating this matching degree is challenging. Existing methods are trained with contrastive learning, which only enforces relative ordering rather than calibrated absolute scores. Consequently, model-predicted similarities are unreliable measures of true semantic alignment. To address this, prior works map raw similarities to [0, 1] weights [Huang et al., 2021; Ma et al., 2024] or construct local graphs for label propagation [Li et al., 2025]. Despite their progress, these approaches fundamentally rely on the model's own similarity predictions, assuming higher scores indicate better matches. This assumption fails under noisy training: the model can be misled to assign deceptively high similarity to partially aligned pairs (see Section 4.6), causing inaccurate re-weighting.

To overcome this limitation, we evaluate matching quality through ranking consistency rather than absolute similarity. Our key insight is that a well-aligned pair (I, T) should occupy semantically consistent positions in the embedding space; hence, using I and T as queries to retrieve other texts (or images) should yield consistent rankings. Figure 2 illustrates this: although T1 and T2 are equidistant from I1, T2

Similarity View

produces rankings more consistent with I1's neighborhood structure and therefore deserves a higher weight. Case studies in Section 4.6 provide further intuitive examples.

In practice, we treat intra-modal retrieval as the ideal ranking and cross-modal retrieval as the actual ranking, then measure their agreement via Normalized Discounted Cumulative Gain (NDCG) [J¨arvelin and Kek¨al¨ainen, 2000]. Because this weight derives from the global manifold structure rather than point-wise predictions, it remains stable even when individual similarity scores are unreliable. To sum up, our main contributions are as follows:

• We propose a new re-weighting method based on ranking consistency between cross-modal and intra-modal retrieval, offering a novel solution to handle noisy correspondence in an end-to-end manner.
• Our method can finely distinguish between well-aligned and partially aligned pairs even when they receive similar similarity scores, enabling more precise soft-label estimation in noisy correspondence.
• Extensive experiments on three benchmarks (Flickr30K, MSCOCO, and CC120k) under both synthetic and realworld noise demonstrate our method achieves state-ofthe-art performance.

2 RelatedWork

2.1 Cross-modal Retrieval

Cross-modal retrieval aims to align data from different modalities within the same embedding space. Current methods mainly focus on how to measure the similarity between images and text, which can be broadly categorized into two types: (1) coarse-grained measurement [Kiros et al., 2014; Faghri et al., 2018], which measures whether the global features of images and text have the same semantics. (2) fine-grained measurement [Lee et al., 2018; Li et al., 2019; Chen et al., 2020; Diao et al., 2021; Xie et al., 2025b; Xie et al., 2025a], which measures the semantic consistency of fine-grained fragments and considers the relationships between fragments. Existing models have achieved significant success in cross-modal retrieval. Pre-trained models like CLIP [Radford et al., 2021] demonstrating powerful zeroshot cross-modal retrieval performance. However, prior work reveal they remain sensitive to noisy correspondence issues. When the fine-tuning data in downstream applications contains mismatched data, model performance is severely compromised.

2.2 Noisy Correspondence Learning

Noisy Correspondence Learning is first introduced by Huang et al. [2021], aims to develop robust cross-modal alignment algorithms to reduce the impact of mismatched and partially matched data on model performance. The core of existing methods is to accurately estimate a soft label reflecting the semantic matching degree of each pair. To achieve this, NCR [Huang et al., 2021] first divides samples into clean and noisy subsets using the sample selection method in DivideMix [Han et al., 2018; Arazo et al., 2019; Li et al., 2020], then converts the model's predicted imagetext similarity into soft labels using an adaptive prediction function, and finally trains the model using triple loss with soft margin. Subsequent work has investigated more fine-grained sample partitioning schemes [Ma et al., 2024; Zha et al., 2024; Lyu et al., 2025] and considered the relationships between samples, estimating soft labels through optimal transmission processes [Han et al., 2024] or label propagation [Li et al., 2025]. These methods typically assume that image-text pairs with higher predicted similarity (or lower loss) are more matched, making them sensitive to inaccurate image-text similarity predictions by the model.

Other methods consider image-text pairs with smaller losses as clean anchors to estimate soft labels. BiCro [Yang et al., 2023] assumes that similar images should correspond to similar text and estimates soft labels by comparing the distances of other image-text pairs to the clean anchors. Recent work NPC [Zhang et al., 2024] estimates soft labels by analyzing changes in the model's predictions on clean anchors before and after gradient updates. The performance of these methods is highly dependent on the selection of clean anchors. If the clean anchors contains partially matched pairs, the estimated soft labels are unreliable. Furthermore, because the selection of clean anchors depends on model predictions, these methods are also sensitive to inaccurate image-text similarity predictions by the model.

In addition to soft label estimation, existing work has also explored other directions, including developing loss functions robust to noisy correspondence [Qin et al., 2022; Hu et al., 2023; Qin et al., 2023], introducing regularization terms to constrain the feature space [Yang et al., 2024; Zhao et al., 2024], and generating pseudo captions for mismatched pairs [Duan et al., 2024; Han and Cao, 2025; Xie et al., 2025c]. These methods are orthogonal to soft label estimation, improving performance from different aspects.

Our method offers a novel approach for soft label estimation: comparing the relative relationships between image and text with other data points. This method focuses on relative ranking structure, which is more robust to inaccurate similarity predictions by the model, and eliminates the need for clean anchor selection, offering a simple and effective method for noisy correspondence learning.

3 Method

3.1 Preliminary

Cross-modal retrieval [Kiros et al., 2014] aligns different modalities into a shared embedding space. Formally, given a training set {(Ii, Ti), yi}Ni=1 , (Ii, Ti) denotes the i-th imagetext pair with label yi ∈ {0, 1} (yi = 1 for matched, 0 for mismatched). The noisy correspondence problem [Huang et al., 2021] occurs when partially or fully irrelevant pairs are incorrectly labeled as matched (yi = 1). The goal is to learn robust correspondence despite such label noise. Following prior work [Zhang et al., 2024; Li et al., 2025], we adopt CLIP [Radford et al., 2021] as the backbone, comprising an image encoder f(·) and a text encoder g(·). The similarity between a pair is measured by cosine similarity:

CLIP is trained with in-batch negative sampling strategy and optimized by the InfoNCE loss [van den Oord et al., 2018]:

where b is the batch size, τ is the temperature parameter which controls the sharpness of the probability distribution.

3.2 Data Partitioning

To prevent the model from memorizing the noisy correspondences, we propose a two-phase strategy: first filtering out obviously mismatched image-text pairs based on loss distribution; then re-weighting the remaining pairs according to their semantic matching degrees. This subsection details the first phase: filtering mismatched pairs by fitting Gaussian Mixture Models (GMM) [Arazo et al., 2019] on per-sample losses.

Sample Loss Computation. Memorization effect in Deep Neural Networks (DNNs) indicates DNNs typically learn from correct labels before fitting noisy ones [Arpit et al., 2017; Liu et al., 2020]. Therefore, we treat pairs with higher losses as mismatched data. For an image-text pair (Ii, Ti), we compute its contrastive loss in both directions:

Loss Distribution Modeling. To distinguish high-loss samples, we fit the loss distributions using one-dimension two-component Gaussian Mixture Models: the component with the higher mean corresponds to mismatched sample pairs, and the other component corresponds to matched or partially matched sample pairs. The distribution is formally defined as:

where πk, µk, and σ2 k denote the weight, mean, and variance of the k-th Gaussian component, respectively. We fit this GMM independently to the sets ℓi2t i and ℓt2i i using the Expectation-Maximization (EM) algorithm.

Sample Filtering. After fitting the GMM, for each sample i, we calculate the posterior probability that ℓi belongs to the lower-mean Gaussian component k′ , which is computed via Bayes' theorem:

We require a sample to exhibit low loss in both directions to be considered a candidate for subsequent re-weighting. The filtered candidate set C is defined as:

where δ is a probability threshold. Samples outside C are considered clear mismatched pairs and are assigned zero weight during the subsequent re-weighting stage.

3.3 Sample Reweighting by Ranking Consistency

This section address the core challenge: reliably re-weighting pairs in C, which contain both well matched and partially matched pairs.

Figure 3 illustrates our basic idea: if image Ii and text Ti describe the same semantics, they should occupy similar positions in the feature space. Consequently, using Ii and Ti as queries to retrieve other texts (or images) in the dataset should yield two similar rankings lists. We measure the consistency of ranking lists by Normalized Discount Cumulative Gain (NDCG), which is a standard information retrieval metric that evaluates the quality of a ranking list against an ideal order. Take estimating the matching degree of (Ii, Ti) via text retrieval as an example. We compare following two ranking:

• Ideal Ranking: The ranking of all texts based on their similarity with Ti. We denote this ranked list as Rt2t (Ti), and Rt2t (Ti)j represents the text in the training set with the j-th largest similarity to Ti.
• Actual Ranking: The ranking of all texts based on their similarity with Ii. We denote this ranked list as Ri2t (Ii) and Ri2t (Ii)j refers to the text in the training set with the j-th largest similarity to Ii.

We quantify this top-rank consistency between Ri2t (Ii) and Rt2t (Ti) using NDCG. For a given truncation level K, the Discounted Cumulative Gain (DCG) of the actual ranking is:

We treat S(Ti,Ri2t (Ii)j) as the "relevance" of the text at position j in the actual ranking. This design is crucial: it uses the text-to-text similarity as the ideal relevance score to assess whether the image Ii has retrieved the texts that are semantically close to Ti. Generally, the higher the similarity between the top-ranked texts in Ri2t (Ii) and Ti, the higher the DCG score. Since the DCG score is unbounded thus unsuitable as sample weights, we utilize the DCG score under ideal ranking Estimate Matching Degree of

(i.e., IDCG) to normalize it to [0, 1], which generates NDCG score in text retrieval:

Symmetrically, when estimate matching degree of (Ii, Ti) by image retrieval, we use S(Ii, Ij) as the ideal relevance to evaluate the ranking produced by using text Ti to retrieve images. Denote the NDCG@K score for (Ii, Ti) in image retrieval as NDCG@Kimg(Ii, Ti), the final ranking consistency weight for the pair (Ii, Ti) is the average of the assessments in both image and text retrieve:

A high wi indicates that Ii and Ti have highly consistent neighborhood structures across modalities, implying strong semantic matching. During training, we assign zero weights to samples outside set C, avoiding the learning from mismatched pairs. For samples from set C, we reweight them using wi to prioritize learning information from reliable pairs:

We term our method as Ranking Consistency Re-weighting (RCR).

4 Experiment

4.1 Datasets and EvaluationMetrics

Datasets. We evaluate our method on three widely-used image-text matching datasets: Flickr30K [Young et al., 2014], MS-COCO [Lin et al., 2014], and Conceptual Captions [Sharma et al., 2018]. For well-annotated Flickr30K and MS-COCO, we randomly shuffling the captions of training images to simulate noisy correspondence. Conceptual Captions is crawled from the Internet, which contains 3% ∼ 20% mismatched pairs. Following NPC [Zhang et al., 2024], we use the CC120K subset in evaluation.

Evaluation Protocol. We use the recall rate at K (R@K) metric to evaluate retrieval performance. R@K measures the proportion of relevant items retrieved within the top K items. We report R@1, R@5, R@10 for both image-to-text and textto-image retrieval, and their sum (rSum).

4.2 Implementation Details

Following prior work NPC [Zhang et al., 2024] and GLP [Li et al., 2025], we take CLIP with ViT-B/32 as backbone and train it using AdamW optimizer [Kingma and Ba, 2015; Loshchilov and Hutter, 2019] with a weight decay of 0.2. The initial learning rate is set to 5e-7, and cosine annealing is used for training. On all datasets, we train the model for 5 epochs with a mini-batch size of 256. The hyperparameter δ is chosen from {0.5, 0.9}, and K is chosen from {5, 10} based on the validation performance.

4.3 Comparison with AdvancedMethods

Following previous work GLP [Li et al., 2025], we compare our Ranking Consistency Re-weighting (RCR) method with a comprehensive set of state-of-the-art baselines. These include robust learning methods built upon the SGRAF [Diao et al., 2021]: NCR [Huang et al., 2021], DECL [Qin et al.,

2022], BiCro [Yang et al., 2023], L2RM [Han et al., 2024], PC2 [Duan et al., 2024], CREMA [Ma et al., 2024], and SPS [Xie et al., 2025c]. To ensure a direct and fair comparison on the same foundation, we evaluate against SOTA methods based on the CLIP backbone: fine-tuned CLIP [Radford et al., 2021], NPC [Zhang et al., 2024], and GLP [Li et al., 2025]. Notably, NPC and GLP are highly relevant baselines as they also address the problem through sample re-weighting schemes, making the performance comparison particularly indicative of the effectiveness of our proposed ranking consistency paradigm versus existing weighting strategies.

Results on Simulate Noise. Table 1 compares performance on Flickr30K and MS-COCO 1K under varying noise ratios. Our method achieves highest rSum scores in all configuration, reaching state-of-the-art performance across noise levels. A particularly insightful finding is that RCR also demonstrate significant improvements on the original, clean splits (0% noise). This indicates that even well-annotated datasets contain partially-matched or ambiguous pairs. Existing methods that rely on similarity predicted by the model often overestimate the reliability of these pairs. In contrast, our global ranking consistency measure naturally down-weights such pairs because their cross-modal and intra-modal retrieval results exhibit inherent inconsistencies, thereby refining the learning signal even in "clean" datasets. Our case study in Section 4.6 intuitively shows this advantage.

Results onReal-worldNoise. To validate the effectiveness of our method in real-world scenarios, we further evaluate it on the CC120K dataset, which contains a significant proportion of naturally occurring noisy correspondences. As shown in Table 2, our RCR method consistently achieves the best performance across nearly all metrics. Specifically, RCR attains the highest R@1 scores for both image-to-text (73.3) and text-to-image (73.0) retrieval, which are key indicators of precise matching. The overall retrieval sum (rSum) of 517.3 also represents a new state-of-the-art, surpassing the previous best method (GLP) by +2.3 points.

4.4 Ablation Study

To further analysis the effectiveness of our method, we evaluate the performance when replacing the wi calculate in Eq. (12) with following strategies:

• No Re-weight: wi = 1 for all pairs in C.
• Image-Text Similarity: Using the cosine similarity between image and text features as weight, i.e., wi = S(Ii, Ti).
• Loss Distribution: Using the posterior probability from the GMM (Eq. 7) as the reliability weight, i.e., wi = 1 2 (αt2i i +αi2t i ). This typically assigns higher weights to samples with lower losses.
• Model Prediction: Using the model's own in-batch prediction probabilities as weights. P i2t i in Eq. (2) denotes the probability that given the image Ii, its paired text Ti is correctly matched among all texts in the batch. Conversely, P t2i i in Eq. (3) represents the probability that given the text Ti, its paired image Ii is correctly matched among all images in the batch. We average these two probability and use it as the sample weights, i.e., wi = 12 (P i2t i +P t2i i ).

Table 3 compares the performance of different re-weight strategy on CC120K dataset. The results lead to two critical findings that reinforce our motivation.

Firstly, using the raw image-text similarity as weight results in the worst performance (rSum: 491.6), even below the no re-weight baseline (rSum: 509.7). This stark outcome empirically confirms the challenge we highlighted in the introduction: models trained with contrastive objectives (like CLIP) are optimized to produce relative orderings which only ensures matched pairs have higher scores than mismatched ones, rather than output absolute, calibrated measures of semantic matching degree. Consequently, the similarity scores (e.g., typically within a narrow range like 0.2-0.4 in our experiment) are unreliable as direct confidence weights.

Secondly, strategies that derive weights from processed model outputs like loss distribution and model prediction only yield marginal improvements over the no re-weight baseline. Also, their performance fall short of the previous state-of-theart, GLP (rSum: 515.0 in Tab. 2). This indicates that while they mitigate some noise, they remain fundamentally constrained by the inherent biases and unreliability of the model's direct predictions in noisy settings. In contrast, our method derives weights from the global agreement of relative ranking orders rather than any absolute score, achieves a substantial gain (rSum: 517.3). It is the only re-weighting paradigm that surpasses GLP. This key distinction demonstrates that our method's effectiveness stems from its novel weighting criterion, rather than from a generic re-weighting effect.

4.5 Training Overhead Comparison

Flickr30K Processing Time

Figure 4 compare the training time on Flickr30K for finetuning CLIP, NPC, GLP and our method (all methods train CLIP ViT-B/32 for 5 epochs with a batch size of 256.). While our RCR introduces inevitable overhead compared to vanilla CLIP for noise handling, it is significantly more efficient than other state-of-the-art re-weighting methods. Specifically, RCR is only about 2.2x slower than CLIP, whereas NPC and GLP are 4.7x and 7.4x slower, respectively. This efficiency stems from our method's non-iterative nature. GLP conduct costly iterative label propagation and NPC need additional gradient steps for weight-estimation. In contrast, our method the additional computational overhead of our method comes from NDCG calculations, which mainly involve similarity computation and ranking, both are highly parallelizable matrix operations.

T1: Jockeys racing horses down a T3: Two Moto GP racers are neck and neck going around a turn. S(I2,T3)=0.322, P(I2,T3)=0.987

T4: Two people are competing GLP weight=1.0, RCR weight=0.911

in a motorcycle race. S(I2,T4)=0.325, P(I2,T4)=0.993 GLP weight=1.0, RCR weight=0.837

Query

Two Moto GP racers are neck and neck going around a turn.

Two people are competing in a motorcycle race.

Top-5 Similar Image

4.6 Case Study

Figure 5 shows cases from Flickr30K with 20% injected noise. The upper panel compares different re-weighting strategies for two image-text pairs. For image I1, text T1 is more descriptively accurate than T2, since T2 omits the grasstrack context. Similarly, for image I2, text T3 is more precise than T4, as T4 fails to mention the cornering action.

Despite these semantic differences, the fine-tuned model assigns nearly identical cosine similarities to all four pairs: S(I1, T1) = 0.316 vs. S(I1, T2) = 0.324, and S(I2, T3) = 0.322 vs. S(I2, T4) = 0.325. In fact, the partially aligned pairs (I1, T2) and (I2, T4) receive marginally higher similarity scores. This miscalibration also misleads GLP [Li et al., 2025], which assigns the maximal weight 1.0 to all four pairs because its graph-based propagation is fundamentally anchored to these unreliable point-wise similarities.

Our RCR method successfully distinguishes the partially aligned pairs by inspecting the global neighborhood structure. Specifically, RCR assigns (I1, T1) a weight of 0.916 while down-weighting (I1, T2) to 0.827; likewise, (I2, T3) receives 0.911 whereas (I2, T4) receives 0.837. The lower panel of Figure 5 reveals why: when I2, T3, and T4 are used to retrieve images, the Top-5 results for I2 and T3 are both dominated by motorcycle-racing images featuring sharp cornering scenes. In contrast, the Top-5 results for T4 are less focused, mixing generic motorcycle images with other racing contexts. Because T3's retrieval ranking aligns more closely with I2's intra-modal neighborhood than T4's does, NDCG assigns T3 a higher consistency score. This example illustrates that RCR's reliance on relative ranking agreement, rather than absolute similarity magnitude, enables finer-grained discrimination between well-aligned and partially aligned pairs.

5 Conclusion

In this paper, we address the noisy correspondence problem in cross-modal retrieval by proposing a novel sample reweighting method based on ranking consistency. Unlike existing methods that heavily rely on the unreliable image-text similarity predicted by the model, our key insight is to assess the quality of an image-text pair by measuring the consistency between cross-modal and intra-modal retrieval results. This design makes the weighting signal inherently more robust as it leverages the global manifold structure rather than point-wise, noisy model outputs. Extensive experiments on Flickr30K, MS-COCO, and CC120K under various synthetic and real-world noise levels demonstrate that our method consistently achieves state-of-the-art performance. The ablation and case study further validate that our ranking consistency strategy is more effective compared to weighting strategies based on model prediction or loss distribution. In conclusion, by shifting the focus from absolute similarity scores to relative ranking consistency, we provide a robust solution for learning with noisy correspondence, offering a fresh perspective for building reliable cross-modal retrieval systems.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant No. 62276110 and in part by the fund of Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL). The authors would also like to thank the anonymous reviewers for their comments on improving the quality of this paper.