Seeking Proxy Point via Stable Feature Space for Noisy Correspondence Learning

1 Introduction

Cross-modal retrieval, a key task in mutilmodal learning, is gaining significant attention as a crucial cornerstone to achieve Artificial General Intelligence[Morris et al., 2024], which focuses on aligning and comparing data across modalities (e.g., images, text, audio) to enable efficient and accurate retrieval of diverse information. Most existing methods[Yang et al., 2024][Chen et al., 2020b][Hu et al., 2021] rely on contrastive learning, aiming to reduce the feature space distance between matching pairs (positive examples) to bridge semantic gaps. However, these methods often assume perfect alignment in training data pairs, which is unrealistic. Unlike unimodal tasks, annotating cross-modal data is costly, especially for large-scale, high-quality pairs, making it labor intensive and sometimes unfeasible. To address this, a more economical and efficient method has been proposed[Sharma et al., 2018][Jia et al., 2021]: directly crawling native cross-modal data pairs from the Internet. Yet, this method inevitably introduces N Noisy Correspondence[Huang et al., 2021b][Qin et al., 2022], that is, some data pairs that are inherently mismatched, significantly compromising the retrieval performance of the model.

∗Corresponding author

In recent years, the issue of Noisy Correspondence has garnered significant attention and has been extensively researched. Initially, NCR[Huang et al., 2021b] reformulates the binary hard labels of the original data into soft labels within the [0, 1] interval by estimating the relevance degree of each data pair and employs a soft-margin triplet loss constructed based on these soft labels. Subsequently, MSCN[Han et al., 2023] adopted meta-learning[Li et al., 2019a] to perform consistency correction. UGNCL[Zha et al., 2024] partitioned the data under the guidance of uncertainty. L2RM[Han et al., 2024], based on the Optimal Transport[Cuturi, 2013] theory, selects reliable samples from negative examples to compensate for the reduced training information caused by noise. However, noise data reweighted by soft labels only mitigate noise's negative impact without providing reliable supervision signals. Additionally, whether using the triplet loss[Faghri et al., 2018] or cross-modal InfoNCE[Hu et al., 2023] contrastive loss, the representation space lacks geometric stability, making it hard to ensure prediction consistency between image and text domains under Noisy Correspondence. This instability significantly degrades the retrieval performance.

To address the aforementioned issues, we propose a novel Noise Correspondence robust learning framework, Seeking Proxy point via Stable feature space (SPS). This framework leverages the Memory Effect[Arpit et al., 2017] of deep neural networks (DNN) to extract a high-confidence reliable set from the original dataset and further divides the remaining data into a quasi-clean set and a noisy set using posterior probabilities, enabling subsequent fine-grained learning strategies. Specifically, we begin by adequately mining the supervisory signals latent within the reliable set. By imposing intermodal cross-transformation consistency constraints and intramodal metric consistency constraints, we construct a stable joint feature space, effectively alleviating the inconsistency in predictions across modalities caused by Noise Correspondence. Subsequently, based on this stable representation space, we seek proxy points for the noisy pairs, enabling them to be accurately embedded into appropriate positions within the feature space, even if they are noisy. This provides additional reliable supervisory signals and eliminates the impact of Noise Correspondence. Furthermore, the quasi-clean pairs predominantly represent the partial matching problem. We address this issue by refining the internal consistency of quasi-clean pairs to achieve partial alignment. Finally, we integrate these constraints and employ a cross-modal bidirectional contrastive loss and co-teaching paradigm to achieve robust learning under Noisy Correspondence conditions. The main contributions and innovations of this paper can be summarized as follows:

• We propose a novel noisy robust learning framework, SPS, which effectively resolves the Noisy Correspondence problem through fine-grained data partitioning and stable feature space constraint learning.
• Our framework innovatively introduces a noise proxy representation learning method based on a stable feature space, eliminating the impact of Noisy Correspondence while providing reliable supervision signals for model training.
• Extensive experiments across multiple cross-modal retrieval tasks validate the effectiveness of SPS, demonstrating that it significantly outperforms existing methods, particularly under high noise rates.

2 RelatedWork

2.1 Cross-modal Retrieval

Cross-modal retrieval[Cheng et al., 2022], essential in multimodal learning[Huang et al., 2021a][Zolfaghari et al., 2021] and information retrieval[Hambarde and Proenca, 2023], enables mutual retrieval across modalities like images and text by bridging the modality gap and aligning features semantically. A common approach uses contrastive learning to create a shared embedding space, bringing similar cross-modal samples closer. Existing methods to improve retrieval include SCAN[Lee et al., 2018] (stacked cross-attention for imagetext similarity), VSRN[Li et al., 2019b] (GCNs for semantic reasoning), IMRAM[Chen et al., 2020a] (iterative matching with attention), and SGRAF[Diao et al., 2021] (similarity graph reasoning). However, these methods assume perfectly aligned training data and overlook Noisy Correspondence-mismatched image-text pairs common in real-world data due to collection and annotation challenges. When noise is present, model performance drops as mismatched samples are incorrectly aligned, corrupting the feature space.

2.2 Noisy Correspondence Learning

Noisy Correspondence Learning (NCL) is a novel paradigm addressing semantically mismatched or partially matched cross-modal data pairs, unlike traditional noisy label[Liu and Tao, 2016][Xia et al., 2020] learning focused on single-modality data (e.g., image classification). Introduced by[Huang et al., 2021b], NCL leverages DNNs' memorization effect to partition datasets and correct labels adaptively. Subsequent improvements include: DECL[Qin et al., 2022] (combining Cross-modal Evidence Learning and Robust Dynamic Hinge Loss), BiCro[Yang et al., 2023] (using bidirectional similarity consistency for soft labels), CREAM[Ma et al., 2024] (adapting InfoNCE loss to uncover consistency within mismatched pairs), L2RM[Han et al., 2024] (using Optimal Transport to filter reliable samples), and PC2[Duan et al., 2024] (pseudo-caption methods with oscillation for relevance correction). However, limitations persist: insufficient feature space stability due to noise-induced drift, and limited utilization of noisy data, as methods only re-weight rather than extract reliable supervision. Our proposed SPS framework effectively tackles these challenges.

3 Methodology

3.1 ProblemFormulation

In cross-modal retrieval, we consider a dataset D = {(Ii, Ti, yi)}Ni=1 of N samples, where each triplet (Ii, Ti, yi) consists of an image-text pair (Ii, Ti) and a binary label yi. The label yi indicates whether Ii and Ti are positively correlated (yi = 1) or uncorrelated (yi = 0). Uncorrelated examples can introduce erroneous supervisory signals, misleading the model during training and degrading its overall performance. Thus, our objective is to learn a robust feature embedding that faithfully captures the true relevance between images and text, even in the presence of mislabeled or partially incorrect pairs.

3.2 Vanilla Cross-modal Loss

The core of cross-modal retrieval lies in accurately measuring the consistency between different modalities. We employ modality-specific encoders f(·) and g(·) to map an image I and its corresponding text T into a shared feature space, with f(I) and g(T) representing their respective feature embeddings. The semantic similarity between image I and text T is measured as S(f(I), g(T)). To learn the encoders f(·) and g(·), we adopt a variant of the InfoNCE [Radford et al., 2021] loss tailored for cross-modal scenario. This contrastive loss, derived frommutual information, aims to bring positive pairs closer while simultaneously pushing negative pairs as far apart as possible:

where H denotes the unidirectional cross-entropy function. Since cross-modal retrieval involves bidirectional queries, L consists of two symmetric terms H. P i2t i represents the matching probability of query Ii with respect to Ti in the image-text pair (Ii, Ti), P t2i i represents retrieval in the opposite direction, expressed as:

where τ is the temperature parameter, which is fixed at 0.07 in this experiment. When Noisy Correspondence occurs, there exist triplets (Ia, Tb, y∗ = 1) ∈ D, where Ia and Tb are not actually matched, but the corresponding label y∗ is incorrectly annotated as 1, which leads the model to erroneously align originally irrelevant cross-modal features.

3.3 Fine-grained Data Partitioning

Directly training the model on the raw dataset mentioned in 3.1 without preprocess will lead to overfitting on noisy samples, significantly degrading cross-modal retrieval performance. The memorization effect of DNNs, indicates that DNNs tend to prioritize memorizing clean training data before memorizing noisy training data. Leveraging this characteristic, we can achieve dataset partitioning by analyzing the loss distribution differences among sample pairs.

Given the raw dataset D, we compute the loss value Li for each sample using Equation (1):

Then we employ a two-component Gaussian Mixture Model (GMM)[Li et al., 2020][Permuter et al., 2006] to fit the probability distribution of the loss values Li:

where K = 2, and πk and N(Li|µk, σk) represent the mixing coefficient and probability density function of the k-th component with parameters µk > 0, σk > 0. We employ the Expectation-Maximization (EM) algorithm to ensure the convergence of the GMM. Subsequently, we calculate the probability pi that Li belongs to the component k′ with a smaller mean loss (i.e., the probability that (Ii, Ti) is a positive pair):

Given that the construction of a stable feature space highly depends on reliable data, we design a fine-grained data partitioning strategy. Specifically, we use the posterior probability pi to approximate the matching degree of iamge-caption pairs. By setting thresholds ϵ1 and ϵ2, the original training set D is meticulously divided into three subsets: the reliable set Dre, the quasi-clean set Dqc and the noisy set Dn:

For the quasi-clean set Dqc, we treat it as partially matched samples, remove the original labels, and use the adjusted consistency coefficients ˆ yqc as the new labels:

The noisy set Dn is identified as mismatched sample pairs, and their original labels are discarded entirely.

To address the issue of bias accumulation in single-model training, which is common in noisy label learning, we improve the co-teaching [Yu et al., 2019][Han et al., 2018] framework by training two networks Ma = {fa, ga, Sa} and Mb = {fb, gb, Sb} with identical architectures simultaneously and adjusting the data interaction mechanism: at the batch level, only the data partitioning strategy is shared, rather than the raw data. This approach enhances training efficiency while adaptively correcting training errors and avoiding bias accumulation in a single model.

3.4 Seeking Proxy Point via Stable Feature Space

Traditional contrastive loss functions, such as InfoNCE and Triple Loss, primarily construct the feature representation space from two perspectives: alignment and uniformity[Wang and Isola, 2022][Pu et al., 2022]. These loss functions aim to maximize the alignment of positive sample pairs while simultaneously pursuing a uniform distribution of features on the hypersphere, with the goal of preserving the maximum amount of information. However, in the cross-modal retrieval tasks we study, due to the presence of noise and the inherent modality gap[Liang et al., 2022], simply pursuing alignment and uniformity struggles to effectively address this challenge[Jiang et al., 2023]. In such cases, the constructed feature space often suffers from prediction inconsistency[Goel et al., 2022] issues owing to the lack of structural stability. When Noisy Correspondence are introduced, the model erroneously align irrelevant features, further destabilizing the model structure and exacerbating this problem ,as illustrated in Figure 1. Inspired by this, we propose a novel framework that addresses the NC problem by constructing a stable cross-modal feature space, enabling proxy point learning for the noisy set and partial alignment for the quasi-clean set.

Stable Feature Space Constraint

For data pairs identified as clean samples with high confidence, we should fully exploit the supervisory information they contain. Additionally, in the cross-modal joint embedding space, the feature representations of different modalities should maintain metric consistency, without significant discrepancies. Based on these two intuitive insights, we introduce an intermodal cross-transformation consistency constraint Lcro and an intramodal metric consistency constraint Lmet on the reliable set Dre, aiming to construct a stable joint feature space.

The constraint Lcro ensures that the cross-transformation between clean sample pairs remains consistent, which contributes to achieving a stable geometric structure. It is formulated as:

The constraint Lmet requires that the metric differences within each modality should be consistent. It is formulated as:

where S(Ii, Ij) denotes S(f(Ii), f(Ij)), and S(Ti, Tj) denotes S(g(Ti), g(Tj)), Bre represents a batch sampled from Dre, [x]+ = max(x, 0) and ∥ · ∥2 denotes the L2 norm. Following[Wang et al., 2023], we utilize a margin parameter α to control the strength of regularization.

Proxy Representation Learning

Unlike previous approaches that widely adopted re-weighting strategies for noisy data, as discussed in the 1, such methods merely mitigate the negative impact of noisy data on the model, essentially providing low-quality and unreliable supervisory signals. Instead, we propose to search for proxy points for noisy data based on the stable feature space constructed in 3.4, thereby providing reliable supervisory signals.

Without loss of generality, we use the image as a query to find a proxy caption, while a similar method is applied when using the caption as a query to find a proxy image. Specifically, given a noisy data pair (In, Tn) ∈ Dn, we first discard the caption Tn. Then, using the image In as a query, we search for the image ˙ I with the highest similarity to In within the current batch Bre, along with the corresponding caption ˙T of ˙ I. This process can be formulated as:

yielding a clean triplet ( ˙ I, ˙T, y = 1). Subsequently, we derive the proxy label yp based on the similarity S(In, ˙ I ):

where γ and β are hyperparameters. For demonstration purposes, we can define the proxy radius rp as rp = 1 - yp, as illustrated in the figure 2. Next, we treat ˙T as the proxy caption for the image I, constructing a new triplet (I, ˙T, ˆ y = yp).

These triplets, formed by noisy samples and proxy points, constitute a new dataset ˆDn, which is used for more robust training. The loss function for the noisy set is defined as:

where Bn is a batch sampled from ˆDn. Consequently, even noisy data initially provided as mismatched pairs can be accurately embedded into appropriate positions within the stable feature space, determined jointly by the proxy point and the proxy label yp.

Partial Alignment

The quasi-clean sample pairs primarily reflect partial matching issues. Based on the previously constructed stable feature spave, we only need to adjust the consistency coefficients according to the internal similarity of the quasi-clean sample pairs to achieve partial alignment. Given a quasi-clean sample pair (Ii, Ti, ˆ yqc) ∈ Dqc, like[Ma et al., 2024], we adjust ˆ yqc using the posterior probability pi and the binary hard label yi collectively:

where ˆ yi represents the predicted matching degree of the sample pair (Ii, Ti) by the model, obtained by averaging the bidirectional cross-modal matching probabilities:

Finally, we compute the loss for the quasi-clean set using the following formula:

where Bqc denotes a batch sampled from Dqc.

3.5 Global Training Objective

Before implementing co-teaching, we need to perform Warmup training for models Ma and Mb to achieve preliminary parameter convergence. Details are provided in the Appendix. For the fine-grained partitioned reliable set Dre, we minimize the following loss function using batch data Bre ⊂ Dre:

In summary, the overall loss function of the SPS method can be expressed as:

where λ1 > 0 and λ2 > 0 are hyperparameters used to balance the weights of the two spatial constraints, thereby achieving optimal performance. The entire framework of SPS is illustrated in the figure 2.

4 Experiments

4.1 Datasets and EvaluationMetrics

We employ three widely used cross-modal datasets to validate our proposed framework.

• Flickr30K [Young et al., 2014] consists of 31,000 images sourced from the Flickr platform, with each image annotated by five distinct captions, yielding a total of 155,000 image-text pairs for training purposes. Following [Lee et al., 2018], we allocate 1,000 images for validation, 1,000 images for testing, and utilize the remaining images for training.
• MS-COCO [Lin et al., 2014] contains 123,287 images, each paired with five captions, resulting in a total of 616,434 image-text pairs for training. We divide the dataset into 566,435 pairs for training, 25,000 for validation, and 25,000 for testing.
• Conceptual Captions [Sharma et al., 2018] is a largescale dataset comprising 3.3 million image-text pairs, which inherently exhibit real-world Noisy Correspondence challenges. Each image is paired with a single caption. Following [Huang et al., 2021c], we utilize a subset of the dataset, CC152K, for our experiments. Within CC152K, 150,000 images are designated for training, while 1,000 images each are reserved for validation and testing.

Given that Flickr30K and MS-COCO are meticulously annotated, we simulate Noisy Correspondence by randomly shuffling the captions of training images at predefined noise ratios. In contrast, Conceptual Captions, being automatically curated from the Internet, inherently contain approximately 3% to 20% of native mismatched noise, thus requiring no additional artificial shuffling. Following [Huang et al., 2021b], the retrieval performance is assessed using the recall at K (R@K) metric, which quantifies the percentage of relevant items correctly identified within the top K retrieved results. Our experimental evaluation includes R@1, R@5, R@10, as well as the cumulative recall scores RSum for bidirectional matching tasks.

4.2 Implementation Details

The proposed SPS is a universal Noisy Correspondence robust framework that can be directly applied to most existing cross-modal retrieval models. We adopt the widely used cross-modal retrieval model SGR[Diao et al., 2021] as the backbone network, integrating SPS to enhance its robustness to noise. In all our experiments, we use the Adam [Kingma, 2014] optimizer with default parameters for updating the model parameters. To better control the learning progress, we initially train the model exclusively on the reliable set and then gradually incorporate the quasi-clean set and the noisy set into the training process. We select the best checkpoint on the validation set to evaluate performance on the test set. All experiments were conducted on Linux using NVIDIA A100 GPUs.

4.3 Comparisons with State-of-The-Art

In this section, we validate the effectiveness of our method on both artificially noisy and natively noisy datasets. For the former, we report experimental results for noise rates of 20%, 40%, and 60%. The baselines include standard cross-modal retrieval model that lack robustness to noise, SCAN[Lee et al., 2018], VSRN[Li et al., 2019b], IMRAM[Chen et al., 2020a], SAF and SGRAF[Diao et al., 2021], noise-correcting methods NCR[Huang et al., 2021b] and DECL[Qin et al., 2022], and recently proposed noiseresistant approaches BiCro[Yang et al., 2023], L2RM[Han et al., 2024], PC2[Duan et al., 2024], and CREAM[Ma et al., 2024].

Results on Simulated Noise

Following [Huang et al., 2021b], we adopt two evaluation protocols to validate the performance on MS-COCO: 5-fold cross-validation on 1,000 test images (referred to as MSCOCO 1K), and evaluation on the full 5,000 test images (referred to as MS-COCO 5K). Due to space limitations, Table 1 only reports the bidirectional retrieval results on Flickr30K and MS-COCO 1K compared to recent models. For the complete experimental results, please refer to the appendix. The results for MS-COCO 5K are reported in Table 3. The experimental results demonstrate that SPS significantly outperforms existing methods in terms of robustness to Noisy Correspondence, achieving notably higher RSum scores than all baselines. Specifically, SPS outperforms the previous best baseline, CREAM, on Flickr30K and MS-COCO at noise ratios of 20%, 40%, and 60% by margins of 4.8, 3.8, and 6.7 on Flickr30K and 2.5, 5.3, and 7.3 on MSCOCO, respectively. Notably, as the noise rate increases to 60%, SPS maintains high stability, particularly on MS-COCO 5K, where it surpasses the second-best method by 13.5, demonstrating a significant improvement.

Results on Inherent Noise

To further validate the noise robustness of SPS in realworld application scenarios, we report results on the CC152K dataset, as shown in Table 2. According to the results, SPS achieves the best performance with an overall RSum score of 376.3. Compared to the backbone network SGR and the robustness method NCR, SPS achieves improvements of 9.9% and 4.8%, respectively. The experiments demonstrate that SPS effectively handles both simulated and real-world noisy environments.

4.4 Ablation Study

In this section, we conduct ablation experiments on the proposed SPS to validate the contribution of each major module within the framework, including WarmUp , Space Constraint (SCT), Lqc , and Ln . All experiments are performed on Flickr30K with a 40% noise rate, and to ensure a fair comparison, the same parameter settings are used across all experiments. The results are reported in Table 4. From the results, we observe that the best noise robustness is achieved only when all components are included, which fully demonstrates the effectiveness of each module.

4.5 Analytical Experiments

We analyze the impacts of hyper-parameters ϵ1 and ϵ2 under a 60% noise rate on Flick30K, as illustrated in the Figure 4 . It can be observed that the best retrieval performance is achieved when the threshold ϵ1 is set to a relatively strict value, specifically ϵ1 = 0.99. We attribute this to the fact that constructing a stable feature space highly depends on reliable positive pairs. Therefore, when the dataset permits, it is reasonable to impose stricter requirements on the data partitioned into the reliable set.Additionally, for the parameter ϵ2, experimental results indicate that maintaining a moderate value, specifically ϵ2 = 0.5 yields the best performance. This is intuitive, as it effectively distinguishes the quasi-clean set from the noisy set. Next, we investigated the performance of SPS in terms of dataset partitioning accuracy. Specifically, we observed the distribution changes of Ground Truth (GT) (i.e., truly clean pairs) in the Flickr30K dataset with different noise rates (20%, 40%, and 60%) throughout the training process, as shown in Figure 3. This includes the number of GT samples classified into the reliable set (denoted as GT Count) and the proportion of GT samples in the noisy set (denoted as GT Rate). The results demonstrate that SPS can accurately partition the data even as the noise rate increases, which aligns with the superior retrieval performance observed in the previous experiments.

5 Conclusion

In this paper, we investigate a relatively underexplored issue in the field of cross-modal retrieval: the Noisy Correspondence problem. To address this challenge, we propose a reliable set-driven approach to construct a stable feature space. Building on this foundation, we seek proxy points for noisy data to enable reliable feature learning. Extensive experiments on several cross-modal datasets demonstrate the effectiveness of the proposed method. In the future, we will explore the implementation and improvement of this framework in other cross-modal domains.

Acknowledgments

This work was supported in part by the National Key Research & Development Program of China under Grant 2022YFA1004100, the Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of China (ZYGX2022YGRH009 and ZYGX2022YGRH014), and supported by National Natural Science Foundation of China under Grant 62476048.