Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Models

1 Introduction

In recent years, Vision-Language Models (VLMs) have achieved significant zero-shot performance advancements in multimodal tasks such as retrieval [Huang et al., 2017; Huang et al., 2025], classification [Metzen et al., 2023; Novack et al., 2023] and segmentation [Xu et al., 2022; L¨uddecke and Ecker, 2022], establishing themselves as a foundation for multimodal tasks. However, despite their impressive performance, VLMs still face notable challenges in handling compositional reasoning (CR) tasks [Doveh et al., Aligned:

Figure 1: CLIP scores of the image and corresponding descriptions. The "Aligned" descriptions correctly reflect the image content, while the "Unaligned" descriptions represent incorrect descriptions of the image.

2023a; Thrush et al., 2022]. CR tasks demand that models accurately understand and reason about fine-grained relationships among objects, attributes, and actions in images or texts. For instance, they must be able to distinguish sentences differing only in word order or accurately identify objects in images that have similar colors but different attributes. Such tasks go beyond the need for coarse-grained semantic understanding, requiring precise reasoning in complex scenarios.

Yuksekgonul et al. [Yuksekgonul et al., 2022] highlight the limitations of existing VLMs in CR tasks. They found that current state-of-the-art VLMs often perform poorly in distinguishing semantic differences caused by subtle changes in word order or attribute variations. As demonstrated in Fig. 1, the CLIP scores show that, in some cases, the unaligned descriptions receive higher scores, indicating that VLMs still struggle to distinguish fine-grained semantic differences in object relationships and attributes. To address this issue, researchers attempted to enhance models' ability to discern complex semantic relationships by generating hard negative samples. These hard negative samples are semantically similar to the positive samples but have subtle semantic differences. By generating these challenging samples to fine-tune the model, its performance on CR tasks is improved.

However, current methods for incorporating hard negative samples into contrastive learning face several issues. First, some methods focus solely on hard negatives in the text modality, overlooking equally important hard negatives in the visual modality [Huang et al., 2024], resulting in insufficient training of the image encoder and thereby affecting the overall model performance. Second, these methods typically treat all negative samples equally, without considering their varying difficulty levels [Zhang et al., 2024c; Zhang et al., 2024a]. This prevents the model from effectively capturing subtle differences in hard negatives during training, thus hindering its ability to learn fine-grained distinctions. Additionally, these methods often neglect the adjustment of alignment for positive samples, which may cause the model to align only simple positive pairs while struggling to align more challenging ones.

To address these issues, we propose Adaptive Hard Negative Perturbation Learning (AHNPL) to enhance VLMs' performance in CR tasks. Our AHNPL method explicitly maps the subtle semantic variations of text-based hard negatives into the visual space, generating corresponding image embeddings for these negative texts, thereby improving the model's ability to discern subtle differences in the visual modality, which ultimately enhances the overall performance of the model. Moreover, AHNPL introduces dynamic hard negative contrastive learning, comprising a multimodal hard negative loss and a dynamic margin contrastive loss. The multimodal hard negative loss reduces the similarity between negative and positive samples across both text and visual modalities, improving the model's ability to distinguish hard negatives. The dynamic margin contrastive loss adjusts the contrastive margin based on sample complexity, enabling the model to focus more on challenging negatives and positives, which improves the alignment quality of challenging sample pairs.

Overall, our contributions are as follows:

• We propose a novel method that generates image-based hard negative samples by mapping subtle semantic shifts from text-based negatives into the visual domain, finetuning the model to enhance its overall performance.
• We introduce a dynamic hard negative contrastive learning approach. This includes a multimodal hard negative loss to distinguish hard negatives across text and image, and a dynamic margin loss that adapts to sample difficulty, focusing on challenging examples.

2 RelatedWork

Contrastive Learning. The objective of contrastive representation learning is to learn representations that are close to each other for similar samples and distant from each other for dissimilar samples. As one of the representative examples, CLIP [Radford et al., 2021] has become a milestone in this field. CLIP is a Transformer-based [Vaswani et al., 2017] model that consists of an image encoder and a text encoder, which are trained simultaneously. The objective is to maximize the cosine similarity of the image and text embeddings from the correct image-text pairs and to minimize the similarity between the incorrect pairs. A batch of N training samples (i.e., matching image-text pairs) results in a similarity matrix for each image-text combination. The main diagonal indicates the correct pair matches; the remaining entries correspond to negative entries. The InfoNCE loss is applied to the N × N similarity score matrix, and through contrastive learning, the image and text encoders are more effectively aligned across the two modalities, thus enhancing the performance of VLMs in multimodal tasks.

Compositional Reasoning. CR in vision and language tasks evaluates a model's ability to understand and manipulate complex ideas by breaking them into simpler components and recombining them in new ways. While VLMs excel at handling multimodal data, research shows they struggle with analyzing complex information, such as object relationships. Thrush et al. [Thrush et al., 2022] first identified this issue, demonstrating that VLMs often fail to distinguish semantic differences caused by changes in word order, with performance sometimes no better than random guessing. This type of task, which tests a model's ability to understand finegrained semantic structures, is referred to as CR. Many researchers have proposed methods of generating negative samples to fine-tune models to improve their performance in CR tasks. For example, Doveh et al. [Doveh et al., 2023a] introduced DAC, which fine-tunes the model to improve its performance in CR tasks by using a Large Language Model (LLM) to generate texts of different scenes. Zhang et al. [Zhang et al., 2024b] proposed CE-CLIP, which refines contrastive objectives with diverse negative samples to enhance semantic understanding. While effective, we believe current methods haven't fully explored hard negative mining, as they overlook the importance of image-based negative samples and fail to adjust the alignment process of positive sample pairs. To address this issue, we propose AHNPL, which translates textbased hard negatives into the visual domain, generating semantically disturbed image-based negatives, and dynamically adjusts the training process based on the similarity of each positive sample pair.

3 Methodology

In this section, we provide a detailed introduction to our proposed AHNPL method. Fig. 2 shows an overview of the entire pipeline.

3.1 Hard Negative Generation

In contrastive learning, hard negatives are samples that are semantically similar to positive samples but have subtle semantic differences. For example, consider the caption: "A boy wearing a red hat is playing on the beach". A potential hard negative could be: "A boy wearing a blue hat is playing on the beach". While this hard negative accurately describes most elements in the image, it differs from the positive sample in the color of the hat. Including such hard negatives in the training process can help the model recognize subtle differences, thereby improving overall accuracy and performance.

To generate these hard negatives, we use the natural language processing tool Spacy [Honnibal, 2017] to parse the captions and assign part-of-speech tags to each word. Specifically, we generate two types of hard negatives. The first type involves swapping two nouns in a sentence to generate hard negatives. This aims to alter the relationships between entities, and thus train the model to learn how to distinguish between different entity relationships. The second type involves randomly masking words in the sentence based on their partof-speech tags (such as nouns, verbs, or adjectives), and then using the RoBERTa [Liu et al., 2019] model to predict and fill in the masked parts. This approach aims to generate hard negatives that are similar in context and sentence structure, which helps the model enhance its sensitivity to semantic changes and lexical variations.

3.2 Visual Perturbation with Semantic Shift

In the previous section, we identify keywords in the captions through part-of-speech parsing and construct textual negative samples. Furthermore, we design a method for generating visual negative samples to train the model's image encoder, thereby enhancing the model's overall performance.

Existing negative sample generation methods often overly focus on textual negative samples, neglecting the role of visual negative samples, which leads to insufficient training of the image encoder and subsequently affects the overall model performance. To solve this problem, we propose a visual perturbation with semantic shift method for generating image negative samples. Specifically, we first capture semantic changes by computing the deviation embedding between the original text and the negative text. Given the original text Torig and the negative text Tneg, we use a pretrained CLIP model to extract their high-dimensional embedding vectors eTorig and eTneg , respectively. These vectors represent the semantic information of the texts within the multimodal embedding space of the CLIP model. The deviation embedding

∆e is computed as follows:

where ∆e reflects the semantic shift from the positive caption to the negative caption in the text. This shift not only represents changes in specific words or phrases within the text, but also implies adjustments in the contextual semantic structure. By utilizing this deviation embedding, we can capture the subtle semantic changes in the text.

Next, we directly incorporate the generated deviation embedding ∆e into the image embedding space by adding it to the original image's embedding vector eIorig to create the embedding representation eIneg for the image negative sample:

This approach ensures that the generated negative samples maintain appropriate semantic relevance to the original image in the embedding space, while deviating from the original image in terms of semantics. The image negative samples generated by this method maintain semantic consistency with the textual negative samples, providing more challenging training examples for the model.

3.3 Dynamic Hard Negative Contrastive Learning

Existing contrastive learning methods have some obvious limitations. First, these methods fail to adapt their alignment strategy according to the varying difficulties of positive samples, specifically by failing to adaptively enhance the learning signal for hard positive pairs that exhibit lower similarity, which causes the model to overly focus on aligning simple positive samples while lacking the capability to handle more challenging positive pairs. Additionally, these methods typically not only treat all negative samples as equivalent, disregarding the difficulty levels of negative samples during training, but also fail to effectively leverage multimodal information to distinguish the fine-grained semantic differences between hard negative samples and their corresponding positive samples. This approach makes it difficult for the model to establish clear decision boundaries and leads to misjudgments when processing hard negative samples that are semantically highly similar to positive samples, thereby affecting the effectiveness of contrastive learning.

To address these limitations, we propose dynamic hard negative contrastive learning (illustrated in Fig. 3), which dynamically adjusts its learning strategy based on sample difficulty, specifically enhancing the model's ability to distinguish difficult samples, and also establishes crucial semantic differences between hard negative samples and their corresponding positive samples in both visual and textual modalities, further improving the model's capacity to discern fine-grained semantic distinctions between them.

Contrastive Loss Our method uses a contrastive loss, which is applied to a text and image pair (T, I) as input and consists of two components: (i) an image encoder eI = fv(I); (ii) a text encoder eT = ft(T). In this setting, the text-to-image similarity score is computed as:

As with most contemporary VLMs, we employ the contrastive loss as one of our losses for each batch:

where τ is a temperature parameter.

Multimodal Hard Negative Loss. To enhance the model's ability to distinguish negative samples, we specially introduce a negative sample loss term. By simultaneously handling negative samples from both visual and textual perspectives, we help the model more accurately distinguish these challenging negative samples, thereby improving the effectiveness of contrastive learning.

The visual negative loss is designed to enhance the similarity differences between the visual negative embeddings In and the original image I. Specifically, it is computed as:

where Ihs is the set of all such generated visual hard negative embeddings In for an original image I, and S(I, In) represents the similarity between the original image I and each visual hard negative In in the set Ihs. By addressing the subtle semantic differences between hard visual negatives, this loss encourages the model to better distinguish visually similar negative samples.

Next, the textual negative loss is defined. This loss enhances the similarity differences between the original text T and the hard negative texts Tn. The textual negative loss is given as follows:

where Ths is the set of all types of such textual hard negative samples Tn generated for an original text T, and S(T, Tn) denotes the similarity between the original text T and each textual hard negative Tn. This loss encourages the model to distinguish the original text from semantically similar hard negative texts by minimizing their similarity.

Finally, the total negative loss is obtained by combining the visual and textual negative losses:

By integrating the losses for both textual and visual negative samples, this approach aims to comprehensively reduce the similarity between negative and positive samples across both modalities, thereby enhancing the model's ability to distinguish negative samples. This not only improves the model's performance in cross-modal alignment tasks but also boosts its robustness by focusing on difficult-to-distinguish negative samples in both image and text domains.

Dynamic Margin Contrastive Loss. In contrastive learning, the alignment of positive samples and the distinction of negative samples are key to improving model performance. Existing methods typically use a fixed margin (Margin Loss) threshold to adjust the similarity between positive and negative samples, but they still perform poorly when handling samples of varying difficulty.

When it comes to positive samples, such fixed margin approaches actually hold the assumption that all samples have the same level of difficulty. As a result, when handling challenging samples, the model tends to treat them the same as simple samples, making it difficult to capture the subtle semantic relationships, ultimately reducing the model's ability to align positive samples effectively. For negative samples, these methods generally introduce them into the contrastive learning framework without considering the variations in their difficulty. Although hard negatives are semantically similar to positive samples, they are characterized by subtle semantic differences. This subtle semantic difference makes it challenging for the model to effectively distinguish between positive samples and hard negatives, thereby limiting the model's performance.

To effectively enhance the model's capability in aligning hard positive samples, we introduce a learnable parameter a, which is initialized as a random value from a standard normal distribution. We set a lower bound of 0.2 for a to ensure it maintains a positive value throughout the entire training process. When the similarity between positive samples is low or there are significant semantic differences, the model automatically increases its focus on these challenging positive pairs, thus improving alignment accuracy.

Specifically, we compare the similarity S(I, T ) of the positive sample pairs with the learnable parameter a to adjust the alignment strategy for positive samples:

This loss function ensures that positive samples with greater difficulty receive more attention during training, enabling the model to learn more effectively from challenging positive samples, thereby improving its overall performance in aligning positive samples.

To enable the model to better distinguish difficult negative samples, we introduce an adaptive threshold updating mechanism in margin loss based on the difficulty of the samples. The specific loss function is defined as follows:

where M tn is the adaptive threshold computed for each textual hard negative Tn in the training step t. This threshold is computed as follows:

where B represents the set of samples in the current training batch, St-1 (I, T) and St-1 (I, Tn) represent the similarity between image I and the positive text T, as well as the similarity between the image I and a textual hard negative Tn, respectively, in the previous training step.

The final margin loss combining both the negative margin loss and the positive margin loss is defined as:

This adaptive threshold updating strategy allows the model to dynamically adjust its learning strategy during the training process, ensuring effective alignment of positive samples and proper distinction of negative samples.

The final loss function can be expressed as follows:

4 Experiments

4.1 Experimental Settings

All experiments are conducted using the PyTorch framework on a single NVIDIA A40 GPU and an Intel ® Xeon ® Gold 6330 CPU, running on Ubuntu 22.10. During the training phase, we initialize a pretrained CLIP model and fine-tune it on the MSCOCO [Lin et al., 2014] dataset for 10 epochs with a batch size of 128. The learning rate is set to 2 × 10-5 , and weight decay is set to 0.1.

4.2 Dataset

Training Datasets. The CLIP model is originally pretrained on 400 million image-text pairs sourced from the web, and we continue to fine-tune the model based on this pretraining. In our experiments, we use the widely adopted cross-modal text-image retrieval dataset MSCOCO [Lin et al., 2014]. MSCOCO is chosen for its rich annotations of diverse objects, attributes, and relationships, which enhance the model's semantic understanding in multimodal tasks.

Evaluation Dataset. We evaluate our method on several vision-language compositional benchmarks: ARO, VALSE [Parcalabescu et al., 2022], and SugarCrepe [Hsieh et al., 2023] (a bias-mitigated version of CREPE [Yu et al., 2023]). Each test example in these datasets includes an image along with a corresponding correct description and a modified incorrect description. The model's task is to determine which description is correct.

• ARO [Yuksekgonul et al., 2022]. This is a CR benchmark that includes positive or negative captions and evaluates sensitivity to word order. Word-order negative sentences are created by reordering words, which changes sentence semantics in attributes, relationships, and word order meaning.
• VALSE [Parcalabescu et al., 2022]. This benchmark evaluates VLMs' understanding of linguistic phenomena. It includes six tests on structures: morphological syntax, verb-argument structure, and word order. These tests require models to accurately link visual elements to linguistic descriptions, assessing visual understanding and alignment of these phenomena.
• SugarCrepe [Hsieh et al., 2023]. This benchmark uses LLMs to generate fluent and meaningful negative samples, avoiding biases introduced by traditionally used rule-based templates. It employs an adversarial improvement mechanism to minimize evaluation biases and ensures the reliability of results.

4.3 Overall Results

We present the evaluation results for the ARO and VALSE benchmarks in Tab. 1, and Tab. 2 shows the evaluation results on the SugarCrepe benchmark. As a widely used evaluation benchmark in this field, VALSE provides a solid basis for comparing our model against various mainstream models. Our AHNPL demonstrates effective improvements over all methods that utilize hard negatives, achieving excellent results across all benchmarks.

As shown in Tab. 1, our method achieves new state-ofthe-art (SOTA) results on the VALSE dataset, with an average matching score of 75.9%, outperforming the current SOTA model CE-CLIP by 3.4%. Other models, such as NegCLIP, also perform well with a score of 71.6%, but our model still leads by 4.3%, particularly excelling in the quantifiers and clean subtasks, where it outperforms CE-CLIP by 5.4% and 3.8%, respectively. On the ARO benchmark, AHNPL outperforms CE-CLIP in the ARO-Relation task, indicating an improvement in relationship understanding. In the AROAttribute task, AHNPL slightly surpasses CE-CLIP, suggesting that our model is comparable to the current SOTA model in attribute understanding.

SugarCrepe is a debiased dataset that effectively avoids the biases introduced by traditional rule-based negative sample generation, making it a benchmark of relatively high difficulty. We choose only the best-performing and most representative models for evaluation on this benchmark. On the SugarCrepe benchmark, AHNPL achieves higher overall average scores than CE-CLIP, particularly in the SWAP and ADD tasks for the Object category, where it outperforms CECLIP by 3.5% and 4.8%, respectively. These results further demonstrate that our model has effective advantages in handling complex semantic relationships and reducing language bias.

4.4 Ablation Study

We conduct ablation experiments on multiple enhanced versions of the CLIP base model to evaluate the contribution and effectiveness of each component in our method. The specific impact of different loss functions on model performance is shown in Tab. 3. First, the results indicate that the introduction of hard negative samples effectively improves the model's performance, with an improvement of up to 11.4% compared to the zero-shot CLIP, highlighting their key role in contrastive learning. Furthermore, compared to merely introducing negative samples into contrastive learning, each individual loss function we introduce shows varying degrees of performance improvement across all benchmarks. When all the loss functions are combined, the model achieves optimal performance. This improvement is fully reflected in our final model, demonstrating the superiority of the method in extending and enhancing contrastive learning objectives.

4.5 Case Study

We present several case studies illustrating the performance of CLIP and AHNPL across four subtasks of the SugarCrepe benchmark, as shown in Tab. 4. SugarCrepe uses LLMs to generate fluent captions with common sense, thereby challenging VLMs to distinguish negative captions effectively. In the "Swap Object" task, where models must understand object relationships (e.g., "A painting of a vase with a sunflower on a table" vs. "A painting of a sunflower with a vase on a table"), AHNPL outperforms CLIP. In the "Swap Attribute" task requiring accurate identification of object attributes (e.g., "Four surfers are trying to catch a wave as they stand" vs. "Trying to catch four waves, surfers stand"), CLIP struggles, whereas AHNPL consistently selects the correct caption. For the "Replace Relationship" and "Replace Attribute" tasks, which involve subtle distinctions (e.g., "A man is flying a kite at the beach" vs. "A man is holding a kite at the beach" or "Three teddy bears laying in bed under the covers" vs. "A single teddy bear laying in bed under the covers"), AHNPL effectively handles these nuances through negative caption contrastive learning, outperforming CLIP.

4.6 Visualization

To demonstrate the clear effectiveness of our proposed visual perturbation strategy, a compelling visualization study is provided. Tab. 5 shows two illustrative examples, where each example generates textual negative samples using the methods in Section 3.1. Specifically, for each example, we extract eIorig , eTorig and eTneg using zero-shot CLIP, AHNPL at epoch 5 and 10 (training end), respectively. The visual negative embedding eIneg is then computed from these extracted embeddings via our proposed visual perturbation. We illustrate the cosine distances between key pairs of these four embeddings. The changes in cosine distances demonstrate that the visual perturbation strategy effectively pushes the original image features towards negative texts, improving feature distinguishability and thereby helping the model capture subtle positive-negative semantic differences.

5 Conclusion

This paper proposes AHNPL, enhancing VLMs' performance in CR tasks via Visual Perturbation with Semantic Shift and Dynamic Hard Negative Contrastive Learning. AHNPL transforms text-based hard negatives into semantically perturbed image negatives, effectively improving the overall performance. Additionally, Dynamic Hard Negative Contrastive Learning strengthens the model's ability to align positive samples and distinguish hard negatives. Experimental results show that AHNPL outperforms existing methods in downstream tasks, showing robust complex semantic understanding in multimodal scenarios. Future work will explore integrating large-scale knowledge graphs and implicit knowledge to enhance complex semantic relationship acquisition.

Acknowledgments

This work was supported by Humanities and Social Sciences Youth Foundation, Ministry of Education of the People's Republic of China (No. 24YJCZH135), and Science and Technology Projects of Henan Province (Nos. 242102211019 & 252102210027).