C3L: Content Correlated Vision-Language Instruction Tuning Data Generation via Contrastive Learning

1 Introduction

In recent years, significant advancements have been made in the field of natural language processing, with the emergence of Large Language Models (LLMs) revolutionizing the landscape of this field [Chung et al., 2022; Chowdhery et al., 2023; Touvron et al., 2023]. Leveraging the powerful reasoning capabilities of LLMs, researchers [Li et al., 2023c; Zhu et al., 2023; Liu et al., 2023c] have proposed integrating visual encoders with LLMs to construct multi-modal perception and reasoning systems. This integration empowers LLMs with the ability to perceive and process visual information, leading to the substantial strides of Large VisionLanguage Models (LVLMs).

In practice, most of the existing LVLMs adopt a twostage training paradigm [Liu et al., 2023c; Zhu et al., 2023; Liu et al., 2023b; Li et al., 2023c; Dai et al., 2023]. In the first stage, a substantial amount of image-text pairs are used to pre-train LVLMs, typically employing the image-text contrastive approach [Liu et al., 2023c; Liu et al., 2023b; Zhu et al., 2023]. The objective of this pre-training phase is to develop the fundamental cross-modal alignment capabilities of LVLMs.

In the second stage, a shift is made from traditional taskspecific fine-tuning methods to a more general approach [Li et al., 2023b]. Instead of relying on task-specific data for individual downstream tasks [Devlin et al., 2018; Raffel et al., 2020], models are now fine-tuned using high-quality VisionLanguage Instruction Tuning (VLIT) data. This approach aims to develop general abilities of the models to understand and follow various types of instructions while generating helpful, factual, and harmless responses. The primary approach [Liu et al., 2023c; Wang et al., 2023b] to generating VLIT data is through using GPT-4(V) [OpenAI, 2023] to reduce manual annotations. With the improving capabilities of open-source LVLMs, researchers are increasingly applying these models to generate VLIT data, enabling them to overcome the limited accessibility of GPT-4(V) [Wang et al., 2023a; Bai et al., 2023; Li et al., 2023b].

Although significant progress has been made, this paradigm still faces the following challenges: 1) A longstanding issue with multi-modal models is to overly depend on prior language knowledge [Goyal et al., 2017], leading them to ignore visual content. Therefore, when generating VLIT data using LVLMs, the models often fail to effectively focus on the visual information and instead heavily rely on prior language knowledge. As shown in Fig. 1, the resulting VLIT data (denoted in red) demonstrates limited correlation with the corresponding images. 2) Current LVLMs are primarily trained with an emphasis on developing strong reasoning capabilities rather than their abilities as data generators. In fact, a straightforward approach [Wang et al., 2023a] to address this issue is by introducing an additional training phase where the models are trained with high-quality VLIT data as ground truth labels. As evidenced in [Ranzato et al., 2015; Lee et al., 2020], models that are only exposed to correct samples suffer from the "exposure bias" problem, which will restrict generalization abilities of the models when encountering unseen samples. Therefore, this training paradigm hinders the effectiveness of turning the models into data generators.

To address the aforementioned challenges, we propose a new Content Correlated VLIT data generation via Contrastive Learning, which is called C 3L for short. In C 3L, we first apply the LVLM to generate a set of initial VLIT data. Then, to enhance the content relevance between VLIT data and images, we propose a novel content relevance module where Image Instruction Correspondence (I 2C) scores are computed based on whether or not the images are provided. On the other hand, due to the training paradigm that only utilizes high-quality VLIT data as ground truth labels lacks exposure to low-quality samples, we divide the initial VLIT data into two categories based on the I 2C scores. The data samples with high I 2C scores are considered as positive pseudolabels, and vice versa. By employing contrastive learning framework [Lee et al., 2020] with positive-negative pseudolabels, we can effectively alleviate the "exposure bias" problem and enhance the capability of the model as a data generator. Benefiting from the above methods, C 3L effectively improves the content relevance of the model-generated VLIT data with images and enhances the capability of the model as a data generator.

According to automatic evaluations, fine-tuned LVLMs using data generated by C 3L achieve comparable performance to the state-of-the-art models on four recent multi-modal benchmarks. More importantly, only 5k VLIT data samples generated by our C 3L are used to fine-tune these models, significantly reduces the computational cost. In summary, we make the following contributions:

1. We propose a new content relevance module that can model the content correlation between VLIT data and images. This module effectively improves the content relevance and the utilization of visual information.
2. We develop an advanced contrastive learning module for VLIT data generation, which applies generated samples as pseudo-labels to boost the data generation capacity of the LVLMs further. To the best of our knowledge, we are the first to explore contrastive learning on the VLIT data generation.
3. The LVLMs fine-tuned using data generated by our C 3L achieves comparable or even better performance on four multi-modal benchmarks (i.e., SEED [Li et al., 2023a], LLaVAW [Liu et al., 2023b], MMB [Liu et al., 2023d] and POPE [Li et al., 2023d]). Meanwhile, automatic measures and ablation studies all show the effectiveness of our method.

2 RelatedWorks

2.1 Large Vision-Language Models

Most current LVLMs typically comprise three main components [Hamadi, 2023; Gu et al., 2023]: visual encoder, LLM and projection layer to connect visual encoder and LLM. In order to connect the visual encoder and the LLM, a connectivity structure is employed [Dai et al., 2023; Liu et al., 2023c; Zhu et al., 2023]. Typically, the majority of LVLMs adopt a two-stage training paradigm, encompassing Vision-Language Pre-training (VLP) and VLIT. In the first stage, large scale image-text pairs are employed [Li et al., 2023c; Gan et al., 2022] to establish fundamental vision-language alignment of LVLMs. This process involves training models to comprehend the visual information in the images and generate captions that accurately depict the visual content. In the second stage, a significant corpus of instruction data generated by models [Bai et al., 2023; Wang et al., 2023b] is utilized to enhance the capacity of LVLMs in comprehensively understanding vision-language instructions and generating appropriate responses. Existing LVLMs tend to excessively rely on prior language knowledge [Guan et al., 2023]. Therefore, when employing them to generate VLIT data, the generated data exhibits limited content relevance with images. In order to alleviate this issue, we design a novel content relevance module to improve the content correlation between modelgenerated VLIT data and corresponding images.

2.2 VLIT Data Generation

In order to enhance the capabilities of LVLMs to understand and follow instructions, the VLIT phase based on VLIT data has been introduced [Li et al., 2023b; Hamadi, 2023]. To reduce the cost of human annotation, the generation of VLIT data primarily relies on automatic model generation [Li et al., 2023b]. This is achieved by utilizing closed-source, powerful large models such as GPT-4(V) [Liu et al., 2023c; Wang et al., 2023b]. Specifically, [Liu et al., 2023c] collects VLIT data based on the existing image-pair datasets. By prompting text-only GPT-4 [OpenAI, 2023], high-quality question-answer pairs are obtained and used as VLIT data (i.e., LLaVA-158k). Moreover, as generating VLIT data solely by prompting text-only models like GPT-4 would inevitably result in the loss of visual information, previous work [Wang et al., 2023b] proposes to leverage GPT4V [OpenAI, 2023] to generate VLIT data with the entire visual context. Recently, as open-source LVLMs continue to evolve, researchers [Bai et al., 2023; Wang et al., 2023a] have turned to generate VLIT data by LVLMs. [Bai et al., 2023] employs self-instruction to acquire VLIT data, which can be used to improve the image content comprehension ability of the LVLMs. These approaches contribute to further reducing the cost associated with assessing GPT-4(V). However, as current open-source LVLMs are primarily trained with an emphasis on developing strong reasoning abilities, directly applying the models to generate VLIT data may result in undesirable results. Different from the above methods, we introduce a new contrastive learning module to boost the data generation capacity of the LVLMs.

2.3 Contrastive Learning

Contrastive learning proposed in [Hadsell et al., 2006] has been demonstrated to be an effective method for visual feature extraction [He et al., 2020]. [Chen et al., 2020] shows that contrastive learning benefits from larger batch sizes and can boost the performance of self-supervised learning in computer vision tasks. Some studies have also applied contrastive learning to multi-modal sequence modeling tasks, such as image captioning [Dai and Lin, 2017] and visual question answering [Liang et al., 2020; Lai et al., 2023; Suo et al., 2023]. However, these studies primarily focus on utilizing contrastive learning methods to improve the multimodal reasoning abilities and robustness of models. In contrast, our contrastive learning module is applied to enhance the capability of the models in generating VLIT data that exhibits higher content relevance with images. By employing positive-negative pseudo-labels with the contrastive learning framework, our contrastive learning module can alleviate the

"exposure bias" problem [Ranzato et al., 2015] which hurts the generalization of the models and thereby transform the model into a more capable data generator.

3 Method

The aim of this paper is to generate VLIT data that exhibits high content relevance with corresponding images and enhance the capability of the model as a data generator. We achieve this by addressing two issues that prior works have ignored: 1) Due to the tendency of multi-modal models to overly rely on language prior knowledge [Goyal et al., 2017], the relevance between the model-generated data and images is limited. 2) Considering the low capacity of current LVLMs in generating data, VIGC [Wang et al., 2023a] introduces an additional training phase to transform the model into a data generator using high-quality VLIT data as ground truth labels. This training paradigm has been demonstrated to result in the "exposure bias" problem [Ranzato et al., 2015; Lee et al., 2020]. In this section, we introduce our Content Correlated VLIT data generation via Contrastive Learning (C 3L). As shown in Fig. 2, the C 3L comprises an augmented pipeline with content relevance module and contrastive learning module. Next, we would introduce our method in detail.

3.1 Conventional VLIT Data Generation Pipeline

VLITdata. Given the image Ii, the VLIT data is structured as question-answer pairs [Qj,Aj], where j ∈ {1, 2, ...,Nj} denotes the j-th question-answer pair associated with the image Ii.

Conventional pipeline. As in [Wang et al., 2023a], the LVLM that used for generating VLIT data is first trained with high-quality VLIT data as ground truth labels. Next, given a set of instructions and images, the enhanced LVLM is utilized to generate VLIT data which is in the form of questionanswer pairs [Qj , Aj ]. The instructions here are used to guide the enhanced LVLM in generating VLIT data. Finally, the data can be used to fine-tune LVLMs in order to enhance the general capabilities of the models.

Limitations. 1) The process overlooks the impact of language priors [Goyal et al., 2017] embedded in LLM counterparts of LVLMs, resulting in limited correlation between the generated VLIT data and the corresponding images. 2) The training phase only uses high-quality VLIT data as ground truth labels. As evidenced in [Lee et al., 2020], this training paradigm suffers from the "exposure bias" problem [Ranzato et al., 2015] and hinders the effectiveness of transforming the model into a data generator.

3.2 Content Relevance Module

To solve the first issue, we propose a novel content relevance module which computes the Image Instruction Correspondence Scores S(I 2C) based on whether the image is present or not.

Given a set of instructions and images, we first apply a widely used LVLM such as LLaVA [Liu et al., 2023c] to generate the initial VLIT data. Due to the influence of language prior knowledge, the initial dataset contains data samples that exhibit low relevance to the image content, as shown in Fig. 1. Next, for a given data sample [Qj,Aj] in our initial VLIT dataset, we set Qj and the corresponding image as inputs to the model. After obtaining the outputs of the model, we compute the probability value pvt of the outputs for each answer token in Aj , where t denotes the t-th token. Then, these probability values are concatenated to obtain the Visual Answer Scores S(A|V ).The S(A|V ) scores measure the responses of the LVLM to a specific VLIT data given the image. We can also compute the probability values pdt and obtain the Direct Answer Scores S(A) in the absence of the image. The S(A) scores quantify the responses of the model to a given VLIT data in the absence of an image. The final scores S(I 2C) is computed by using KL-divergence:

The S(I 2C) scores measure the differences in responses of the model when provided with and without an image. Based on the properties of KL-divergence [Bu et al., 2018], a low S(I 2C) means that the divergence between S(A|V ) and S(A) is small, indicating that the content relevance between the data sample and the image is low.

After obtaining the S(I 2C) scores for all initial VLIT data samples, we train the model by using 10 % data samples with higher S(I 2C) as ground truth labels, the standard crossentropy loss Lr is used during training.

3.3 Contrastive LearningModule

Due to the presence of the "exposure bias" problem [Ranzato et al., 2015] when training models solely with high-quality VLIT data as ground truth labels, we introduce an additional contrastive learning module in which the model also learns experience from low-quality data samples in the initial VLIT dataset. Following contrastive learning framework [Lee et al., 2020], we maximize the similarity between the anchor and positive pseudo-labels while minimizing the similarity between the anchor and negative pseudo-labels as follows.

In particular, given an image Ii, the data sample [Qj,Aj] with the highest S(I 2C) is selected and employed as the positive pseudo-label. We concatenate [Qj,Aj] and embed the sequence s using text encoder in the LVLM, denoted as es ∈ Rd×l where l represents the length of sequence. Affine transformation ξ with the ReLU and AvgPool is used to project es onto the latent space hs ∈ Rd . This process is denoted as:

where ξ(·) = AvgPool(ReLU(·)).

Similarly, the remaining data samples ˆs ∈ S corresponding to the image Ii are selected and employed as the negative pseudo-labels. The representations hˆs are computed using eq.2. Then, we input an instruction along with the image Ii into the model and obtain the anchor data y. Finally, we compute the contrastive loss Lc as follows:

where sim(·, ·) is a cosine similarity function.

3.4 Augmented VLIT Data Generation Pipeline

By using our content relevance module and contrastive learning module, we can construct an augmented VLIT data generation pipeline.

Initial data generation. Given a set of instructions and images, we first employ a widely used LVLM such as LLaVA [Liu et al., 2023c] to generate a set of initial VLIT data, wherein the data is in the form of question-answer pairs. Inspired by [Changpinyo et al., 2022], we use a simple twostep generation (i.e., first generate captions for the images and then use the LLM-counterparts of LVLMs to generate data based on captions) to further fill the gap between the modalities. Following [Wang et al., 2023a], the instructions are primarily categorized into three main classes: 1) conversation 2) detailed description 3) complex reasoning. The instructions are used to guide the model in generating VLIT data (e.g., Generate five in-depth reasoning questions and then answer them based on the image. or Generate five questions in order to describe the image and then answer them.).

Content relevancemodule. Due to the language priors embedded in LLM counterparts of LVLMs, the initial VLIT dataset contains data samples that exhibit low content relevance between the VLIT data and the corresponding images. By using our content relevance module, we can compute the S(I 2C) for all VLIT data samples based on whether or not the image is provided and train the model by using 10 % data samples with higher S(I 2C) as ground truth labels. The standard cross-entropy loss Lr is used during this training phase.

Contrastive learning module. Due to the presence of "exposure bias" problem, we introduce the contrastive learning module for models to learn experience from low-quality samples in the initial dataset. Specifically, given an image Ii, the data sample with the highest S(I 2C) is employed as the positive pseudo-label and the remaining data samples as negative pseudo-labels. By incorporating the positive-negative pseudo-labels and the contrastive learning framework, our module effectively enhances the capability of the LVLM to generate VLIT data.

Final data generation. The final 5kVLIT data is generated using the enhanced LVLM along with the two-step generation and can be employed to fine-tune other LVLMs, thereby enhancing the overall capacity of the models.

4 Experiments

4.1 Experimental Setting

Implementation details. We select two representative LVLMs (i.e., LLaVA [Liu et al., 2023b] and MiniGPT-4 [Zhu et al., 2023]) as our backbones to demonstrate the generality of our method. During the initial data generation phase, we utilize images from the COCO 2014 [Lin et al., 2014] training set and generate 20k initial VLIT data samples. When finetuning backbones using our final data, following [Liu et al., 2023c], we fix the weights of the CLIP [Radford et al., 2021] visual encoder during fine-tuning. The AdamW [Kingma and Ba, 2014] is used as our optimizer with the weight decay 1e5. We fine-tune all models with a learning rate of 2e-5 and batch size is set to 2. For MiniGPT-4, two 3090Ti GPUs are used while for LLaVA we use eight 2080Ti GPUs.

Benchmarks. To comprehensively evaluate the performance of LVLMs after VLIT using data generated by our C 3L, we select four benchmarks (i.e., SEED, MMB, LLaVAW and POPE) for evaluation.

1. SEED [Li et al., 2023a] consists of 19k multiple-choice questions to evaluate the image and video understanding capabilities of LVLMs across 12 different tasks. Following [Du et al., 2023b], we only evaluate the capacity of LVLMs on image-related tasks.
2. MMB [Liu et al., 2023d] develops a comprehensive evaluation pipeline which incorporates a novel CircularEval strategy to effectively evaluate the capabilities of LVLMs systematically. We use dev split of MMB for evaluation.
3. LLaVAW [Liu et al., 2023b] encompasses indoor and outdoor scenes, memes, paintings, sketches, etc, to evaluate the capabilities of LVLMs in more challenging tasks.

4) POPE [Li et al., 2023d] aims to effectively evaluate the common issue of hallucinations in almost all LVLMs. By constructing three types of sampling strategies along with a polling-based query method, they can accurately measure the severity of model hallucinations for different objects.

Generated dateset. As shown in Table 2, we conduct a statistical analysis of our final VLIT dataset. It is evident that our dataset of 5k instances is smaller in number compared to other VLIT datasets, such as LRV-400k [Liu et al., 2023a]. In Table 3, it can be observed that the performance of LLaVA7B [Liu et al., 2023c] when fine-tuned using our 5k data is competitive to the performance achieved when using LRV400k, suggesting the effectiveness of our C 3L.

4.2 Quantitative Evaluation

To assess the improvements in the overall capabilities of LVLMs fine-tuned using data generated by our C 3L, we compare these fine-tuned LVLMs with the state-of-the-arts models on four benchmarks in Table 1. The C 3L∗ denotes that VLIT data used to fine-tune the LVLM is generated by the other LVLM (e.g., MiniGPT-4 w/ C 3L∗ denotes that data is generated by LLaVA). We observe that LVLMs with C 3L achieve comparable or even better results on these four widely used benchmarks compared to other state-of-the-art models. More importantly, only 5k VLIT data generated by models is utilized to fine-tune them. On the SEED, LLaVA w/ C 3L outperforms the previous models with an improvement of 2.7%. This demonstrates the effectiveness of our C 3L in improving the content relevance between VLIT data and images, thereby further enhancing the image understanding capabilities of LVLMs. LLaVAW is designed to evaluate the capacity of LVLMs in more challenging tasks, we observe that MiniGPT-4 w/ C 3L achieves better performance than previous models on this benchmark. On the POPE, models with C 3L achieve comparable results with previous models which suggests that data generated by our C 3L exhibits higher content relevance with images, thus can be applied to improve the factualness in LVLMs.

4.3 Ablation Studies

In order to demonstrate the effectiveness of our method in improving the overall capacity of the LVLM including factualness, we conduct several ablation studies on the SEED and POPE. As shown in Table 4, in the first row, the baseline is established by fine-tuning LLaVA-7B [Liu et al., 2023c] using VLIT data generated with instructions and basic filtering. In the second row, it can be found that the performance on SEED and POPE exhibits an improvement of 0.9% and 1.2% respectively. This could be attributed to the effectiveness of our contrastive learning module in solving the "exposure bias" problem and enhancing the capacity of the model as a capable data generator. In the third row, we can find that our content relevance module plays a pivotal role, bringing a 2.6% improvement on SEED and a 1.9% improvement on POPE in comparison to the baseline model. These results prove that our content relevance module can enhance the correspondence between VLIT data and images, thus improving the overall capabilities of models when fine-tuned using the data. In the last row, it can be observed that additional improvements are achieved when the two modules work collaboratively, which demonstrate that our C 3L is remarkably effective in improving the content relevance of the modelgenerated VLIT data with images and enhancing the capability of the model as a data generator.

4.4 Alternative Data Selection Setting

We propose two alternative data selection settings and conduct several experiments to measure the reliability of our method.

Alternative data selection methods. To further demonstrate the effectiveness of our S(I 2C) in selecting positive/negative pseudo-labels, we compare our method with two other approaches (i.e., Self-Instruct [Wang et al., 2022] and VIGC [Wang et al., 2023a]) on SEED. As shown in Table 5, we conduct experiments by replacing the I2C scores with these two methods using two backbones (i.e., LLaVA7B and LLaVA-13B). For Self-Instruct, we apply the same

(a)

(b)

(c)

w/o 𝐶"𝐿:

Q:What is the purpose of the fire hydrant? A: The purpose of the fire hydrant is to provide a reliable and accessible source of water for firefighting operations.

w/ 𝐶"𝐿:

Q: How does the color of the main object in the image affect its visibility?

A: The fire hydrant is the main object in the image, the bright pink color of the fire hydrant makes it more noticeable and easily identifiable. This is important because fire hydrants need to be easily located in case of emergencies.

w/o 𝐶3𝐿:

Q: What are the potential challenges faced by the classic truck owner?

Answer: The truck owner may face several challenges in maintaining and preserving the classic truck, such as finding rare parts, ensuring proper restoration techniques, and addressing potential mechanical issues.

w/ 𝐶3𝐿:

Q: What role does the truck play in the image? A: The truck serves as an attraction at the event in the image. Its classic design and shiny appearance draw attention and create a sense of nostalgia, making it an engaging focal point for

attendees.

filtering and post-processing methods following [Wang et al., 2022] to identify low-quality data, the remaining instruction samples are considered as positive pseudo-labels. For VIGC, we generate data using two approaches: direct generation and iterative generation [Wang et al., 2023a]. The data generated by iterative generation is considered as positive pseudo-labels. The results show that selecting positive/negative pseudo-labels by S(I 2C) is more effective compared to other alternative data selection methods.

Alternative data selection proportions. In order to investigate the impact of the data proportions selected by content relevance module on the performance of the LVLMs (i.e., LLaVA and MiniGPT-4) fine-tuned with C 3L, we compare the performance of LVLMs on LLaVAW after fine-tuning with different data proportions. As shown in Fig. 3, the blue line denotes the performance of MiniGPT-4 changes with different data selection proportions, while the orange line corresponds to LLaVA. The results demonstrate that there is a marginal difference in the performance of LVLMs when finetuned with different data proportions. Therefore, we choose 10% data in the content relevance module in order to balance performance and cost.

4.5 Qualitative Evaluation

As shown in Fig. 4, we show some qualitative results generated by C 3L. Through overall comparison, the data generated by our C 3L exhibits higher content relevance with images. For example, in Fig. 4 (a), answering the question generated without C 3L does not require leveraging information from the image, as the fire hydrant is primarily used to supply water during fire incidents. On the contrary, answering the question generated by C 3L not only requires common knowledge about the fire hydrant but also necessitates a comprehensive understanding of the image. Additionally, the data generated by C 3L is also more detailed and factual. In Fig. 4 (b)-(d), the generated data containing the question about the man's posture and the answer about the role of the truck, which is "serves as an attraction at the event".

5 Conclusion

In this paper, we propose a new Content Correlated VisionLanguage Instruction Tuning Data Generation via Contrastive Learning (C 3L). We first design a new content relevance module to improve the correlation between VLIT data and images. Meanwhile, we propose a contrastive learning module, which utilizes the generated samples as pseudo-labels to boost the data generation capacity of the LVLMs. Furthermore, an augmented VLIT data generation pipeline is proposed by combining these two modules, which can be applied to generate VLIT data that exhibits high content relevance with images. According to automatic evaluations, fine-tuned LVLMs using data generated by our C 3L achieve comparable performance to the state-of-the-art models on four benchmarks and provides a new paradigm for our community.

Contribution Statement

Ji Ma and Wei Suo contribute equally to this work. Peng Wang is the corresponding author.

w/o 𝐶3𝐿:

Q: What is the surfer riding on?

• A: The surfer is riding on a surfboard, which is a specialized board designed for surfing. Surfboards come in various shapes, sizes, and materials, each tailored to specific types of waves and surfing styles.

w/ 𝐶"𝐿:

Q: What is the man's posture in the image? A: The man in the image is a surfer crouching down on his surfboard while riding a wave. The surfer is leaning forward on his surfboard, which helps maintain balance and control while riding the wave.

w/o 𝐶3𝐿: Q: How do the sheep's woolly coats help them adapt to their environment?

A: The sheep's woolly coats provide insulation and protection against the cold and harsh weather conditions.

w/ 𝐶"𝐿:

Q: What is the primary purpose of the sheep in the image?

A: The primary purpose of the sheep in the image is to graze on the lush green grass in the field, contributing to the maintenance and growth of the grass.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No.U23B2013), Shaanxi Provincial Key R&D Program (No.2021KWZ-03), and Natural Science Basic Research Program of Shaanxi (No.2021JCW-03).