From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping

1 Introduction

In the development of multi-modal learning, two recursive levels arise: (1) multi-modal association; (2) cross-modal generation. The former relies on multi-modal inputs and calculates the association scores for given multi-modal inputs through association expressions. Typical tasks include image classification, image-text retrieval, object detection, etc. The latter is to convert the input from one modality to other modalities, which requires a cross-modal transformation relationship to ensure that the same concept within different modalities can be represented accurately. Typical tasks include image-to-text and text-to-image generation.

∗Work done during internship at DAMO Academy, Alibaba Group.

†Corresponding author

Currently, Vision-Language Pre-training Models (VLPMs) represented by CLIP [Radford et al., 2021] and ALIGN [Jia et al., 2021] have been successful at the first level by advantage of the multi-modal association-based pre-training task: contrastive learning. With 400 million massive (image, text) training data, CLIP successfully models the association between vision modality and language modality. Benefiting from the diverse web data, CLIP has an extensive perception of open-world knowledge. Without fine-tuning, CLIP achieves an accuracy of 76.2% on ImageNet in the zero-shot setting and rivals or even surpasses the fine-tuning model on datasets with multiple domains [Radford et al., 2021]. Many multi-modal association-based works have benefited from the powerful zero-shot capability of CLIP: not only avoids the high collection cost of supervised data but also simplifies the deployment process.

The great success of CLIP at the association level has sparked the exploration at the generation level. However, since CLIP does not have a decoder architecture and a pretraining task for generation, it is not competent for the generation-based task. Nevertheless, the dominant performance of large-scale language models such as BERT [Devlin et al., 2018] and GPT [Radford et al., 2019] makes it possible to decode from the embedding space of CLIP. It is based on this arising the idea of zero-shot generation based on joint space: CLIP encodes pairs of image and text close enough in embedding space [Su et al., 2022; Wang et al., 2022;

Nukrai et al., 2022]. However, [Liang et al., 2022] has shown that CLIP encodes images and text into two separate subspaces and there is a significant gap between them as shown in Figure 2 (a) top. This means that the decoder is only effective in one modality. When transferring the decoder from one modality to another modality, the modality gap leads to the failure to accurately understand representations as shown in Figure 2 (b).

To eliminate the impact of the modality gap, a major problem is how to establish the transformation relationship between the two modalities. A natural idea is to model the relationship through a large amount of supervised data. However, this requires significant supervised data and training resources [Ramesh et al., 2022]. We argue that this relationship can be established by an unsupervised method through the association capability of CLIP. Based on this, we propose the K-nearest-neighbor Cross-modality Mapping (Knight), a text-only captioning method as shown in Figure 3. First, we collect the captions from the image-text and video-text datasets as the corpus. In the training phase, we first select captions from the corpus for training and use the CLIP similarity to retrieve the k-nearest-neighbor captions that are most similar to the training captions. Then, we use the CLIP features of the training captions to train the decoder by an autoregression loss. In the inference phase, Knight can be applied to both image and video captioning. For image captioning, we retrieve the k-nearest-neighbor captions that are most similar to the inference image. For video captioning, we average the retrieved results for each keyframe to achieve multi-frame input for the video. Knight makes the decoder only act in the text subspace, thus eliminating the effect of the modality gap as shown in Figure 2 (c).

We summarize the contributions as follows:

• We propose a text-only captioning method called Knight based on the unsupervised cross-modal mapping. The method achieves the representation mapping from vision
• modality to language modality, thus greatly alleviating the impact of the modality gap on cross-modal generation.
• We compare Knight with the other current zero-shot image and video captioning baselines. Experimental results show that Knight achieves state-of-the-art performance. We explore the possibility of employing CLIP association capacity to address the zero-shot generationbased tasks.

2 Background and RelatedWork

2.1 CLIP

By contrastive learning on a dataset of 400M (image, text) pairs, CLIP [Radford et al., 2021] models the association between the images and text. CLIP is widely used for zeroshot tasks such as classification, retrieval, etc [Radford et al., 2021]. The zero-shot performance is claimed to be close to or even better than fine-tuned models [Radford et al., 2021]. Many works have applied the zero-shot capacity of CLIP to specific application scenarios such as image segmentation [Xu et al., 2021], image generation [Ramesh et al., 2022], and object detection [Zhong et al., 2022].

2.2 Text-only Captioning

Mainstream methods for captioning tasks are mainly divided into two types: (1) extracting visual features typically using a pre-trained network and training a decoder that produces the final captions [Chen and Zitnick, 2014; Chen et al., 2017; Yang et al., 2019; Anderson et al., 2018; Luo et al., 2021]; (2) bridging the gap between vision and language by employing pre-training to create a shared latent space of images and text [Lu et al., 2019; Laina et al., 2019; Tan and Bansal, 2019; Li et al., 2020; Zhou et al., 2020; Zhang et al., 2021; Hu et al., 2022]. With the rise of CLIP, recent captioning methods use CLIP for reducing training time [Mokady et al., 2021] and improved captions [Shen et al., 2021; Cornia et al., 2021; Kuo and Kira, 2022].

However, all of the previous works require extensive training and large paired data that are hard to collect. To address this, [Gan et al., 2017] and [Zhao et al., 2020] have suggested style-guided captioning, but also employ training over paired data. OpenAI has conducted remarkable work in learning cross-modal patterns with the CLIP trained by a large number of resources [Radford et al., 2021]. This means that the effective exploitation of the pre-trained knowledge of CLIP can be free from the constraints of supervised data. [Tewel et al., 2022b] attempted to generate a caption with the highest CLIP similarity to a given image using a pre-trained language model. However, the pre-trained language model and CLIP's text encoder use different pretraining data and paradigms, making it difficult for the language model to generate high-quality captions that are relevant to images. Researchers realized that additional language data are needed for aligning the embedding space between the language model and CLIP, hence the rise of textonly captioning methods. [Su et al., 2022] made the language model fit the domain of CLIP by fine-tuning it with unsupervised corpus. However, the above works require the language model to provide candidate words and thus prove to have a severe language prior [Wang et al., 2022]. [Wang et al., 2022; Nukrai et al., 2022] proposed the idea of joint space that argues that CLIP encodes pairs of image and text close enough in embedding space, thus overcoming the problem of language prior by training the decoder in the language modality and transferring it to vision modality in inference phase.

2.3 Addressing the Modality Gap

The previous text-only methods argued that embedded text is relatively close to its corresponding visual embedding [Nukrai et al., 2022]. However, [Liang et al., 2022] demonstrates that images and text are embedded into two subspaces separately and a significant modality gap is between them as shown in Figure 2 (a) top. This gap limits the generation quality of previous methods. Although it is possible to learn the pattern between two modalities with a large amount of data, [Ramesh et al., 2022] demonstrates that the overhead of this process is huge. Nevertheless, we note that the effective cross-modal interaction of CLIP is under the association of paired data as shown in Figure 2 (a) bottom. This makes us realize that although the modality gap makes it hard to achieve cross-modal generation directly, it can be achieved by association indirectly.

3 Method

3.1 Preliminaries

Notations. We first explain the definition of text-only captioning method and the requirements for the training data. The supervised dataset Ds = {(x1, y1), . . . , (xn, yn)} consisting of n pairs with images or videos xi and reference captions yi = {c1i , . . . , c|yi| i }, where yi is a set of the captions that describe the xi from different perspectives, and cji denotes the jth caption of yi. The unsupervised data include unlabeled image or video dataset DIu = {x1, . . . , xi} and text datasets DT u = {y1, . . . , yj}. Traditional captioning methods use supervised dataset Ds for training, while text-only captioning methods only assume the availability of unlabeled dataset DT u .

CLIP is a VLPM with dual-encoder architecture. It consists of two independent encoders for vision and language modalities. Similarities between vision and language representations on large-scale image-text pairs are used to pre-train CLIP, bridging the gap between vision-language semantics in the embedding space of CLIP. The similarity is calculated as

where fV and fL are the image encoder and text encoder of CLIP respectively.

Unsupervised Language Modelling is a method for selfsupervised training on the unsupervised corpus. It learns the relationship of sentence context by the next-token prediction. The pre-training loss is the Maximum Likelihood Estimation (MLE) that calculated as

where θ denotes the parameter that needs to be optimized for the model M.

3.2 Knight

We consider a training caption y′ fromDT u . The Tt and T1∼n calculated as

where n is the size of training corpus. Then, we get the CLIP similarities S between Tt and T1∼n calculated as

Then we sort the S from large to small and get the T ′ = { Ts1, . . . , Tsk }, where Tsi is the i-th largest of the sorted result and k is a hyperparameter. Finally, we use the T ′ for autoregression training as following equation

Image Captioning

After training, we fix the parameters of the decoder. In the inference phase, we replace the input of the training caption with the inference image x and get the It = fV(x). We get the S as the following equation

Finally, we follow the training process to obtain T ′ and feed it into the decoder to get the generated captions as the equation (5).

Video Captioning

The above process of image captioning can be understood as a process of filtering and regrouping the information from knearest-neighbor captions. This process is similar to the generation of video caption: the decoder establishes the connection between the preceding and following frames and reasons out the combined information of these frames. Therefore, we argue that Knight can be transferred to video captioning.

Compared with the image, video is a set of multiple frames. We first extract the keyframes x = { x1, . . . , xm } of the inference video x, where m is the number of keyframes. Then, we get the I i t = fV(xi), where xi is the i-th keyframe. For each keyframe, we get the Si as the following equation

We sort each Si and select the k representations with the greatest similarity thus obtaining T ′ i . Finally, we get the T ′ for video as the following equation

Finally, we feed T ′ into the decoder to generate the captions as the equation (5).

4 Experiment

4.1 Experimental Settings

Evaluation Benchmarks. For the image captioning task, we conduct experiments on two widely used benchmarks: Flickr30k [Plummer et al., 2015] and MS-COCO [Lin et al., 2014; Chen et al., 2015]. And we set up the training, validation, and test splits according to the protocols provided by [Karpathy and Fei-Fei, 2015] for both datasets. For the video captioning task, we choose two video datasets: MSRVTT [Xu et al., 2016] and MSVD [Wu et al., 2017]. We use the captions in the training set as the training corpus and evaluate the methods on the test set, as in other text-only captioning works.

Implementation Details. For CLIP, we choose the Resnet50x64 architecture which encodes each image as a 1024-dimension vector. For the decoder, we choose the large vision of GPT-2 [Radford et al., 2019] with a 1280-dimension embedding space. To align CLIP and decoder on the representation layer, we use a 3-layer MLP that transforms the representation of CLIP into 1280 dimensions. We optimize the decoder with the Adam optimizer [Kingma and Ba, 2014] and a learning rate of 1e-6. The training process is less than 6 hours with 1 Tesla A100 GPU. In the inference phase, we use beam search as same as other methods. For the acquiring of video keyframes, we choose the isometric sampling.

Baselines. For image captioning, we include several zeroshot methods as our baselines. First, we compare with a training-free method, called CLIPRe [Su et al., 2022]. Given an image, it retrieves the most related caption from the training corpus based on the image-text similarity as measured by CLIP. Then, we compare two training-free methods with the language model ZeroCap [Tewel et al., 2022b] and SMs [Zeng et al., 2022]. Finally, we compare with text-only methods MAGIC [Su et al., 2022], CLMs [Wang et al., 2022], and current state-of-the-art method CapDec [Nukrai et al., 2022]. For video captioning, we compare it with the state-ofthe-art method EPT [Tewel et al., 2022a]. EPT is a zero-shot video captioning method based on evolving pseudo-tokens. And we also adapt the baselines of image captioning to video captioning by referring to [Tewel et al., 2022a]. These methods use the average features of video keyframes as input.

Evaluation Metrics. Following the common practice in the literature, we perform an evaluation using BLEU-1 (B@1), BLEU-4 (B@4) [Papineni et al., 2002], METEOR (M) [Denkowski and Lavie, 2014], ROUGE-L (R-L) [Lin and Och, 2004], CIDEr [Vedantam et al., 2015], and SPICE [Anderson et al., 2016].

4.2 Performance Comparison

Image Captioning

The results of image captioning are shown in Table 1. We see that Knight achieves the best performance in all 12 evaluation metrics. It is worth noting that compared to the current stateof-the-art method CapDec, Knight achieves significant performance improvement using fewer training parameters. This is because CapDec is based on joint space and alleviates the impact of the modality gap by noise interference. However, it is not reasonable to model the modality gap by random Gaussian noise because the offset direction of the modality gap is directional, while random Gaussian noise is non-directional. Compared with CapDec, Knight utilizes the similarity mapping that is best suited for CLIP. The direction of this mapping is consistent with the offset direction of the modality gap.

To test the generalization ability of Knight, we conduct a cross-domain experiment. Specifically, we apply the text-

†The method is training-free.

only training on the corpus of the source domain (e.g., MSCOCO) to perform inference on the test set of the target domain (e.g., Flickr30k). The results of the cross-domain experiment are shown in Table 2. From the results, we can see that Knight has a significant advantage in generalization ability.

We show some examples generated by Knight compared with CapDec in Figure 4 (a). We see that the caption generated by Knight corresponds well to the content in the image, both in the foreground and in the background. We understand the reason for the high-quality generation of Knight by visualizing the results in Figure 4 (b). We reduce the embedding space representations of the k-nearest-neighbor captions to a 2-dimensional representation by t-SNE (solid-colored dots), while bringing the generated caption back to the embedding space to obtain the corresponding representation (black-circle dots). We can see that the k-nearest-neighbor captions of different images are significantly distinguished on the embedding space, which indicates that Knight has a good discriminatory ability for the different concepts.

Video Captioning

The results of video captioning are shown in Table 3. From the results of the methods adapted from image captioning, we see that these methods do not apply to video captioning. This is because the input to these methods is a representation of only a single image or text. When multiple keyframes from a video are used as input, these methods need to fuse the representations of multiple frames into a single representation. This not only causes the loss of features, but also fails to model the relationship between different keyframes, and thus fails to inference about the behavior of the whole video.

From the results of ETP, we see that although it is a method designed for video captioning, it still does not work well. This is for the same reason that ZeroCap in image captioning fails to generate high-quality captions: the CLIP used for matching computation cannot be aligned with the language model used for a generation without text training. Finally, although there are still gaps in Knight compared to the full-supervised method, Knight achieves the best in all 12 evaluation metrics in the unsupervised method and is far ahead of the other methods.

In the example of Figure 1, although Knight does not use the image of Squidward for training, it successfully maps to Squidward by unsupervised cross-modal mapping of CLIP. In Figure 5, we showmore examples. It is clear from the results that Knight has a strong capacity to recognize the concepts in open world. For example, the game name Minecraft in the first example, math in the second example, and the Middle East in the third example.

4.3 Discusions

A. Are the results of Knight due to the captions in the training corpus that are similar to the inference images or videos?

We compare CLIPRe (Table 1) to analyze the issue. Given an image, CLIPRe retrieves the most related caption from the training corpus based on the image-text similarity as measured by CLIP. From the result, we see that a better result cannot be achieved by only retrieving from the corpus, although the Flickr30K and MS-COCO are large enough. This indicates that our method is not a retrieval method but a generated method.

B. Does irrelevant content in the k-nearest-neighbor captions of an image impact the generation quality?

From the result in Figure 4 (b), we can see that although the generated caption is strongly associated with the k-nearestneighbor captions of the image, there is still a distinction between the two. This indicates that the decoder is not overly dependent on a caption and generates according to its feature, but is a reorganization of the information in the k-nearestneighbor captions. Thus even if there is irrelevant information in one of the k-nearest-neighbor captions, the decoder does not rely on this information completely.

C. What effect does the choice of k have on the performance of Knight?

We set different values of k to explore this issue. The results on image captioning are shown in Figure 6 and the results on video captioning are shown in appendix. It is worth noting that k equals 0 means that the representation of the inference image is used to generate directly. First, the performance at k of 0 is not good. This confirms the existence of the modality gap. Then, when k is 1, i.e., the nearest-neighbor mapping, the performance is still not optimal. Finally, as k increases, the performance of Knight no longer changes significantly after rising to a certain level. Although the value of k increases, there is a limited number of captions associated with inference images in the corpus. An excessively large k maybe introduce the noise.

D. Does Knight show a significant performance decline when the size of the training corpus is reduced?

We set different sizes of the training corpus of Knight to explore this issue. The results of B@4 are shown in Figure 7 and the results of other metrics are shown in the appendix. We observe that the performance of Knight does decline when the size of the training corpus is reduced. However, it is worth noting that Knight does not suffer from a catastrophic performance drop even with just 10% of the training corpus. Compared to other methods, Knight still has a significant improvement.

5 Conclusion

In this paper, we explore how to achieve generation by utilizing the zero-shot capability of CLIP. To alleviate the negative impact of the modality gap on the text-only captioning method, we propose the association-to-generation method, Knight. Unlike previous works, in the inference phase, we represent a given image as k-nearest-neighbor captions by computing the similarity between the inference image or video and the captions in the corpus. This unifies the decoding range in the training and inference phases and makes the generated captions independent of the modality gap. Experimental results show that Knight achieves state-of-the-art performances on both text-only image and video captioning.

Acknowledgments

This work is supported by the Beijing Natural Science Foundation (No.JQ20023), National Natural Science Foundation of China (Grant No. 61832002, 62172094), and CCF-Zhipu AI Large Model Fund.