Prompt Learns Prompt: Exploring Knowledge-Aware Generative Prompt Collaboration For Video Captioning

1 Introduction

In 1959, three computer science pioneers envisioned that AI is to create a computer program that simulated human problem-solving behavior. Humans can process novel tasks effortlessly by using existing knowledge and learning from new information. Artificial systems, however, need a heavily pre-trained base model (e.g., CNNs [He et al., 2016] and Transformers [Dosovitskiy et al., 2021; Wang et al., 2022a]) with extensive fine-tuning on curated data for each problem. This practice is common in deep learning. But adapting these large-scale models to downstream tasks is hard. Full finetuning requires storing and deploying a separate copy of the backbone parameters for every task, which is expensive and

∗Equal contributions.

†Corresponding author.

(2) Context Prompt Recovery: Diffusion

Figure 1: Our method includes two phases: 1) Key Information Learning: model pre-training to extract key information for each video with a knowledge-aware prompt. 2) Context Prompt Recovery: model fine-tuning to recover the pre-trained knowledge-aware prompt for each video and generate the full caption.

impractical, especially for modern Transformer-based architectures such as Swin-L [Liu et al., 2021d] (284M parameters), ViT-Huge [Dosovitskiy et al., 2021] (632M parameters), and iGPT-L [Chen et al., 2020] (1362M parameters). Inspired by human-like learning, a substantial amount of concurrent scientific efforts for intelligent systems have been devoted to the development of a novel training strategy in both natural language processing (NLP) and computer vision (CV) to efficiently transfer knowledge across domains.

This appetite for training has been successfully addressed in natural language processing (NLP) by prompt tuning. The solutions are based on generative language modeling in GPT [Alec and Karthik, 2018] and masked language pretraining in BERT [Devlin et al., 2018]. The idea is to reformulate downstream tasks to look more like those solved during the language model (LM) pre-training with the help of a textual prompt [Liu et al., 2021b]. For example, when recognizing the emotion of a social media post, we may continue with a prompt "I felt so ", and ask the LM to fill the blank with an emotion-bearing word. Creating and experimenting with these prompts takes time and experience, so methods have been proposed to automate the template design process. These methods can be separated into two broad types: a) discrete prompts, which automate search for templates described in a discrete space, usually corresponding to natural language phrases [Zhengbao et al., 2019; Shin et al., 2020], and b) continuous prompts, which are directly described in the embedding space [Li and Liang, 2021; Lester et al., 2021]. These emerging methods enable quick training of generalizable NLP models containing over one hundred billion parameters for novel tasks [Liu et al., 2021c].

Prompt tuning is a generic form of prefix virtual tokens construction that is natural and applicable in computer vision as well [Lester et al., 2021] [Li and Liang, 2021]. Visual Prompt Tuning (VPT) [Jia et al., 2022] introduces only a small amount of trainable parameters into the image feature input space while keeping the model backbone frozen. Several CLIP-based [Radford et al., 2021] methods adopt VPTlike architectures into their image encoders [Uzair Khattak et al., 2022; Huang et al., 2022], achieving impressive performance in terms of efficiency and accuracy. Researchers recently began to investigate how to jointly optimize prompts across vision and language. For example, UPT [Zang et al., 2022] trains a tiny neural network to generate the prompt for CLIP text and visual encoders, both of them are started with a shared initial prompt. MVLPT [Shen et al., 2022] finds that many target tasks can benefit each other from sharing prompt vectors and thus can be simultaneously learned via multitask prompt tuning.

Despite significant interest in this idea following the triumph of VPT, however, prompt tuning methods for crossdomain tasks have been lagging and face limitations.: First, learnable prompts lack explainability and their embeddings are too abstract to provide a human-understandable explanation. Concurrent work has not explored what these prompts actually learn. Second, learnable prompts lack explainability. The concurrent work fails to explore what those learnable prompts learn. The embeddings of these learnable prompts are so abstract that it is difficult to provide a humanunderstandable explanation. Third, vision-language Transformer models require extensive self-attention computation, leading to inefficiencies and lack of knowledge transferability due to the heavy parametric architecture. In light of this view, we ask: how to learn explainable prompts to enable effective learning for across-domain language-vision tasks?

To address these limitations, we present VL-Prompt, a powerful, explainable, and efficient prompt tuning approach. Our contributions are three-fold, as shown in Fig. 1:

• We introduce VL-Prompt, a novel framework for video captioning that splits the task into two parts: a) the Key Information, which is extracted from a pre-training module with flexible textual prompts; b) the Context Prompt, which is fine-tuned with the frozen pre-trained model. The design enables VL-Prompt to handle the translation of a large-scale vision-language model.
• We propose Knowledge-Aware Prompt (KAP) and Vision-Language Prompt (VLP) to investigate the ex-
• plainability of the learned prompts. KAP uses syntactic knowledge to guide sentence generation in pre-training. VLP inserts context prompts between keywords to decode captions in fine-tuning. VLP recovers KAP's information and learns the mutual information in the key information.
• Our method has two main benefits. First, in pre-training, it trains the Transformer model to extract key information with sparse attentions, simplifying the model. Second, in fine-tuning, it can handle more frames on a limited GPU memory, because the main network is frozen and does not need gradient storage.

VL-Prompt is an intuitive yet general video-language framework; it is compatible with different video-language network architectures and tasks. We experimentally show: In §4.2, with efficient video-language Transformer [Tay et al., 2020], VL-Prompt outperforms other Transformer-based counterparts, O2NA [Liu et al., 2021a] and SwinBERT [Lin et al., 2022], ↑ 7.5 ∼ 31.7 in terms of CIDEr and ↑ 5.3 ∼ 8.1 in terms of B@4, on MSVD benchmark. In §4.4, VL-Prompt acquires further improvement by applying our pre-training with KAP on a larger dataset and tuning into a smaller dataset, i.e., for MSVD benchmark, pre-trained on MSR-VTT (↑ 0.6 in B@4, ↑ 0.2 in M), and pre-trained on VATEX (↑ 1.4 in B@4, ↑ 0.4 in M). These results are particularly impressive, considering the number of parameters in the tunable prompt is very small.We hope this work could bring fundamental insights into related fields.

2 RelatedWork

2.1 Video Transformer and Video Captioning

Recently, base on the impressive performance of Vision Transformers (ViT) [Dosovitskiy et al., 2021], TimeSformer [Bertasius et al., 2021] and ViViT [Arnab et al., 2021] are two popular video Transformer method. More recently, after Swin Transformer [Liu et al., 2021d] is introduced as a general-purpose vision backbone for image understanding, Video Swin Transformer [Liu et al., 2022] extends the scope of local attention computation from only the spatial domain to the spatio-temporal domain. Traditional video captioning methods are based on CNN-RNN structure, including S2VT [Venugopalan et al., 2015], PickNet [Chen et al., 2018], OA-BTG [Zhang and Peng, 2019], SAAT [Zheng et al., 2020], ORG-TRL [Zhang et al., 2020], GLR [Yan et al., 2022a], etc. Most recently, SwinBERT [Lin et al., 2022] claims to be the first end-to-end fully Transformerbased model for video captioning.

2.2 Prompt Tuning

Prompt tuning methods can be divided into two categories.

Discrete Prompt Tuning can tune themselves for different tasks by manually constructing different prompts (e.g., CLIP [Radford et al., 2021]) like "A photo of a [object]". This strategy has been widely used in recent researches, for example, ALPRO [Li et al., 2021] and STALE [Nag et al., 2022] proposes a prompt of "A video of [ENTITY]" for video classification and action detection. A good design for prompt

sentence can achieve a better performance. For example, DetPro [Du et al., 2022] judge whether a predicted object detection box is good by two types of prompt: given a ground truth bounding box of an object class, it says "a photo of [CLASS]"; while given a foreground proposal of a partial object, it would instead say "a photo of partial [CLASS]". Other methods such as KDDAug [Chen et al., 2022] design different strategies to generate prompts for different VQA question types (e.g., "[NUMBER] [OBJECT] are there" when asking the number of an object). All of these methods generate fixed prompts with invariant structure.

Continuous Prompt Tuning automates the process by learning soft prompts (e.g., embeddings). For example, CoOp [Zhou et al., 2022b] tries to learn a prompt content optimization for CLIP text encoder on image classification tasks. CoCoOp [Zhou et al., 2022a] optimizes that learnable text prompt contents by the output of the CLIP image encoder. TPT [Shu et al., 2022] learns textual prompts in a zero-shot manner, via different augmented views of a single test image. For video tasks, [Ju et al., 2022] attempts to learn prompt vectors for CLIP text encoder on some simple video understanding tasks including action classification and localisation. Recently, VPT [Jia et al., 2022] introduces only a small amount of trainable visual prompts in the input space while keeping the model backbone frozen. MaPLe [Uzair Khattak et al., 2022] uses VPT to fine-tune the text encoder and image encoder in the CLIP [Radford et al., 2021]) model. VoP [Huang et al., 2022] also inserts prompt embeddings into the CLIP network for text-video retrieval. However, to the best of our knowledge, all these methods have not explored vision-language generative tasks, and none of them have explored what the prompts have learned.

3 VL-Prompt

3.1 Overview

The task of video captioning is to generate a text sequence that summarizes a given video. In this work, we propose a video-language prompt tuning approach for effective caption generation. The overall model architecture of VL-Prompt is shown in Fig. 2, which consists of two key phases, pretraining with Knowledge-Aware Prompt (KAP) and finetuning with Video-Language Prompt (VLP). Intuitively, in the pre-training phase, the video encoder and decoder effectively learns the complex knowledge (the "difficult" part) about the actions and objects from the videos, with the guidance of textual knowledge-aware prompts generated from the annotations. During fine-tuning, the model only needs to tune a few trainable video-language prompts representing the structure of the caption (the "easy" part) while freezing the well-learned encoder and decoder. We adopt an Video Swin Efficient Transformer [Lin et al., 2022] as the video encoder, while the decoder is constructed by a stack of Efficient Transformer Layers [Tay et al., 2020].

3.2 Pre-Training with Knowledge-Aware Prompt

The main responsibilities of the pre-training module is to learn an efficient and effective video encoder for video representation. To this end, we introduce a novel pre-training approach with knowledge-aware prompts.

Textual Knowledge-Aware Prompt. A textual KAP is a discrete prompt which is automatically generated from the annotations of caption. For example, the KAP from original caption of "A teacher is writing a mathematical problem on a whiteboard in a classroom" is "A is writing a mathematical on a in a ". The pipeline consists of the following two steps: 1) Predict Part-Of-Speech (POS) tags on those annotations via POS tagging model (e.g., FLAIR [Akbik et al., 2019]); 2) Select important types of POS tags (e.g., noun or verb) and mask the corresponding words in the original caption; 3) The knowledge-aware prompt X = {x1, x2, ..., xm} (a.k.a., function words) contains remaining words in the caption. The purpose of pre-training is to learn effective video encoder and decoder by recovering the sequence of the marked out keywords Y = {y1, y2, ..., yn} with the guidance of the KAP. Essentially, KAP helps reduce the search space of the target words, which provides additional supervision for video captioning.

Sparse Attention Learning. Dense attention for video features in the Transformer decoder is very computationally intensive. To remove unnecessary attention and refine efficient features in the model, the decoder is supervised by the above keywords to learn the sparse attention patterns [Beltagy et al., 2020; Zaheer et al., 2020; Qin et al., 2023; Wang et al., 2022b], which reduces redundancy among the learned video representation. Following SwinBERT [Lin et al., 2022], the attention mask is defined as a learnable matrix with a size of M ×M, where M is the length of the video embedding from the encoder. Each value Ci,j in this matrix indicates the attention connection between the ith position of the input video embedding and the jth position of the output. This matrix is trained to be more sparse, with the loss designed to reduce the percentage of non-zero elements in the attention connection maps.

Learning Objective. To sum up, the final training objective of the pre-training model is defined as the Cross-Entropy (XE) loss collaborated with the number of non-zero elements in the attention connection maps:

where y and ˆ y denotes the predicted words and the groundtruth respectively, and Y represents the set of notional words (e.g., noun or verb) that removed from the sentence in the KAP. The weight of attention connection count ω will be analyzed in the experiments.

3.3 Fine-Tuning with Video-Language Prompt

With the limited memory of the GPU, it is often impossible to sample a large number of frames from the video when training a large video-language model. To reduce GPU memory usage and increase the model capability for processing more frames, inspired by VPT [Jia et al., 2022], we propose to finetune the pre-trained large-scale model with Video-Language Prompt (VLP), which introduces only a small amount of trainable parameters while keeping the model backbone frozen. We transfer the VPT into a multi-modal prompt tuning framework via designing video-language prompt tokens into the collection of text embeddings and video embeddings. We denote the collection of text embeddings and video embeddings as Etext and Evideo respectively.

Video-Language Prompt. The prompt content is defined as a learnable d-dimensional vector as Tj ∈ Rd for the jth prompt token. Those prompt tokens are fed into a fully connected linear layer f(·) to get video-language prompt embeddings. Following VPT, our tuning process with VLP also has two variants, VLP-Shallow and VLP-Deep.

VLP-Shallow. Prompts are inserted into the first Transformer layer T1 only. Each video-language prompt is a learnable d-dimensional vector. A collection of N prompt tokens is denoted as T(0) = {Tj = (t0, t1, ..., ti, ...., td) | ti ∈ R}Nj=1 , the shallow-prompted decoder with l Transformer layers is defined as:

where E(k) prompt represents the prompt embeddings computed from the kth Transformer layer (1 ≤ k < l), and Logit indicates the Transformer head which convert embeddings into the log-likelihood scores Z for word prediction.

VLP-Deep. Prompts are introduced at every Transformer layer's input space. For kth Transformer Layer Tk, we denote the collection of input learnable prompts as T(k) = {Tj = (t0, t1, ..., ti, ...., td) | pi ∈ R}Nj=1 . As shown in Fig. 2, the deep-prompted decoder with l Transformer layers is formulated as:

Different colors indicate learnable and frozen parameters.

Fine-tuning Objective. During fine-tuning, the sparse attention mask is also frozen. The VLP linear projection and the decoder head are trainable. Following BERT [Devlin et al., 2018], the ground-truth text sentence ˆ Z is randomly masked (e.g., "[MASK] teacher is [MASK] a [MASK] problem on [MASK] whiteboard in [MASK] classroom") and the model prediction Z try to store the masked tokens. Therefore, the fine-tuning objective with VLP becomes:

where z and ˆ z denotes predicted words and the ground-truth respectively. Z denotes all possible words in vocabulary.

3.4 Theoretical Analysis - Prompt Learns Prompt

The core idea of our system is that different words carry different amounts of information. For example, a frequent word (e.g., function words like "on") carries very little information, while a rare word (e.g., notional words like "teacher") is much more informative. In the following analysis, we show that the entropy of the video-language prompt (VLP) and knowledgeaware prompt (KAP) are equivalent, indicating that our model is able to transfer the prompt knowledge from KAP to VLP.

Assuming that the KAP only contains function words with all notional words (nouns and verbs) removed. Mathematically, given an input video V and a knowledge-aware prompt X, our pre-training model M is trained to output a sequence of notional words Y , as shown in Fig. 3. Those function and notional word sequences can be represented by these two random variables. According to the Shannon's theory [Shannon, 1948], the information content or entropy of the pre-trained model can be defined as a conditional entropy:

where X and Y denote the set of function words and notional words respectively. When fine-tuning the model with the video-language prompt network F , which consists of the learnable vectors T and the linear layer f(·), the final sentence output Z can be viewed as a combination of X and Y (i.e., Z is the union distribution of X and Y ). We denote pZ(z) = pX,Y (x, y). Then the joint entropy of the full model is given by:

Then we can measure the expected entropy of the videolanguage prompt F in condition of M as:

where I(M; F) denotes the mutual information between the distribution of the model M and the prompt parameters F. Since M is pre-trained to predict notional words with the guidance of function words and then frozen in the fine-tuning phase, the mutual information I(M; F) is very small. Therefore, we can approximate the expected information of our proposed video-language prompt F to the information of knowledge-aware prompt X, according to the information theory [Thomas and Joy, 2006]:

From the above analysis, we show that the information contents in video-language prompt (VLP) and knowledge-aware prompt (KAP) are equivalent, indicating that VLP learns from KAP, to guide the recovery of the full caption. The full sentence recovery can be viewed as a diffusion process, where function words are diffused among those predicted notional words and then restored.

4 Experiments

4.1 Experimental Setup

Datasets. We conduct experiments on two public video captioning datasets: a) MSR-VTT [Xu et al., 2016] consists of 10K video clips. Each video clip has 20 ground-truth captions. We use the standard split, which has 6.5K training videos, 497 validation videos and 2.9K testing videos. b) MSVD is a collection of 2K video clips downloaded from YouTube. Each video clip has roughly 40 ground-truth captions written by humans. Similar to the prior articles [Chen et al., 2018; Zheng et al., 2020], we use the standard split which contains 1.2K training videos, 100 validation videos, and 670 test videos. We further leverage a larger dataset, VATEX, to study the effect of pre-training. VATEX contains 41.3K videos. Each video clip has 20 ground-truth captions. We use the official training set for training, and evaluate the results using the public test set.

Implementation Details. The number of the Swin Transformer layers in the video encoder is set to 3. The output length of the video feature embedding from the encoder is set to M = 392. The dimension of the video feature embedding from the encoder is 768, while the dimension of the hidden state of the decoder is 512. To ensure the same dimensionality of the video embedding in the encoder and decoder, we transform the video embedding using a linear fully connected network. The number of the transformer layers in the decoder is set to 11. The dimension of the video-language prompt is set to d = 1024. The number of the video-language prompt tokens is set to 100. The max length of the text sequence is set to 50. The max epoch number is set to 10 for both pretraining and fine-tuning. The learning rate is set to 10-4 in pre-training, and 10-5 in fine-tuning.

4.2 Main Results

We compare our VL-Prompt with several state-of-the-art methods on the above commonly used benchmarks. Following previous research [Yan et al., 2022b; Lin et al., 2022], we provide detailed comparisons using a diverse set of performance metrics, including BLEU4 [Papineni et al., 2002], METEOR [Banerjee and Lavie, 2005], ROUGE-L [Lin and Och, 2004] and CIDEr [Vedantam et al., 2015]. Table 1 shows detailed comparisons with eleven LSTM-based and five transformer-based methods for video captioning. In our method, we sample a large number of frames (F = 64) from each video, since our Efficient VL Transformer can avoid overly dense features. It can be seen from the results that: a) Compared to those methods using 2D and 3D CNN encoders, Transformer based method can jointly learn the features of 2D appearance and 3D motion. For example, the popular SAAT [Zheng et al., 2020] uses spatial and temporal feature to represent the static scene and the dynamic motions, but it fails to jointly train the 2D and 3D representation, which can be learned via a visual Transformer. Thus, on MSR-VTT test, our method outperforms SAAT ↑ 4.3 in B@4, ↑ 2.4 in M, ↑ 1.5 in R, and ↑ 4.3 in terms of CIDEr. b) Compared to those methods using Transformer as decoder, prompt tuning methods rely on the well pre-trained large-scale models, which is difficult to be trained. SwinBERT [Lin et al., 2022]

introduces a BERT-based training method with sparse attention, which indeed benefits the pre-training of the large-scale video-language models, but fails to explore the inner relationships over the semantic context. Thus, on MSVD test, our method outperforms SwinBERT ↑ 5.3 in B@4, ↑ 0.3 in M, ↑ 1.4 in R, and ↑ 7.5 in terms of CIDEr. c) VL-Prompt outperforms those prompt tuning baselines, indicating the effectiveness of our prompt design and learning. The learned video-language prompts are more explainable [Wang et al., 2023], which effectively decouple the structured knowledge in the captions and restore them efficiently.

4.3 Ablation Study

We conduct a comprehensive ablation study on MSVD and MSR-VTT benchmarks to investigate the capability of the proposed model.

Impact of KAP and VLP. To analyze the impact of KAP and VLP, we conduct ablation experiments on two variants by removing KAP and VLP from the VL-Prompt respectively. As reported in Table 2, we observe significant performance drop after removing either type of prompts, indicating the importance of both KAP and VLP in the model. This demonstrates that our VLP prompts can capture the internal relations, which denote the semantic correlation between tokens within a sentence, by applying reasoning and linking the input keywords (nouns and verbs) together. Nevertheless, our VL-Prompt with both prompts achieves the best performance with slightly lower training fps.

Prompt Design in KAP. We further conduct experiments of different masking strategies in learning context prompt, including alternative masking, random masking, noun masking and verb masking, to understand the effectiveness of different prompt designs. Table 3 shows the ablation results on MSVD. It can be seen that our original KAP (Noun and Verb masking) outperforms all other strategies with big margins, while only masking nouns or verbs achieves a faster speed. This is consistent with our expectation as these nouns and verbs are the most informative words in the caption, representing the objects and actions in the video.

VLP-Shallow vs. VLP-Deep vs. Full. We compare the results of different VLPs with Linear Probing and Full finetuning in Table 4. It is clear that VLP-Deep outperforms all the other parameter-efficient tuning protocols, including VLP-Shallow, Linear Probing, and even outperforms full fine-tuning. VLP-Shallow achieves the fast speed with a comparable performance over all methods. Similar observations have been found in VPT [Jia et al., 2022].

Prompt Length & Dimension. To investigate the impact of prompt length and dimension in VLP, we conduct ablation study of different combinations of VLP length and dimension. As shown in Table 5, the best choice for prompt tokens is a length of 100 and a dimension of 1024.

Sampled Frame Number. We uniformly sample F = {2, 4, 8, 16, 32, 64} frames from the given video clip to train and test our method on both MSVD and MSR-VTT datasets. As we increase the number of frames, we observe consistent performance improvements in terms of CIDEr (see Table 6). We also find that without VLP fine-tuning, the network is too large for F = {32, 64} which causes an Out-Of-Memory error, as shown in Fig. 4. In contrast, the network works well on that 11GB GPU during fine-tuning with our VLP, demonstrating the efficiency of our VL-Prompt.

4.4 Effect of Pre-training

To further illustrate the effect of our proposed pre-training method, Table 7 shows the performance of VL-Prompt with additional pre-training data. It can be seen that pre-training on a large dataset achieves better performance in all cases, especially when combining with large VATEX dataset in pretraining. For example, VL-Prompt improves the performance by ↑ 1.2 and ↑ 1.7 in terms of B@4 on MSVD through adding MSR-VTT and VATEX to pre-training, respectively. Moreover, we evaluate the efficiency of our approach by comparing it with the baselines. For example, on the MSVD dataset, our model attains an average fps of 7.7, and outperforms SwinBERT and GL-RG in terms of efficiency (which achieve 7.2 and 6.3 fps respectively). This observation validates the effectiveness of pre-training, which is crucial in the later VLP fine-tuning.

4.5 Impact of ω

To understand the impact of the hyper-parameter ω, we evaluate the model performance by varying the hyper-parameters ω from {0, 0.1, 0.3, 0.5, 0.7}. The model performances with different hyper-parameter values are reported in Table 8. It can be seen that ω = 0.5 achieves the best performance, indicating that the model needs to identify a good trade-off between efficiency and effectiveness.

5 Conclusion

In this paper, we propose a novel prompt tuning based video captioning approach, VL-Prompt, by designing two different sets of prompts in pre-training and fine-tuning phases respectively. We first pre-train a video-language model to extract key information with the guidance of knowledge-aware prompts. Then, we design a video-language prompt to transfer the knowledge from the knowledge-aware prompts and fine-tune the model to generate full captions. Experimental results show the superior performance of our approach over several state-of-the-art baselines. Theoretical analysis on how the information from knowledge-aware prompts is transferred to the video-language prompts is also conducted. A potential drawback of our method is the low generalization ability that may result from the limited number of parameters and data samples. In future, we plan to investigate VL-Prompt in zero-shot settings. We also plan to apply VL-Prompt in other downstream tasks such as VQA. We hope this work can inspire future studies in video-language prompt tuning.