Differentiable Prompt Learning for Vision Language Models

1 Introduction

Parameter-Efficient fine-tuning (PEFT) offers a paradigm to adapt pretrained model to downstream tasks using a small set of trainable parameters. Among the PEFT methods, the prompt learning can perform few-shot learning or even zeroshot learning to adapt the pre-trained models to new scenarios. Using appropriate prompts, both text prompts and visual prompts, to query foundational models have shown great potential. At the same time, the selection of context tokens in prompts has a pronounced effect on the performance of the prompt learning methods [Zhou et al., 2022b; Schick and Sch¨utze, 2021]. Continuous prompts use differentiable vectors that are in the embedding space of the foundation model. Continuous prompts use trainable parameters that are learned during the fine-tuning process. It avoids the time-consuming trial-and-error process using discrete context tokens. Besides, the continuous prompts are not constrained to be the embeddings of word tokens in the vocabulary. The selection for the embeddings of context tokens becomes continuous.

Deep continuous prompts add prompts to multiple layers. This method has shown superior performance compared to only adding continuous prompts to the input in both vision and language fields. There are two hyperparameters: the context length of continuous prompts cp and prompt depth ℓp. Let the hidden dimension be d, continuous prompts E ∈ Rcp×d are inserted to the input to the neural network layers up to ℓp layers. Since E are automatically determined during the fine-tuning process, a natural question is: can we automatically determine cp and ℓp? Recently, it has been found that training merely part of deep neural network layers can achieve a performance comparable to or even better than training all layers. This line of work indicates that there is a subset of layers in the pre-trained model, depending on the distribution shift between the pre-training dataset and fine-tuning dataset, whose parameters might be close to a minima for downstream tasks [Lee et al., 2022; Lodha et al., 2023].

In the prompt learning, we postulate that there might be some layers that do not need deep continuous prompts or only need continuous prompts with shorter context length compared to rest of layers. In the existing prompt learning works, however, a fixed number cp is used for each layer. To examine our postulation, we design our method to insert continuous prompts with different context lengths. The context lengths of continuous prompts are determined using a differentiable formulation. If a layer has the best context length found to be 0, we do not add continuous prompts to this layer. We name the proposed method differentiable prompt learning (DPL). The few-shot learning experiments show that the DPL method can find continuous prompt configurations, i.e., the context length and depth of continuous prompts inserted to the input of each layer. The performance of downstream fine-tuning over 11 datasets shows the superiority of the proposed method.

In summary, our DPL method has the following contributions:

• The DPL method automates the continuous prompt learning process. It automatically determines the continuous prompts and associated hyperparameters including the context length of the continuous prompt inserted for each neural network layer and prompt depth.
• The DPL method removes the constraint in the manually designed continuous prompt methods that the context length for each layer is fixed. The heterogeneous design of inserting continuous prompts enables a more flexible design for the prompt learning. The method is simple and focuses only on continuous prompts, which means it can be combined with sophisticated designs in existing prompt learning methods.
• We find the optimal deep continuous prompt configuration is dataset-dependent. By tailoring deep continuous prompts for each dataset, the DPL method can achieve the performance better than manually designed prompt learning methods.

2 Background and RelatedWork

Prompt Learning. Prompt learning offers an efficient way to adapt pre-trained foundational models to downstream tasks. This technique is of great interest in both vision and natural language. CoOp [Zhou et al., 2022b] inserts continuous prompts to the input of the vision-language model. The visual prompt tuning [Jia et al., 2022] proposes the deep continuous prompts to vision transformer [Dosovitskiy et al., 2020]. MaPLe [Khattak et al., 2023] combines the ideas from these two works by using deep continuous prompts in both the text branch and the image branch. Similar to CoOp, CoCoOp, PLOT and ProGrad insert continuous prompts only in the input. CoCoOp [Zhou et al., 2022a] inserts the continuous prompts conditioning on the input images. PLOT [Chen et al., 2022] uses the computationally costly iterative algorithm to compute the transport plan for aligning word tokens and image patches. ProGrad [Zhu et al., 2023] uses the gradientaligned knowledge distillation to avoid the overfitting problem. All of the above-mentioned methods use fixed context length as opposed to our proposed DPL method which adapts the context length for each layer dynamically based on the dataset.

Neural Architecture Search. Neural architecture search (NAS) automatically determines the architecture by minimizing the objective function. One-shot NAS is an efficient way to search neural architectures. In the one-shot NAS, a supernet [Saxena and Verbeek, 2016; Pham et al., 2018; Liu et al., 2018] is introduced comprising all possible architecture in the search space. After training the supernet, a subnet is uniquely determined as the neural architecture determined by NAS.

2.1 Revisiting Differentiable NAS

Differentiable NAS [Liu et al., 2018; Yan et al., 2021] uses a cell-based search space. A cell is represented by a directed acyclic graph (DAG) G(V, E). Each node is a hidden representation and each directed edge is an operation transforming the hidden representation. The supernet incorporates all the operations. Let O be a set of candidate operations. The categorical choice of a particular operation is relaxed by the softmax over all possible operations:

where α(i,j) are trainable parameters that determine the searched operation between the latent representation i and j. The goal of the search process is to replace ¯ o(i,j) with the most likely operation. The optimal neural architecture is obtained after the search process.

Inspired by differentiable NAS methods, we relax the categorical selection on the context lengths of continuous prompts to make the search space continuous. We use the search space to determine the optimal prompt configuration. The continuous prompts are trained from scratch. Existing vision-language prompt learning methods use the deep prompt method [Jia et al., 2022] where continuous prompts are added to transformer blocks and removed after the selfattention [Vaswani et al., 2017]:

Here we denote l-th transformer block as f (l) . Different options for adding continuous prompts cannot be mixed as the dimension of the input [x(l-1) ,E(l-1) ] is different due to different context lengths. We solve this problem by using crossattention. Details are reported in the following section.

3 Differentiable Prompt Learning

We use the pre-trained CLIP model [Radford et al., 2021] in our experiments. The CLIP model is pre-trained on over 400 million image-text pairs. The DPL method has two stages: the searching stage and the training stage. The goal of the searching stage is to determine the context length of the continuous prompt to be added to each transformer block of the CLIP model. The best prompt configuration is used in the training stage. Figure 1 shows the proposed method.

3.1 Searching Stage

We prepend continuous prompts {E(l) } to inputs to transformer blocks {x(l) } in the text branch and image branch, where 1 ≤ l ≤ ℓ, l ∈ N+. The concatenated inputs are fed into transformer blocks f(·). w.l.o.g, for the l-th layer, there are t options for adding continuous prompts of different context length E(l) i ∈ Rci×d , where 1 ≤ i ≤ t, i ∈ N+, to the input x(l) ∈ Rcl×d . ci is the context length for i-th option, d is the hidden dimension. For the text branch, the hidden dimension is d = dtxt. For the image branch, the hidden dimension is d = dimg. Continuous prompts in the text branch are independent of those in the image branch. The output h(l) of the transformer block is:

i = 1 accounts for the case where no continuous prompt is added. Different from the traditional transformer [Vaswani et al., 2017] where self-attention mechanism is applied, we use the cross-attention mechanism in transformer blocks:

We use parameters {β(l) i }ti=1 to control the contribution of different options to the output. In other words, the mixing weight for the hidden representation h(l) is parameterized by β(l) i :

The Equation 6 indicates that the output of each transformer block is a combination of all possible context lengths, i.e. [0, t]. {β(l) i }ti=1 is computed by differentiable parameters {α(l) i }ti=1 :

Figure 1 (a) shows the illustration of the searching stage. Same as conventional prompt learning methods, the parameters of the transformer blocks of the pre-trained model are frozen. During the searching process, updated parameters are continuous prompts and α parameters. The goal of updating continuous prompts is to minimize the training loss Ltrain while that of updating α parameters is to minimize the validation loss Lval. This implies that the searching stage is a bilevel optimization problem [Anandalingam and Friesz, 1992; Colson et al., 2007] with the upper level parameter α and the lower level parameter E:

Algorithm 1 shows the searching stage. The input is the pre-trained vision-language model and two α matrices (we call them search spaces). Two α matrices are randomly initialized before training. After convergence, the α matrix Aα ∈ Rℓ×t is obtained as shown in Figure 1 (b). The row dimension is associated with the depth and the column dimension is related to t options. The column index of the highest Aαim (1 ≤ m ≤ t, m ∈ N+) for the i-th transformer block determines the context length of the continuous prompts added to that block.

Algorithm 1 Searching stage for vision-language models

• 1: Input: A pre-trained model and two α matrices Aα ∈ Rℓ×t with randomly initialized weights.
• 2: while not converged do
• 3: Update Aα by descending ∇AαLval(E,Aα).
• 4: Update continuous prompts in both text branch and image branch by descending ∇ELtrain(E, Aα).
• 5: end while
• 6: for i = 1 to ℓ do
• 7: Aαik = maxmAαim, k determines the context length of continuous prompts for the i-th block in the best prompt configuration.
• 8: end for
• 9: Output: Prompt configuration for the image branch and the text branch.

We use the term supprompt to represent the combination of the continuous prompts added to the pre-trained model and an α matrix in the searching stage. For vision-language models, there are two sets of supprompts for the image and text branches. After the search, the best context length is identified for each layer and the resulting model is then trained from scratch. We use the term subprompt to denote the final best context length for each layer that is used in the final model. Terminologies are summarized in Appendix A.2.

Similar to the differentiable NAS where the supernet incorporates all candidate operations, our method creates a supprompt that contains all prompt configurations in the search space. Incorporation of all possibilities inevitably leads to a relatively large supprompt compared to the subprompt, which can increase the computational cost in the searching stage. After the optimal prompt configuration is determined, the computational complexity is small compared to the deep continuous prompt method such as MaPLe. The complete comparison is reported in the Appendix A.3.

Figure 2 shows the comparison between our method and prevalent prompting methods for vision-language models. After transformer blocks, the traditional methods have varying context lengths that depend on the number of inserted context tokens. Our method results in a fixed context length.

3.2 Training Stage

We denote Aαtxt for the α matrix of the text branch and Aαimg for that of the image branch. The column index of the maximum value in each row of α matrix determines the context length of the inserted continuous prompts, i.e. Aαik = maxmAαim. After the context length of continuous prompts is determined, we fine-tune the model on various downstream datasets. Figure 1 (c) shows the training stage. The finetuning process is the same as the conventional prompt learning method: model parameters are frozen and only continuous prompts are differentiable.

In the image branch, the input x ∈ RC×H×W is patchified and projected to produce patch token embeddings [Dosovitskiy et al., 2020]. A learnable classification token embedding ucls ∈ R1×dtxt is added to the patch token embedding ˜ximg = [ucls, u1, . . . , unimg ]. In the text branch, a text template such as a photo of a [class] is tokenized and embedded to generate text token embeddings that are formulated as ˜xtxt = [vSOS, v1, . . . , vntxt , vk, vEOS], where vSOS is the start token embedding, vEOS is the end token embedding, and vk is the embedding corresponding for the class name. ˜xtxt and ˜ximg are combined with continuous prompts and fed to transformer blocks in the text branch and image branch for obtaining text embedding ˜g and image embedding ˜f , respective. Within each transformer block, we use the cross attention mechanism as shown in Equation 4 and 5. The output logits are based on the similarity between ˜g and ˜f :

where τ is the temperature parameter. The loss Lde computes the deviation of the prediction from the ground truth labels:

where F(·) is the loss function. We use the cross entropy loss for calculating Lde. Continuous prompts are updated by descending the training loss ∇ELtrain(E).

In addition to the vanilla DPL, we add knowledge distillation in the training stage. Specifically, we use KullbackLeibler (KL) divergence [Hinton et al., 2015] between the model prediction ˆy = p(x) and the zero-shot pre-trained model prediction pzs(x):

The total loss Ltotal is computed by:

where λ is a hypereparameter. The gradient descent of continuous prompts uses Ltotal to update parameters.

4 Experiments

4.1 Datasets and Experiment Setup

Datasets. We evaluate the DPL method on 11 datasets: Caltech101 [Fei-Fei et al., 2004] and ImageNet [Deng et al., 2009] for the generic object classification, DescribableTectures [Cimpoi et al., 2014] for the texture classification, EuroSAT [Helber et al., 2019] for the satellite image classification, FGVCAircraft [Maji et al., 2013], Food101 [Bossard et al., 2014], OxfordFlowers [Nilsback and Zisserman, 2008], OxfordPets [Parkhi et al., 2012], and StanfordCars [Krause et al., 2013] for the fine-grained image recognition, UCF101 [Soomro et al., 2012] for the action classification, and SUN397 [Xiao et al., 2010] for the scene recognition.

Baselines. We compare the proposedmethodwithCoCoOp [Zhou et al., 2022a], PLOT [Chen et al., 2022], MaPLe [Khattak et al., 2023] and ProGrad [Zhu et al., 2023]. The original implementation of PLOT uses ResNet [He et al., 2016] as the backbone model. For a fair comparison, we replace the backbone model of ResNet with the transformer model. In addition to the prompt learning methods, we examine the performance of the zero-shot CLIP (ZS CLIP) and that of training a linear classifier given image and text representations by CLIP (Linear probe) [Radford et al., 2021].

Experiment Details. We use the pretrained ViT-B/16 CLIP model [Radford et al., 2021] in the few-shot learning. In both training stage and searching stage, we use the same hyperparameters except for the number of epochs. The number of epochs in the searching stage is 60 while that for the training stage is 40. The batch size is 4 and we use stochastic gradient descent (SGD) to optimize continuous prompts. In the searching stage, two α matrices are optimized using SGD strategy. Learning rate is 3.5 × 10-3 . Experiments are conducted using a single NVIDIA A40 GPU.

4.2 Determining Context Lengths of Continuous Prompts

We use the few-shot learning setting in the searching stage. The number of shots is the same for the searching and training stages. The number of shots is 16. The results of using 8/4/2/1 shots are shown in Appendix A.5. The evolution of α matrices is visualized to examine the convergence process. The evolution of α matrices using 11 different datasets is reported in the Appendix A.1. We define special α matrices formally:

Definition 4.1 (single-dominant). An α matrix Aα ∈ Rℓ×t is single-dominant if ∀i ∈ N+, 1 ≤ i ≤ ℓ, ∃j ∈ N+, 1 ≤ j ≤ t s.t. Aαij ≫ Aαik , where k ∈ N+, 1 ≤ k ≤ t, k ̸= j.

The single-dominant α matrix means that the searching algorithm has a high confidence in the searched subprompts. This is beneficial to the training stage as the determined α matrix is robust against the fluctuation of the matrix in the updating process. We use the alpha value at each row with the largest value (i.e. argmaxj αij) to determine the context length of continuous prompts added in the training stage. Hence, the training stage does not require the α matrix to be single-dominant. When α matrix is not single-dominant, the decision on adding continuous prompts is not robust. For example, in the case where two α values are close:

the fluctuation can change the order in the comparison of α values. At the epoch of 60, if we use a threshold value to filter out small α values, α matrices become essentially sparse matrices.

Figure 3 (a) and (b) shows α matrices at the epoch of 60 for all 11 datasets. Compared to the image branch, there are considerably more single-dominant α matrices in the text branch. Although we use 16-shots learning, the searching results indicate that the searching algorithm has pretty high confidence for the searched prompt configuration. α matrices at the last epoch are used in the training stage for the prompt learning. To quantify the evolution process of α matrices, we firstly define α difference:

Apparently, single-dominant α matrix has a high α difference value. Figure 3 (c) shows the variation of the α difference. At the early training stage, the α difference increases drastically. there is a fluctuation when the α difference is close to the converged value. The fluctuation comes from two parts: the first part is related to the variation of the largest α value in each row. Even though the α matrix is single dominant, the largest α value is subjected to the variation in the gradient descent. Owing to the softmax operation, the contribution of the largest α value and that of the rest α values in a row is not independent, hence we call this intra factor. The second part is the fluctuation of α values except the largest ones. We call it inter factor.

The number of dominants describes the number of pairs between the largest α values and the rest of α values in the same row:

The ideal converged α matrix has the number of dominants # Dominants = ℓ(t - 1).

4.3 Prompt Learning Based on AlphaMatrix

After determining the context length of added continuous prompts, we add them to the pre-trained CLIP model and conduct the prompt learning in a supervised fashion. We use the few-shot learning and the number of shots is 16. The α matrix in the text branch might be different from that in the image branch as shown in Appendix Figure 5, and there is no coupling function [Khattak et al., 2023; Zang et al., 2022] between added continuous prompts in the two branches. Table 1 shows the performance comparison on downstream datasets. Note that the direct comparison of zero-sho CLIP with other methods is not fair as it does not require any training.

Our proposed method shows a pronounced advantage compared to baseline methods. It boosts the average accuracy by 2.60% test accuracy. The proposed method shows unfavorable performance on the Food101 dataset. When comparing the zero-shot CLIP method with the linear probe method which adds a linear classifier on top of pre-trained CLIP model, there is a remarkable performance drop (77.30% → 70.13%). We postulate that the reason might be related to the forgetting issue [Chen et al., 2019; Boschini et al., 2022] in the fine-tuning process. ProGrad is designed to avoid the forgetting issue by aligning the gradient descent direction of matching predictions and ground-truth labels to that of knowledge distillation when there is a conflict. ProGrad exhibits a good performance on that dataset.

We find that the performance difference between the DPL method and baseline methods is largest on the EuroSAT and Aircraft datasets. There is a large distribution shift on EuroSAT and Aircraft datasets compared to the pre-train datasets of the CLIP model. On the generic dataset such as Caltech101, the distribution shift is small. Hence, the difference between the DPL method and baseline methods is minimal.

We visualize the attention map on the downstream datasets using different prompt learning methods as shown in Figure 4. The Grad-CAM visualization [Selvaraju et al., 2017] indicates that our method is beneficial to focus on key elements in the foreground. In the image of amphibious aircraft, the attention of the model using the DPL method is localized on the float and the window. The model's attention focuses on the vehicle's logo in the car image. In the action classification, the model focuses on riders and horses. Those elements are important to differentiate the target class from the candidate classes. The PLOT method [Chen et al., 2022] is designed to align the prompts with image features using the optimal transport theory [Peyr´e et al., 2019]. It also exhibits a good alignment. The DPL method only relies on the prompt configuration, which means it is compatible with using the optimal transport theory.

4.4 Ablation Studies

We examine the performance of the shallow prompt learning method: continuous prompts with a context length of 16 are added to the inputs of the text branch and the image branch respectively. The equivalent prompt depth is 1. The way of adding context prompts is illustrated in Equation 4 and Equation 5. The result is shown in the Table 2. Compared to the DPL method, the shallow prompt learning method has a remarkable performance drop. There is the largest performance degradation in the EuroSAT and Aircraft datasets. These two datasets also exhibit the advantage of the DPL method over baseline methods shown in the Table 1.

5 Discussion

By using bi-level optimization, the DPL method is able to automatically determine the context length of the continuous prompt for each layer. It seems to indicate that the optimal prompt configuration might be dataset-dependent. The deep neural networks are found to be brittle to even small distribution shits between the pre-training and fine-tuning datasets [Recht et al., 2019; Hendrycks and Dietterich, 2019]. A dataset-dependent training scheme provides the flexibility to the level of distribution shift. Our results show that the fewshot learning with DPL can boost the downstream accuracy.

Existing deep prompt learning works hardly consider a granular level of adding continuous prompts: each layer has the same context length and a hyperarameter of prompt depth is empirically determined. A prompt depth ℓp smaller than the model depth ℓ indicates that the last few layers of the pretrained model might be close to a minima for the downstream dataset. In the transfer learning, not all layer parameters are responsible for the distribution shift. There are numerous researches on training different layers differently to mitigate the forgetting issue [Niu et al., 2022; Toneva et al., 2018] caused by the distribution shift: fine-tuning on the target domain using gradual unfreezing [Howard and Ruder, 2018], using different learning rate for different layers [Ro and Choi, 2021; Shen et al., 2021], and training part of layers [Lee et al., 2022; Vettoruzzo et al., 2024].

If some layer parameters are close to optimal in downstream datasets, we believe there is no need to add continua photo of a Horse Race ous prompts to those layers. The deep prompt methods using ℓp < ℓ avoid adding prompts to layers that are close to the optimal. If we refine this strategy, a natural question is do all ℓ layers require the same context length? Different layers might have a different level of deviation from the minima. The extreme case is that a layer is very close to the minima and no continuous prompts are needed, i.e. the context length is 0 in this layer. For layers with different levels of deviation, we might need to use different context lengths. This is consistent with the result of using the DPL method. Overall, the result reveals two things: (1) the optimal context length depends on the distribution shift. (2) different layers of the pre-trained model might require different context lengths.

Figure 4: Original image and Grad-CAM visualization [Selvaraju et al., 2017] for various methods on FGVCAircraft, StanfordCars and UCF101 datasets. Our method helps the pre-trained model focus on key elements in the foreground and avoid distraction from the background. The text template a photo of a [class] is used in the Grad-CAM calculation.

6 Limitation

The searching stage of the DPL method, similar to NAS, has higher computational costs than conventional soft prompting methods due to the introduction of the supprompt. The computational cost is determined by the size of the search space. At each depth, DPL searches context lengths of {0, 2, 4, 6}. The size of the search space is 2.81 × 1015 . Decreasing the size of the search space can reduce computational costs. Using differentiable approach greatly accelerates the searching process compared to exhaustive search, i.e. ta × |A| ≫ Tdpl, where ta is the training time for prompting methods and Tdpl is the total running time (training state + searching stage) for the DPL method. The training stage of the DPL method is lightweight as it relies completely on prompt configuration without advanced technologies such as coupling functions. Details regarding to the computational costs are discussed in Appendix A.3.

7 Conclusion

In this work, we automate the prompt learning design by relaxing the categorical selection of context lengths to obtain a continuous search space. Using limited data, our method is able to find a prompt learning setting that is better than the existing manually designed prompt learning methods. The method is simple yet effective: it only focuses on determine context lengths of continuous prompts. We empirically find that our searching algorithm has a higher confidence in the text branch compared to that in the image branch, and that the prompt configuration shows a data-dependent behavior. The data-dependent behavior and the asymmetric prompt insertion in two branches demonstrate the strength of the automatic prompt learning design. In summary, our work proposes a new paradigm of adding continuous prompts that removes the restriction of the manually designed context length fixed prompt learning method.

Acknowledgements

This work was supported by the USA National Science Foundation under grant #2047488, and IBM-RPI Future of Computing Research Collaboration.