On Conditional and Compositional Language Model Differentiable Prompting

1 Introduction

Humans have a remarkable ability to solve cognitive tasks by following instructions and using prior relevant experiences. Can a machine do the same? Recent research has shown that textual task instructions [Efrat and Levy, 2020] appended to a Pretrained Language Model's (PLM) input can yield successful results in several NLP tasks such as classification [Schick and Sch¨utze, 2021], image captioning [Tsimpoukelli et al., 2021] and question-answering [Mishra et al., 2021] without fine-tuning the full model. However, PLMs do not always accurately grasp the meaning of textual prompts [Mishra et al., 2021] and often display high sensitivity to the word formulation [Jiang et al., 2020] of instructions. Further, unless the model exceeds billions of parameters, full-model finetuning still typically outperforms human-engineered prompts for frozen PLMs [Lester et al., 2021]. To mitigate such issues, differentiable and continuous prompting techniques [Li and Liang, 2021] have emerged as a viable alternative (illustrated in Figure 1a). However, current techniques optimize prompts solely based on the task learning objective without leveraging information embedded in textual prompts.

∗Work completed during an internship at Amazon Research.

Figure 1: PROPS is a differentiable and conditional prompt generator that outputs a sequence of vectors that is prepended to PLM hidden states at positions ∈ Pidx as seen in (a).

We argue that simple task instructions (e.g., "Translate English to French"), along with other already available or easily extractable metadata, such as the text category of an input sample (e.g., "sports"), can be valuable conditioning context for a model to generate a continuous prompt. The prompt would be specific to the particular task and input, and would elicit task- and input-specific output from a frozen PLM. We propose a model that can, given a PLM, be trained to transform task- and input-specific text into conditional continuous prompt embeddings. The advantage of using text conditioning as input to a continuous prompt generator is that our model can, once trained, repurpose previously unseen instructions from previously trained tasks, without having to retrain a PLM. By sharing aspects in the task instructions that were observed during training of related tasks, the continuous prompts of new tasks are more suitable for transfer learning and few-shot learning. With our method, we seek to combine the advantages of simple task descriptions, which provide useful context, and learned continuous prompts, which work well even with smaller PLMs. In addition, we propose to enhance our model's compositional ability for few-shot and transfer learning by using a modular network structure, based on our own neural formulation of Production Systems (ProdSys, [Newell, 1972]).

ProdSys has several desirable properties. First, the ability to reason in a propositional format (if - then) allows the disentanglement and modularization of knowledge about a task or input sample. Second, using the fact that propositions are discrete and independent from each other, the system can combine productions and compose acquired knowledge. We hypothesize that such properties are desirable to improve generalization, transfer of learning, and reusability of prior task descriptions when generating prompts. ProdSys models symbolic knowledge in the form of "IF" (condition or situation or task) - "THEN" (action or instructions or response) rules, called productions, and a set of mechanisms for matching and applying such rules. Generally, ProdSys contains four basic components [Nilsson, 2010]: 1 a long-termmemory, a knowledge base that contains general information independent of the current situation/task and which persists over time and that may be used as a condition; 2 a working memory that contains temporary information about the current situation or task and that also operates as a sparse rule selector; 3 a rule memory that contains a collection of productions; 4 an inference engine that maps a condition Cj to an instruction Ii

in the form Cj Ri --→ Ij , for selected productions Ri.

We draw a parallel between the process of generating continuous prompts (differentiable instructions for the PLM) from textual conditioning (conditions) and rule productions. Specifically, as shown in Fig. 2: a frozen PLM can be used as a persistent knowledge base [Petroni et al., 2019] or 1 a long-term memory 1 ; a Condition Encoder that maps textual conditioning sequences to a fixed-sized vector to create the Rule-Condition matrix works similarly to 2 a working memory by encoding and sparsely matching conditions and rules; the selected differentiable rules, the summation operation and transition function allows the model to map a condition to differentiable instructions and work as 3 a rule memory and 4 an inference engine; finally, instructions 5 are, in our case, differentiable prompts that we prepend to a frozen PLM's hidden states. Such a system, which we call Prompt Production System (PROPS), shares the four main building blocks of ProdSys, which allows it to conditionally generate prompts while compartmentalizing neural propositional knowledge via its condition-rule selection mechanism. As we will discover, such mechanism equips PROPS with an ability to reuse and re-purpose knowledge from examples of the same tasks or instructions of intermediate and jointly trained tasks that allow the model to better control text generation. This construction allows PROPS to surpass all other prompt generator baselines, and often fully fine-tuned models, on summarization, translation, and semantic parsing benchmarks.

1 The neural long-term memory is used in two ways: the conditions can be hidden states from the previous PLM layer and the gradient updates are dependent of the PLM pretrained weights.

Our contributions are the following:

• We provide strong Conditional Prompt Generator baselines (Section 3.1) on multiple tasks and datasets, showing the advantage of conditioning continuous prompts.
• We propose a new architecture, called PROPS, a prompt generator based on ProdSys (Section 3.2).
• We demonstrate PROPS 's ability to compose knowledge in the four settings described above in both the full-data regime and low-resource settings.
• To better understand PROPS, we provide theoretical and empirical analysis in Appendix D.

We evaluate our methods in four situations as illustrated in Fig. 3 to check if the following can be composed: (1) input text from training set input text segments in Section 4.2, (2) tasks when knowledge is transferred between all tasks during training in Section 4.3, (3) metadata about the input data in Section 4.4, (4) all of the above combined in Section 4.5.

2 RelatedWorks

Differentiable Prompting has seen increased interest lately due to the method's ease of use and compelling performance. Unlike prior work, PROPS is a conditional prompt generator. There are two differentiable prompting methods that are close to PROPS. The first is PTR [Han et al., 2021] that breaks an input into a series of predicates of first order logic. However, unlike PROPS, sub-prompts are manually designed and PTR is mainly useful for classification tasks. Concurrently to our work, PGT [Li et al., 2022] also attempts to transfer prompts learned frommultiple representative support text generation tasks (e.g. paraphrasing, question answering). Instead of using an attention mechanism to select task-specific prior prompts, PROPS compartmentalizes and reconstructs prior prompt knowledge using differentiable rules, allowing module sharing across tasks. Using fewer support tasks, PROPS outperforms PGT on XSum on the fully-supervised and few-shot learning settings.

Modular Networks are a type of neural architecture that splits computation into specialized and independent neural sub-systems (i.e., modules). Modular Networks borrow from conditional computation methods, where modules are chosen dynamically given a certain criterion. Such module specialization has been shown empirically to improve generalization to unseen inputs [Kirsch et al., 2018] or tasks [Goyal et al., 2021]. In a multitask learning setting, several recent approaches have leveraged modularity to decompose knowledge [Ostapenko and Charlin, 2021] such that it could be reused for a new task. Similar to Ponti et al., our method also assumes that tasks can be decomposed into a set of modules that are shareable and reusable across tasks. PROPS differs in three ways: (1) we use both tasks and inputs to condition computation; (2) sub-module selection is informed by external information (text conditioning); (3) we use differentiable prompts on a PLM instead of adapters.

3 Methodology

We now present the two components on which our method is based. We will first provide an overview of Conditional Prompt Generators (CPG), linking conditional continuous prompts and adapters. Then, we discuss PROPS, our novel construction of CPGs using Neural Production Systems.

3.1 Conditional Prompt Generators (CPG)

Prompt Generators such as Prefix-Tuning [Li and Liang, 2021], Prompt-Tuning [Lester et al., 2021] and P-Tuning [Liu et al., 2021] are all Unconditional Prompt Generation (UPG) methods used to adapt PLMs. Generally, prompt vectors Pk,Pv ∈ RTP×d , of length TP and dimension d, are concatenated to the key K and value V vectors to modulate the frozen PLM attention output of each head H. We get for a query Qt at step t:

where σ is the softmax function and α = α(Qt,K, Pk) is a learnable gating function:

Conditional Prompt Generation, as in PROPS, is achieved by plugging condition-dependent prompt vectors [Pk, Pv] = [Pk(ct),Pv(ct)] = P(ct) in Equation 1, where Pk,Pv is created by splitting the last dimension of P(ct) ∈ RTP×2d in two and ct is the word embedding at the tth condition token.

Conditions C ∈ C represent posterior knowledge about a task or dataset as well as instructions. The Conditions or conditioning text are inputs to the CPGs. We provide examples in Table 1 following the different evaluation settings depicted in Fig 3. In certain cases, it can be the input data itself as in the compositional generalization task (e.g.: input "walk right twice and walk left" has the output "turn right walk turn right walk turn left walk"). The conditioning text is easily extractable. For news summarization, we extracted the news outlet and the article type. For the topic-focused news summarization, additional inputs such as the topic is provided with the dataset. Before going through a word embedding layer, each condition is a textual description or an instruction about an example, a type of an example, a task or a dataset.

Instructions were written by the authors. See Appendix A for more details.

3.2 Prompt Production Systems (PROPS)

PROPS is inspired by Neural Production Systems (NPS, Goyal and Bengio), the deep learning version of ProdSys. NPS is an end-to-end model that constructs object- or condition-centric representations on which independent learnable production rules are applied. PROPS also has independent learnable production rules but there are three main differences with NPS. First, rules are applied to sub-sequences of variable lengths instead of a single vector. Second, PROPS is tied to a PLM through gradient updates and the appendage of prompt vectors to the input hidden states whereas NPS was not built to interact with a PLM. Third, our rule selection mechanism has a different design that allows rule overlap amongst conditions of a single instance. Unlike NPS that uses MLPs, our rules are attention heads [Vaswani et al., 2017]. PROPS consists of N separately differentiable rules, {R1,R2, ..,RN} that are each represented byRi = ( ⃗Ri, ˜Hi), where ⃗Ri is a rule embedding vector and ˜Hi is an attention head. As summarized in Algorithm 1, each condition sequence SC = 〈ct|t ∈ {1, . . . , TC}〉C, where TC is the sequence length of condition C ∈ C and C is the set of conditions, is first encoded by a Condition Encoder f(·). We use a light-weight attentive max pooling layer [Wu et al., 2020] to encode condition sequences into a fix sized representation. Each condition vector ⃗C is then concatenated together to create a Condition Embedding matrix [E] ∈ R|C|×d , where d is the embedding dimension. Similarly, the learnable Rule Embedding matrix [R] ∈ Rn×d , for n rules (i.e., attention heads), is created by concatenating each rule representation ⃗Ri. The rule-condition matrix is formed from the dot product operation [E] · [R]⊤ ∈ R|C|×n and provides a template that maps a condition to a set of rules. We use a Gumbel Top-k, our extension 2 of the Gumbel Softmax , to choose k rules out of n for any of the conditions in C. As discussed in Section 4.5, the ability to choose rules allows PROPS to compose modules. Typically k = n2 > 1 is the best choice of chosen rules as it allows a maximum combination of sub-modules (see Appendix D.1 and D.2 for in-depth theoretical and empirical analysis). The rest of the steps are similar to a Transformer layer's. However, instead of using all attention heads ˜H1...n to transform a sequence, condition sub-sequences use a varying subset of k heads. At step t, our prompt vector Pt is generated in the following way:

where LN is layer normalization , ct is the condition word embedding at step t, g(·) is the transition function (i.e., intermediate Transformer layer), W o is a learnable weight matrix, ‖ is the concatenation operator, {r} is the set of indices identifying selected rules Pt can be applied L times for an L layer PROPS. Note that each PROPS layer shares weights to allow greater parameter efficiency and knowledge reusability.

2 The argmax of a Gumbel softmax would only point to one rule. To select multiple rules we apply a differentiable top-k operator.

4 Main Results

In the next sections, we discuss our experimental setup and evaluation results across the four settings seen in Fig. 3 for both full data and low resource settings. Raffel et al. observe significant gaps between full-model fine-tuning (FFT) and adapters [Houlsby et al., 2019; Bapna and Firat, 2019] on both summarization and translation tasks. The gap increases with increasing training data size. Recently, several methods claimed results on-par with FFT on NLU benchmarks [Guo et al., 2021; Hu et al., 2021] such as GLUE [Wang et al., 2018] or on relatively simple CNLG tasks such as E2E [Novikova et al., 2017]. However, our results indicate that such PLM adaptation may not generalize well across various architectures and for more complex tasks, high-resource benchmarks or large multi-task datasets. As demonstrated in this section (details below), PROPS outperforms all other model adaptation techniques on three high-resource benchmarks, two datasets in the low resource knowledge transfer setting and one high resource multilingual translation dataset.

4.1 General Setup

Baselines: We compare PROPS to four main model categories: fully fine-tuned (FFT) models, adapters where trainable layers are inserted between frozen PLM layers, UPG and CPG described in Section 3. A comparison of baselines, including Transformer-Prompt (TRSF-P) and TransformerPrompt with shared weights (S-TRSF-P) generators, is found in Table 2 where we show the percentage of PLM layers adapted, whether the PLM hidden states or the attention is adapted, the types of conditions used (none, embeddings or text prompts). The ADAPTER model uses trainable MLPs inserted in the PLM to transform hidden states. CA-MTL is type of multitask adapter that uses task embedding to conditionally adapt hidden states and the attention of the PLM. PREFIX++ is exactly the same as PREFIX however we additionally append conditions textual prompt to the input text (e.g.: "Abstractive Summarization, BBC, Sports" or "Jump Twice Jump Right"). The baselines allow us to measure the effects of conditions (PREFIX vs. TRSF-P or PREFIX++), the affects of sharing weights (TRSF-P vs. S-TRSF-P) and the affects of rule selection (S-TRSF-P vs PROPS). We also provide side by side comparisons of PREFIX, S-TRSF-P and PROPS in Figure 8 of the Appendix. More details are found in Appendix G.

Datasets, Training and Evaluation: We study four Conditional Natural Language Generation (CNLG) datasets SCAN [Lake and Baroni, 2018], Europarl [Koehn, 2005], XSum [Narayan et al., 2018] and Topic-CNN-DM [Mrini et al., 2021a]. The datasets and tasks for our experiment were chosen since text conditioning frommetadata and instructions is similar to controllable text generation. We describe our datasets, training and evaluation setup in Appendix E. The code and datasets will be made publicly available.

4.2 Can CPGModularization Better Compose Instructions by Composing the Input Text?

In the first type of evaluation depicted in Fig. 3, we test whether modular CPGs such as PROPS can compose instructions from the input. We fist want to test if the model learns from training inputs such as "jump right" and "walk left", how to properly resolve unseen combinations "jump left" or "walk right" at test time. We use a compositional generalization task made from the synthetic dataset SCAN [Lake and Baroni, 2018]. The SCAN dataset is constructed from a rule-based grammar that explicitly composes primitive commands. Lake and Baroni proposed various data splits to measure Compositional Generalization by testing on commands constructed with new primitives, excluded complex command sequences or extended length (length generalization). All adaptation techniques are built on top of a pretrained T5 model. In the "Add Jump" split for example, the training set includes all of the compositional tasks excluding "Jump", which is only seen in isolation, and the testing set includes all compositions using "Jump". The inductive bias in PROPS allows it to modularize knowledge via a sparse selection of attention heads. Experimental results in Table 3 show that PROPS's internal knowledge structure is also composable. Not only does PROPS surpass all other adaptation methods across splits by a large margin, it also improves over T5 FFT on five out of seven cases, including on the more difficult Maximum Compound Divergence (MCD) datasets [Keysers et al., 2020]. We follow the training setup of Furrer et al..

4.3 Can CPGModularization Better Transfer and Compose Jointly Trained Tasks?

Now, we turn our attention to the second evaluation case in Fig. 3: task composition. We hypothesize that modular CPG networks such as PROPS will better segregate and compose knowledge from other tasks trained jointly. We select the Europarl [Koehn, 2005] multilingual dataset since it contains numerous parallel language corpora. The Europarl multilingual corpora is also interesting since our instructions can leverage "bridge tasks". For example, we can translate English to French (en → fr) at test time if we only trained on English to German (en → de) and German to French (de → fr) pairs (i.e., translate en to de and then de to fr to get en to fr). Further, it was shown that multilingual translation can benefit from positive transfer from one language (task) to the other. As previously discussed, it is typically easier to outperform fully-finetuned models with parameter efficient adaptation methods in the low-resource setting. However, it becomes increasingly difficult as the data size scales [Raffel et al., 2020]. We evaluate the performance ---→ yy-xx and ←--- yy-xx where yy, xx ∈ {de,fr,en,es} and yy ̸= xx. With most multi-task learning frameworks, it is paramount to maximize positive transfer while minimizing negative transfers. Since PROPS is able to choose up to k rules to apply on a condition, multiple tasks can choose either common shared rules or independent rules, we believe this mechanism helps enhance positive transfer (shared rules), and mitigate negative transfer (independent rules). We present our mBART [Liu et al., 2020] based multilingual translation results below.

(a) Europarl trained with 50 samples

(b) Europarl trained with 500 samples

Figure 4: Few-shot multitask. PPS = PROPS. BLEU shown for 10 language pairs. Error bars are standard deviations of 3 random seeds.

The few-shot case: In this experiment, each of the 10 language pairs include 50 or 500 training samples. The task instruction have the form "Translate {source} to {target}". For CA-MTL, each translation pair generates different adapter weights from its task embedding. We do not run experiments for ADAPTER since Pilault et al. show that CA-MTL is better equipped to handle multiple jointly trained tasks. In Figure 4, we notice that both Condition Prompt Generators (PROPS and S-TRSF-P) performmuch better than PREFIX++. PROPS improves over all other methods on both average performance and standard deviation, including FFT, for 9 out of 10 tasks when each pair has 50 training examples, and for 8 out of 10 tasks when each pair has 500 training examples.

Full data multilingual experiments: Each language pair direction has 1M training and 100K testing examples. In the full data multitask regime, we see that fully finetuned mBART has an edge over adaptation methods. Our results demonstrate that almost all PLM adaptation struggle for a 4M combined dataset size. On average, PROPS is 1.8% lower than FFT but also 1.8% higher than the second highest performing adaptation method. Comparing the results of PROPS and S-TRSF-P to PREFIX++, it is clear that conditional prompt generation performs better than unconidional prompt generation. This is also true when we compare ADAPTER to CA-MTL (Figure 9), indicating that task-conditioned PLM adaptation is better suited at transferring knowledge. We start noticing that PROPS's modularization of knowledge from learnable rules and the control structure has clear benefits over S-TRSF-P.

4.4 Does Conditioning Prompt Generators on Data Metadata Improve Performance?

In this section, we assess PROPS and our baseline's performance when text conditioning is the news article metadata (third evaluation case). Our results are in Table 5. The instructions remain unchanged (i.e., "Abstractive Summarization") for each dataset and example. However, the metadata such as news outlet, article type or news topic (for Topic-CNN-DM) change for every example. XSum-OOT is a new dataset based on XSum where the training set consists of specific types of articles (i.e., "world", "politics", "business", "sports") and the test set is composed of the remaining news categories (i.e, "health", "tech"). XSum-OOT tests if models can generalize to unseen news categories and extrapolate from seen data metadata.

Overall, we notice that PROPS, other CPGs and even PREFIX++ all perform better with the added information. We then observe that PROPS consistently surpasses ADAPTER, PREFIX++ and our Conditional Prompt Generation baselines (TRSF-P and S-TRSF-P). We note that PROPS and S-TRSF-P adapt 33% of the layers with 5% fewer trained parameters, 50 fewer prompt vectors when compared to PREFIX. PROPS is the only adaptation technique that is consistently on par or outperforms FFT. For close to 4 times less parameters, PROPS consistently outperforms TRSF-P with statistical significance (except on Meteor scores), surprisingly offering a 1.5% relative improvement over TRSF-P on averaged scores across the three datasets. When the task requires more control on the generated output text, we see that PROPS rule selection mechanism boosts relative performance by close to 4% on Topic-CNN-DM [Mrini et al., 2021a] over S-TRSF-P. The improvement in the controlled generation task shows that the separate transformation of conditions help disentangle representations of the topic and other accompanying conditions.

4.5 Can we Compose Instructions, Metadata and Pre-Learned Support Task Knowledge?

In this section, we test the affects of pre-learning specific Conditional Language Generation support tasks along with their instructions. At test time, we evaluate our methods on Abstractive Summarization using an instruction that was textually composed from the pre-learned task instructions. We handpick three support tasks that are related to summarization. Indeed, summarization can be decomposed into several arguably simpler tasks [Pilault et al., 2020]: (1) NLU - understand and interpret a source document, (2) Extract - highlight and prioritize important points, (3) Paraphrase - coherently reformulate key messages. We describe a technique to induce transfer and composition via text conditioning. We vary the overlap of words describing a support task and a target task, and measure the effects on target task zero-shot and few-shot performance. In this setting, support tasks are trained jointly and the target task is evaluated separately. We provide examples of how to create target task descriptions from support task descriptions with certain word overlaps, which is color coded for clarity in Table 8. Note that we did not search over multiple instructions in the experiments below. The experimental setup and the support tasks datasets used or created are discussed in Appendix E.

The few-shot case: For this experiment, we use the same set-up as in the zero-shot section, except our target task is fine-tuned on a few examples. The intermediate task prompts used in this section are presented in Table 8 in the "simple" task instruction column. We also ran experiments on TopicCNN-DM to see if our results generalized to other datasets. Our few-shot summarization results are shown in Figure 5. We see that PROPS average performance is above that of other BART based techniques while also maintaining a lower spread between maximum and minimum values. This means that PROPS better leverages prior task instructions and knowledge to initialize new task prompts. Again, for controlable generation (Topic-CNN-DM), PROPS displays close to a 5 and 2 ROUGE-1 point and 3 and 2 ROUGE-2 point improvement over FFT and PREFIX++ respectively.

The zero-shot case: We first test if pretrained task and "Simple task instructions" in Table 8 are enough to solve our target task without any additional fine-tuning. CPG modularization approaches such as PROPS should improve the composition of task instructions by reusing words and phrases found in support tasks and further enhance PROPS's rule reusability and knowledge composition ability. From our results in the "Simple" column of Table 6, we first notice that PROPS tops S-TRSF-P, our strongest baseline, by 0.8 R1 or a 4% relative improvement.

Improving task instructions: We evaluate if it helps to improve the task instructions content and support-target word overlap. The detailed target instruction contains word segments from all three support tasks whereas the simple target instruction is composed from only two tasks. As shown in Table 8, "Detailed task instructions" allow us to fully connect each support task with the target task, while providing additional and potentially relevant information. Further, the detailed target instruction are lengthier and allow greater overlap between words describing the support tasks as well.

Our hypothesis is that greater overlap and lengthier instruction will improve target task performance by better inducing prior knowledge composition. Task description composition by assembling task descriptions works in conjunction with module composition (see explanation in Appendix C and an empirical analysis in Appendix D.3). Our results suggest that all models benefit from the improved task instruction as observed in the "Detailed" column of Table 6. Specifically, PROPS gets the largest % ∆ gain when moving from "Simple" to "Detailed" instruction, with a 10% increase in R2 while maintaining the same standard deviation.

5 Conclusion

In this work, we evaluated conditional and composable architectures to conditionally generate differentiable continuous prompts. Our new modular prompt generator PROPS with sparse rule selection outperforms strong Transformer-based prompt generator baselines. PROPS excels in the ability to distill knowledge from the conditioned task and example metadata, and often surpasses the fully fine-tuned model, with fewer trainable parameters than other prompt generators. In general, PROPS performed well on three tasks - summarization, translation, and semantic parsing, with three different models (BART, mBART, T5). PROPS provided even larger improvements in controlled CNLG, where disentangling topic information from other conditions is important. The improvement is even larger for task which requires more controlled generation. PROPS showed ability to compose knowledge from bridge tasks to enhance transfer learning and improve unseen tasks in few-shot setting both from a mechanistic and induced standpoint. Finally, via both theoretical (see Appendix D) and empirical analysis, we showed how rules are sparsely selected and composed, the key ingredients that make PROPS more successful and generalizable.