Continual Compositional Zero-Shot Learning

1 Introduction

Human beings can decompose observations into primitive concepts and recombine these primitive concepts to generalize to unseen compositions. Compositional Zero-Shot Learning (CZSL) [Misra et al., 2017] is proposed to mimic the way humans recognize unseen compositions which is often regarded as a hallmark of intelligence [Atzmon et al., 2016], [Lake et al., 2017], [Li et al., 2023]. Generally, CZSL methods are trained on seen compositions and deployed to recognize unseen compositions, both of which are composed of two primitives (attribute and object) in the fixed set. Such methods are infeasible for many real-world applications as they cannot continually learn and accumulate knowledge of new primitives that might emerge after training. On the contrary, humans can not only recognize unseen compositions, but also learn new concepts incrementally throughout their whole lives.

∗Corresponding Authors

In recent years, sporadic researches have emerged in Continual Generalized Zero-Shot Learning (CGZSL) that require model to continually recognize new unseen categories. [Wei et al., 2020] considers each GZSL dataset as an incremental session and accumulate different attribute knowledge from multiple datasets. However, this method needs task-id during testing phase and only test on individual GZSL dataset, which is difficult to meet in realistic scenarios. Then, [Wu et al., 2023b], [Skorokhodov and Elhoseiny, 2020] and [Gautam et al., 2020] divide a dataset into multiple subsets in different ways to simulate the continual learning process. Although these methods propose various reasonable ways to divide the dataset, compared to the first method, these methods only focus on continual learning of categories and ignore continual learning of attributes. That is to say, the above incremental sessions may not bring new attribute knowledge to the model. As the categories all come from the same attribute set and there are no explicit constraints on attribute increments for each incremental session.

Drawing on the experience of CGZSL mentioned above, we propose a new Continual Compositional Zero-Shot Learning (CCZSL) setting. This setting requires the model to recognize unseen compositions composed of learned primitive set while continually increasing the size of this set. As illustrated in Figure 1, we set up multiple incremental sessions to bring new primitives step by step. Concretely, the model learns new primitives through new seen compositions in each incremental session and is ready to recognize unseen compositions composed of primitives in the expanded set, that is, unseen compositions of all sessions up to the current session. From another perspective, we split the standard CZSL into multiple CZSL sub-tasks, as newly added primitives and old primitives are combined into several seen compositions and unseen composition like standard CZSL. After learning the last incremental session, the model learns the same number of attributes, objects and compositions in standard CZSL dataset. Next, we will introduce two challenges in this setting, 1) Contextuality and 2) Catastrophic Forgetting, and their corresponding solutions.

Specifically, contextuality indicates that the appearance of the same attribute (object) undergoes significant changes when combined with different objects (attributes). For example, the same attribute "old" depicts retro style for object "car" but aged appearance for object "dog". The same object "church" exhibits a complete structure in composition "new church", yet only damaged structure and debris are bespoke in "burnt church". [Zhang et al., 2022] ignores the contextuality and aims to learn generic primitive embeddings, which are then used to predict the corresponding primitives in all compositions. [Hu andWang, 2023] learns multiple embeddings for each primitive according to compositions, but is hard to achieve due to data scarcity. Unlike the methods mentioned above, we propose to additionally learn multiple super-primitives which can modify the invariant primitive embedding to better fit contextuality in the current composition. We found that although contextuality significantly varies appearances of primitives, there are still consensuses in these variations. For example, the super-object "mammal" beyond "dog", "cat" and "elephant" has the similar contextuality, like dull fur and sagging skin, when combine with attribute "old". In the same way, the super-attribute "fragmented" beyond "broken" and "sliced" tends to change appearance of object by breaking object into multiple small pieces. Therefore, we propose to learn super-primitives from original primitives, which encapsulate primitives with similar contextuality in compositions. During the inference phase, we recognize attribute (object) by introducing superobject (super-attribute), which can be utilized to simulate contextuality in the current composition and modify the invariant attribute (object) embedding.

Only using samples of new categories to train model will cause forgetting of old knowledge, resulting in a decrease in overall performance, which is known as catastrophic forgetting [Kirkpatrick et al., 2017]. In this setting, as the learned primitives emerge in new compositions with fresh contextuality during subsequent incremental sessions, the model experiences catastrophic forgetting when attempting to relearn these primitives in the new compositions. To be specific, introducing new primitives in incremental sessions leads to the creation of new compositions that incorporate both the new and old primitives. The newfound contextuality within these compositions changes visual appearances of the old primitives, which are distinct from those observed in previous sessions. Learning from the above new compositions will lead to the forgetting of the old primitive characteristics. Data regularization is a popular and concise solution in continual learning, which aims to minimize the impact of new incremental sessions on weights important for previous sessions. In this paper, we propose to utilize knowledge distillation to keep the learned embedding of previous primitives from drifting too much. To maintain the characteristics of networks learned from previous sessions, we adjust the scope of knowledge distillation, introducing constraints on predicting new categories as well.

The main contributions of our work are summarized as follows:

• To the best of our knowledge, we are the first to propose and tackle CCZSL. This setting holds great significance and practicality for real-world applications. Additionally, we design the evaluation protocols and conduct comprehensive experiments.
• We propose to address the issue of contextuality by learning multiple super-primitives corresponding to similar contextuality. Then, we propose a dual knowledge distillation loss to keep old primitive knowledge from catastrophic forgetting and mitigate overfitting on new primitives.
• Experiments and ablation studies affirm the effectiveness of our proposed method.

2 RelatedWork

2.1 Compositional Zero-Shot Learning

Zero-Shot Learning (ZSL) [Palatucci et al., 2009][Zhang and Feng, 2023][Tang et al., 2020] is a popular research topic that provides model with shared semantic knowledge (attribute knowledge) between seen and unseen classes, enabling them to recognize unseen classes without any training data. CZSL is a sub-topic of ZSL, which requires model to recognize unseen compositions with the shared primitive knowledge. In the early attempts at CZSL, methods try to directly learn the classifiers of compositions by transforming the paired primitive embeddings into composition embedding. For example, [Misra et al., 2017] projects the image representations and paired primitives into a common space, and then predicts unseen compositions in the common space. [Nagarajan and Grauman, 2018] and [Li et al., 2020] treat attributes as operators which change original objects into compositions. [Naeem et al., 2021] tries to exploit the dependency between attributes, objects and their compositions within a graph structure to transfer the relevant knowledge from seen to unseen. Recent mainstream works disentangle visual representations of primitives and learn corresponding embeddings respectively. [Li et al., 2022] leverages contrastive loss to excavate discriminative embeddings of primitives. [Zhang et al., 2022] considers CZSL as a Out-Of-Distribution problem and learns invariant primitive embeddings. [Saini et al., 2022] and [Hao et al., 2023] implement cross-attention between two samples sharing the same primitive to disentangle the corresponding representation of this primitive. [Wang et al., 2023] proposes to learn attribute embeddings that rely on objects and images. [Kim et al., 2023] disentangles attribute representation from feature map with the guidance of object visual representation.

2.2 Continual Generalized Zero-Shot Learning

Continual Generalized Zero-Shot Learning (CGZSL) is the most related work to ours. [Wei et al., 2020] is the first work to investigate GZSL in continual learning scenario. They construct a sequence of tasks by assembling existing GZSL benchmarks and the performance is reported for each benchmark individually. However, this setting requires a task-level supervision in the form of task-ids at test time which is hard to achieve. Subsequently, some follow works propose new CGZSL settings, such as [Skorokhodov and Elhoseiny, 2020] and [Gautam et al., 2020]. The former randomly divides individual dataset into T tasks and assumes all previously encountered tasks as seen classes and future tasks as unseen classes. The latter equips an exclusive set of seen and unseen classes for each task and the model can accommodate any number of tasks. Due to the popularity of generative methods [Xian et al., 2018][Tang et al., 2022] in GZSL, generative replay is the mainstream technique for mitigating catastrophic forgetting in the above CGZSL. Although the recent CGZSL settings are more practical, the construction of incremental sessions still has flaws. As attribute plays a important role in GZSL, dividing the dataset based on attribute knowledge is better than random splitting. That is, introducing new attributes for each session, and the new categories are naturally selected from the dataset based on the new attributes.

2.3 Knowledge Distillation

Knowledge distillation (KD) [Hinton et al., 2015] was originally designed to learn a more compact student network from a larger teacher network. Then, it became a major technique for data regularization methods in continual learning. [Li and Hoiem, 2017] first leverages this technique by constructing a cross-entropy loss, known as learning without forgetting loss, to keep representations of previous data from drifting too much while learning new task. This loss was initially employed in task-incremental learning (task-IL) methods [Titsias et al., 2019]. Subsequently, owning to numerous studies observing its effectiveness in settings with small domain shifts between tasks, such as class-incremental learning (class-IL)[Chi et al., 2022], [Wu et al., 2023a], the loss has become a crucial component in many class-IL methods, such as [Dhar et al., 2019] and [Zhang et al., 2020]. It is worthy noting that, as reported in [Masana et al., 2022], continual learning and class-IL are mostly the same.

3 Methodology

3.1 ProblemFormulation

CCZSL requires the model to recognize unseen compositions composed of learned primitive set while continually increasing the size of learned primitive set. To simulate the continual learning scenario, we split the original CZSL dataset into T+1 sessions {D0 ,D1 , ...,DT }, where each session consists of a training set Di tr and a testing set Di ts . Specifically, let Di tr = {(x, a, o, c)|x ∈ X i tr , a ∈ Aitr , o ∈ Oi tr , c ∈ C i tr }, x denotes the image in the sample space of i-th session X i tr and c is the composition label in the label space of i-th session C i tr , each label c is composed of the attribute a and the object o in the tuple respectively, i.e., c = (a, o). In the same way, the testing set is defined as follows Di ts = {(x, a, o, c)|x ∈ X i ts , a ∈ Aits , o ∈ Oi ts , c ∈ C i ts }. From the perspective of increasing sessions, the label spaces of the training set in different sessions are disjoint, i.e., C i tr ∩C j tr = ∅, but the label space of the testing set in current session are the summary of label spaces from all previous sessions, i.e., ∀i < j, C i ts ⊂ C j ts . The primitive set follows the same pattern mentioned above in both training and testing set, i.e., ∀i < j, P i tr/ts ⊂ P j tr/ts , P = {A, O}.

3.2 Preliminary

We first outline the fundamental pipeline of the recent mainstream methods. When provided with an image x, these methods initially utilize a backbone network, i.e., ResNet18, to extract its high-level visual features denoted as X. Subsequently, three feature encoders are respectively employed to disentangle the corresponding attribute, object, and composition representations, which are defined as follows:

To train a model capable of understanding primitives and composing them to recognize unseen compositions, a common approach is to directly maximize the similarity scores, such as cosine similarity, between these extracted representations and corresponding embeddings. The function s(·) calculates the similarity scores as follows:

where p = {a, o}. vp and ep are the corresponding representation and embedding. vc refers to composition representation. φc(·) is the transformation network that transforms paired primitive embeddings into composition embedding.

3.3 Capturing Contextuality by Super-Primitives

Generally, relying on contextuality-free and invariant primitive embeddings to recognize the diverse primitive representations in different compositions is evidently insufficient. Therefore, we propose to learn K super-primitives to capture the similar contextuality, where K is smaller than the number of original primitives. Taking the recognition of attribute as an example, by combining the original attribute embedding with the super-object embedding, we can modify the original attribute embedding based on the contextuality provided by super-object embedding. Similarly, the process of leveraging super-attribute to recognize objects follows a similar way.

Actually, when recognizing a single primitive, we cannot know in advance the super-primitive which another primitive in the current composition belongs to. Therefore, we utilize a super-primitive selection module to determine which superprimitive the another primitive should belongs to. First, we can calculate the correlations between the primitive and all the super-primitives in the latent space as follows:

where f1(·) and f2(·) are the projection layers which project them into a latent space. Subsequently, we utilize the correlation coefficient wp to obtain the final super-primitive:

However, when the module is equipped with the ability to select among K super-primitives, the model could potentially learn to use only a subset of super-primitives for prediction. Inspired by [Huynh and Elhamifar, 2020], we introduce a loss function Ldiv to ensure that each of the K super-primitive will be used for prediction:

where the term k in the first sum indicates the correlation coefficient between primitive representation and k-th superprimitive. The term i in the second sum pertains the i-th sample in the current batch. This loss encourages the module F to distribute its focus across all super-primitives.

With the assistance of super-primitives, we can reformulate the calculation of similarity scores for primitives as follows:

where sa and so represent the selected super-primitives. φa(·) and φo(·) are the transformation networks for each primitive, which transform the original primitive embedding and the corresponding super-primitive embedding into the final primitive embedding with the information of contextuality.

To learn super-primitives, we utilize cross-entropy loss to guide the transformed primitive embeddings towards the corresponding primitive representation:

Additionally, we deploy the cross-entropy loss for recognizing composition:

where τ is the temperature. Aitr , Oi tr and C i tr denote the label space of attribute, object and composition in the training set from the i-th session respectively. Notably, C i tr only includes new compositions, while Aitr and Oi tr encompass all the primitives learned so far. Although jointly optimizing all categories is often associated with error propagation, it may even bring a slight improvement in this issue.

3.4 Mitigating Catastrophic Forgetting by Dual Knowledge Distillation

We now introduce the first application of knowledge distillation (KD) in continual learning, namely Learning without Forgetting (LwF) [Li and Hoiem, 2017]. In brief, LwF takes the model trained from the previous sessions to generate soft labels exclusively for the old categories of the new data, and these soft labels are then used as supervision for the current model. The LwF loss is defined as follows:

where N t-1 indicates all the categories before t-th session. πt-1 and πt are the predictions made by the previous model and the current model.

An illustration of KD between the model trained in previous sessions and the model trained in current session is shown in Fig.2b. In this paper, we propose a dual knowledge distillation which simultaneously mitigates catastrophic forgetting and overfitting. Different from the LwF loss, we also apply KD on the new categories. The reason lies in that, after training on new data, the parameters of main components tend to bias towards the new categories. The dual KD can help the model preserves old primitive embeddings while avoiding the bias of old components towards the new categories. Here, we introduce more details about the dual KD loss:

where N t indicates the number of all categories up to t-th current session, which can be attributes, objects or compositions. pn denotes the prediction of the n-th category generated by Algorithm 1 The optimization procedure of CCZSL.

Input: training sequence {D0 tr , D1 tr , . . . , DT tr }, learning rate γ, weight λ Output: θ = {θE, θF , θφ, θS, θP} 1: randomly initialize parameters θ 2: for each session t ∈ [0, T] do 3: if t > 0 then 4: copy old parameters as θold 5: add new primitive embeddings to θP 6: while not converaged do 7: copy new primitive embeddings from θP to θold P 8: θ = θ - γ∇θ(LCZSL(Dt tr ; θ) + λLKD(Dt tr ; θold , θ)) 9: end while 10: else 11: while not converaged do 12: θ = θ - γ∇θLCZSL(Di tr ; θ) 13: end while 14: end if 15: end for 16: return θ = {θE, θF , θφ, θS, θP}

the model trained in the current session, and ˆ yn represents the soft labels generated from the previous model:

where δ(·) represents softmax operator. For convenience, we simplify the process of obtaining soft labels from previous model into three components Vt-1(·), Tt-1(·) and Pt-1, as shown in the Figure 2b. The subscript t -1 indicates that the component comes from the model trained after the (t - 1)-th session. It is worth noting that when generating soft labels for new categories, we use the primitive embeddings in the current model and keep updating the new primitive embeddings.

3.5 Training and Inference

At the training phase, we formulate our final loss as follows:

During the inference phase, we gather the predictions of primitives and compositions to obtain the final prediction:

4 Experiments

4.1 Experiment Setting

Datasets. We conduct experiments on two widely adopted datasets in CZSL, which are UT-Zappos [Yu and Grauman, 2014] and C-GQA [Naeem et al., 2021]. UT-Zappos is a finegrained shoes dataset which contains 50,025 images, with 16

attributes and 12 objects. It comprises 83 seen compositions and 15/18 (validation/test) unseen compositions. C-GQA is a natural image dataset which contains 39,298 images, with 413 attributes and 764 objects. It includes 5,592 seen compositions and 1,040/923 (validation/test) unseen compositions.

Datasets Split In our CCZSL setting, we assume that introducing new primitives into each session is a prerequisite, followed by the spontaneous introduction of new compositions through the combination of new and old primitives. Whether the composition is seen or unseen follows the same setting as the original dataset. Specifically, We split UT-Zappos into 3 sessions and C-GQA into 6 sessions. Details of splits are present in Table 2, including number of new attributes, new objects, new training compositions, new validation compositions and new testing compositions in each session. We allocate more primitives for the first session, aiming to let the model learn a good initialization.

Evaluation Metrics The harmonic mean H between the top1 accuracy on seen S and unseen U classes is a widely used evaluation protocol in GZSL, i.e., H = 2 × S × U/(S + U). However, due to the model only accessing seen data during training, there is a bias towards seen data during testing, leading to a decrease in the harmonic mean. To alleviate this issue, adding a scalar bias to the prediction of unseen compositions to calibrate the final result has become a widely used technique in CZSL. In this paper, we use the area under the curve (AUC) of H with the scalar bias range from -∞ to ∞ to assess the CZSL performance. And we utilize the average AUC across all the sessions to report the overall performance.

Implementation Details We use ResNet-18 pre-trained on ImageNet to extract 512 dimensional vector, following preceding work [Wang et al., 2023]. The three encoder networks and transformation networks share the same structure of two Fully Connected (FC) layers with ReLU, LayerNorm and Dropout. The super-primitive selection module is implemented with multi-head attention mechanism. The number of super-primitives K is set to 4 for UT-Zappos and 20 for C-GQA. The original primitive embeddings are learned from scratch. For all datasets, we train the model using Adam optimizer with a learning rate of 5 × 10-5 . The temperature factor is 0.05 for all datasets.

4.2 Comparing with the State of the Art

Baseline. Since CCZSL is a newly proposed setting in CZSL field, there does not exist any prior works that can be used for comparison directly. Therefore, we conduct experiments and compare our method with the following state-of-the-art methods in CZSL: 1) AttrAsOp [Nagarajan and Grauman, 2018] views attributes as transformation operations rather than high-dimensional embeddings. It employs various regularization to optimize these operations. 2) SymNet [Li et al., 2020] introduces symmetry into the attribute-object transformation and learns the transformation under the supervision of group theory. 3) VisProdNN [Karthik et al., 2021] revisits visual-only methods and considers CZSL as a multitask problem, predicting objects and attributes separately. 4) SCEN [Li et al., 2022] learns embedding of attributes and objects in a siamese contrastive space and proposes a State Transition Module to increase the diversity of training compositions. 4) CANet [Wang et al., 2023] proposes to recognize attribute conditioned on objects and designs an attribute hyper learner to generate flexible attribute embeddings.

Results and Analysis. Table 1 summarizes the results of all the comparison methods. When comparing the results on UT-Zappos, our method achieves the best AUC in each session. Specifically, we obtain the highest AUC of 45.48%, 24.73% and 18.96% in each session, surpassing the secondbest methods by 2.22%, 4.39% and 2.71% respectively. For the average AUC over all the sessions, we gain 4.19% improvement over the second-best method. Comparing the results on the more challenging dataset C-GQA, we achieve the best AUC in the subsequent incremental sessions. Specifically, our model achieves the best AUC of 3.89%, 3.88%,

2.74%, 3.32% and 1.83% in each subsequent incremental sessions respectively. At the same time, we gain improvements of 1.40%, 1.98%, 1.57%, 1.05% and 0.83% compared to the second-best methods. For the average AUC over all the sessions, we gain a 1.12% improvement over the second-best method. We attribute the success of the proposed approach to the super-primitives and the dual KD. The improvement in the first session illustrates the effectiveness of our superprimitives. By introducing contextuality into invariant embeddings, we achieve improved recognition of diverse primitives which can further improve the accuracy of predicting compositions. Subsequent improvements highlight the importance of mitigating catastrophic forgetting in this setting. Our proposed dual KD can not only preserve old primitive embeddings, but also maintain the characteristics of old network which can significantly improve the performance.

4.3 Ablation Analysis

Impact of main components. To validate the effectiveness of our method, we conduct ablation analysis in Table 3. Specifically, Baseline is a plain pipeline mentioned in Section 3.2. Then we add the dual KD loss LKD to mitigate catastrophic forgetting. As we can see, dual KD can effectively mitigate catastrophic forgetting. To help the model better understand the contextuality, we introduce super-primitive into the recognition of primitive which improves the average AUC on both datasets. At last, we add diversity loss Ldiv to avoid the model from finding shortcuts, which can further improve overall performance.

Impact of λ. The scale of λ can control the balance between learning new knowledge and keeping old knowledge. We can observe that the best average AUCs for both UT-Zappos and C-GQA are achieved when λ equals 0.5 and 1, respectively. This is mainly because C-GQA contains much more and diverse compositions. The significant differences between new and old compositions results in the need for more constraints on old knowledge in C-GQA.

Impact of dual knowledge distillation. We conduct experiments to examining the effects of original KD and dual KD. In Table 4, we report the average AUC across all the sessions. We can observe that dual KD can better prevent the performance degradation caused by the model excessively biasing towards the new categories. But the original KD can only maintain the knowledge of old primitive embeddings.

Impact of joint optimization. We report the average AUC of joint optimizing all primitives or compositions. ✓ indicates the use of joint optimization for the corresponding category. We can observe that, jointly optimizing old and new primitives may enhance discrimination between primitives, resulting in improved performance. However, the number of compositions is greater than the number of primitives and the inter-compositions differences are smaller, which will lead to error propagation during joint optimization process.

5 Conclusion

In this paper, we propose a practical and challenging setting called CCZSL, which requires the model to recognize unseen compositions composed of learned primitive set while continually increasing the size of learned primitive set. To solve contextuality issue, we propose to learn super-primitives to capture the similar contextuality in part of compositions, which is beneficial for recognizing the various primitives. Then, we propose a dual knowledge distillation loss to better preserve old knowledge by performing knowledge distillation on new categories. Extensive experiments demonstrate the superiority of our proposed method.

Acknowledgments

This work was supported by the Beijing Natural Science Foundation (No.4242046), the Tangshan Municipal Science and Technology Plan Project (No.23130225E) and the Fundamental Research Funds for the Central Universities (No.2022JBZY019).