Seeing the Unseen: Composing Outliers for Compositional Zero-Shot Learning

1 Introduction

Compositional generalization, understanding unseen combinations composed of seen primitives, is one of the fundamental properties of human intelligence [Fodor and Pylyshyn, 1988]. To evaluate such ability of vision models, compositional zero-shot learning (CZSL) [Misra et al., 2017;

∗ corresponding author

Purushwalkam et al., 2019] requires recognizing unseen attribute-object compositions by learning from seen compositions in training images, as shown in Fig. 1 (a). In particular, CZSL usually follows a practical and challenging generalized zero-shot learning setting [Pourpanah et al., 2022; Chen et al., 2020; Liu et al., 2021a], where the test set contains both seen and unseen classes, in evaluation. Since the label combinations of unseen compositions were never observed during training, the distribution shift between the seen compositions and unseen compositions is significantly large, which poses challenges to CZSL models [Atzmon et al., 2021; Mancini et al., 2022]. The learned correlations of models in training may be detrimental at test time.

Previous works deal with the distribution shift via invariant representation learning [Atzmon et al., 2021; Zhang et al., 2022], prototypical representation learning [Ruis et al., 2021] or propagating information of seen primitives for unseen compositions [Naeem et al., 2021; Mancini et al., 2022]. They aim to recognize unseen compositions well by adding structural constraints, which may reduce the useful correlations for seen compositions and influence the performance. However, considering the potential differences between unseen and seen compositions, the tasks can be handled more easily if a model can distinguish the seen and unseen compositions and treat them with specific recognition strategies, such as specific expert classifiers. In this work, we explore an explicit way to handle the distribution shift caused by unseen cases, separating images of seen compositions from those of unseen compositions and using different strategies for them. Nonetheless, identifying unseen compositions at test time is non-trivial. The unseen compositions are absent in model learning, and thus their distribution is unknown. Sharing similar primitives with seen compositions, they may be only subtly different from seen compositions.

To address these challenges, we present a novel compositional zero-shot learning method called COMO that composes outliers in training for identifying unseen compositions and further using different strategies for seen/unseen attribute-object compositions. The method composes imagelevel representations for unseen attribute-object compositions based on learned representations of seen primitives. These composed representations are regarded as outliers of the training distribution to encourage the model to identify out-ofdistribution (OOD) samples at testing. Besides, the disentanglement of the primitive representations and the discriminability of the composed representations is considered to guarantee appropriate composed representations.

Specifically, our method combines a composition classification module and a primitive classification module for composition recognition, as shown in Fig. 1 (b). The former module characterizes compatibility between images and candidate compositions, while the latter module independently characterizes the compatibilities between images and the candidate attributes and objects. By enforcing the composition classification module to output the uniform distribution for composed representations, the module learns heuristics to identify images with unseen composition. Considering the two classification modules focus on different perspectives of an image and can supplement each other [Yang et al., 2022; Wang et al., 2023b; Huang et al., 2024], we assign different weights for seen/unseen compositions to combine the two modules to achieve composition recognition. The experimental results on three widely used CZSL datasets under both the closed-world setting and the open-world setting show the effectiveness of the proposed method.

The contributions of this paper are summarized as:

1. We propose a novel compositional zero-shot learning method that can identify images with unseen compositions at test time and uses different strategies for seen/unseen compositions.
2. Our method composes unseen attribute-object compositions based on primitive representations to obtain outliers for the model to identify unseen compositions.

2 RelatedWork

Compositional zero-shot learning. The task of compositional zero-shot learning aims to recognize unseen attributeobject compositions by learning from seen compositions. Existing methods mainly achieve the task via composition classification with a composition classifier [Misra et al., 2017; Naeem et al., 2021], or primitive classification, which independently recognize attributes and objects, [Li et al., 2020; Purushwalkam et al., 2019], or combines composition classification and primitive classification for better contextuality [Yang et al., 2022; Wang et al., 2023b]. With the recent advance in pre-trained vision-language models, CLIPbased CZSL methods [Nayak et al., 2023; Lu et al., 2023; Huang et al., 2024; Bao et al., 2023] achieved state-of-the-art performance. CSP [Nayak et al., 2023] first uses the CLIP [Radford et al., 2021] in CZSL. They replace the classes in textual prompts with trainable attributes and object tokens. Troika [Huang et al., 2024] jointly models the visionlanguage alignments for the attribute, object, and composition using the CLIP. PLID [Bao et al., 2024] leverages pre-trained large language models to enhance the compositionality of the softly prompted class embedding. The aforementioned work mainly focuses on parameter-efficient fine-tuning of CLIP. By contrast, our method focuses on identifying unseen compositions and designing different strategies for seen/unseen compositions and only uses CLIP as the backbone.

Distribution shift in CZSL. Atzmon et al.[Atzmon et al., 2021] points out that the distribution shift between seen compositions and unseen compositions is a fundamental challenge for CZSL. They propose to ensure conditional independence between attribute and object representations via causal inference to handle this issue. Naeem et al.[Naeem et al., 2021] and Mancini et al.[Mancini et al., 2022] use graph convolutional networks to extract attribute-object representations and propagate information from seen compositions to unseen compositions to improve the generalization ability. Ruis et al.[Ruis et al., 2021] learned compositional prototypes of novel attribute-object combinations that reflect the dependencies of the target distribution. Zhang et al.[Zhang et al., 2022] treated CZSL as a domain generalization task, and proposed to learn attribute-invariant and object-invariant representations for CZSL. Li et al.[Li et al., 2023] believe the influence of the distribution shift for the composition classification is larger, and only use a primitive classification module for composition recognition. Different from these works, we explicitly separate the seen class instances from those of the unseen classes, and independently classify seen and unseen class data samples via domain-specific strategies.

Generalized zero-shot learning (GZSL). GZSL is a typical setting of zero-shot learning (ZSL) to imitate the human capability of recognizing samples from both seen and unseen classes [Pourpanah et al., 2022]. Previous work explores embedding-based [Min et al., 2020; Hu et al., 2023] or generative-based [Verma et al., 2021; Liu et al., 2021b] methods for GZSL. In this work, we borrow the idea of embedding-based GZSL methods to deal with the distribution shift issue of CZSL, and propose a simple but effective method tailored for CZSL.

3 Method

In this section, we formulate the CZSL task and illustrate the proposed method, as shown in Fig. 2. Specifically, we illustrate how we achieve composition recognition based on outof-distribution (OOD) detection by combining two classification modules, introduce how we compose representations of unseen compositions as outliers in training to enable the model to identify OOD samples, and introduce the overall learning objective.

3.1 Formulation

Compositional zero-shot learning aims at learning a model from limited compositions of attributes (e.g., young, wet) and objects (e.g., cat, dog) to recognize an image from novel compositions. Given an attribute set A = {a1, a2, . . . , a|A|} and an object set O = {o1, o2, . . . , o|O|}, the compositional class set C = A × O is defined as their Cartesian product. The class set C can be divided into two disjoint sets, the seen set C s and the unseen set C u , where C s ∩ C u = ∅ and C s ∪ C u ⊂ C. In particular, CZSL follows the generalized zero-shot learning setting [Pourpanah et al., 2022; Chen et al., 2020; Liu et al., 2021a], where the training images only contain classes from the C s and the images of testing set contains both seen classes and unseen classes.

Given a test image I ∈ I, the CZSL task requires a model to predict a class label c = (a, o) from the testing class set. In the closed-world setting, only the known compositions (compositions of the whole dataset) are considered, i.e., C test = C s ∪ C u . That is, the test class set contains all seen classes for the training images and unseen classes of the test set. Thus the test class set is only a subset of the compositional class set C. By contrast, in the challenging openworld setting, the test class set is all possible compositions, i.e., C test = C. Formally, the model is required to model a score function S : I × A × O → R between an image I and a candidate composition. During inference, the candidate composition with the highest score is regarded as the final prediction.

3.2 Composition Recognition Based on OOD Detection

Feature encoding. We use the image encoder and text encoder of the CLIP [Radford et al., 2021] as the visual backbone and textual backbone, respectively. Given an input image I, we use the visual encoder of CLIP to obtain the representation of the [CLS] token v ∈ Rd , and then use three multi-layer perceptions (MLPs) to obtain image representation vI ∈ Rd , attribute representation va ∈ Rd , and object representation vo ∈ Rd , respectively.

For candidate compositions, attributes, and objects, we use the soft prompt [Nayak et al., 2023] to obtain textual representations, by using a prompt template like "a photo of [class]". We feed the text encoder of CLIP with "a photo of [attribute] [object]", "a photo of [attribute] object", and "a photo of [object]" for each candidate composition, attribute and object to obtain their textual representations T c ∈ RNc×d , T a ∈ RNa×d , T o ∈ RNo×d , respectively. The Nc, Na, and No are the numbers of candidate compositions, attributes, and objects, respectively. Specifically, the [attribute] and [object] tokens are trainable and initialized with the corresponding word embeddings of CLIP.

Two types of classification modules. To fully characterize the contextuality of attributes and objects, we use two types of classification modules: a composition classification module and a primitive classification module, to achieve composition recognition based on the above-mentioned representations. The composition classification models the composition compatibility Sc (I, a, o). The primitive classification models the attribute compatibility Sp a (I, a), the object compatibility Sp o (I, o), and further obtain primitive-level scores Sp (I, a, o). Specifically, for an input image I and a candidate composition ci = (aj, ok), the composition classification module measures the compatibility score as

where cos(·, ·) denotes the cosine similarity function of two vectors. The primitive classification module independently measures the compatibility scores between an image and the attribute aj and the object ok, and computes the primitivelevel scores by using the aforementioned primitive-level visual and textual representations as

We optimize the two modules in training with the crossentropy loss as

where cgt = (agt, ogt), are the ground truth composition, attribute, and object for the image I, respectively. We set equal weights for the two modules to let them supplement each other.

Combining classification modules for recognition. At test time, we identify images with unseen compositions through OOD detection by using the maximum logit value before the softmax layer [Jung et al., 2021], and then assign different fusion weights for the above-mentioned classification modules for composition recognition.

For a testing image, we select the maximum value of the predicted distribution lmax of the composition classification module over the seen compositions. By performing threshold comparison with a pre-defined threshold T , we can obtain ˆ y, which is the predicted label about whether the image contains a unseen composition. ˆ y is set as 1 if lmax <= T and 0 otherwise. Then we compute the final compatibility score for the image with a candidate composition as

where λs and λu are weights of fusing the two scores for images with seen compositions and unseen compositions, respectively. Finally, the candidate composition with the highest overall score is predicted.

Specifically, the validation set is used to obtain the threshold T and two weights. We first cast identifying unseen compositions in the validation set as a binary classification task based on the maximum probability, and select the best threshold T by varying its value. Then the best threshold is used to classify images in the validation set into two groups: images with seen compositions and images with unseen compositions. We further use grid search to find the best fusion weights λs and λu on the validation, to combine the two classification modules.

3.3 Composing Outliers in Training for Composition Recognition

To identify images with unseen compositions, we borrow the idea of outlier exposure [Hendrycks et al., 2018] to encourage the model to learn effective heuristics for out-of-distribution detection, by exposing OOD examples to the model.

Composing unseen compositions. In real applications, it may be difficult to know the distribution of testing images one will encounter in advance. Fortunately, in CZSL, the set of possible compositions can be composed using seen classes. Thus by using the primitive representations of seen compositions, representations of unseen compositions can be easily obtained.

Intuitively, we can build the representation for an unseen composition wet cat, by using the attribute representation of an image about wet ground and the object representation of an image about young cat. Motivated by the famous mathematics of computational linguistics "King - Man + Woman = Queen", we simply add two primitive representations to obtain composition-level representation for an unseen composition ui = (aj , ok) in model learning as

where ˆ va j /ˆ vo k is the attribute/object representation of the image containing attribute aj or object ok, respectively. Specifically, we compose features for all possible compositions that do not appear in the training set, rather than the unseen compositions in the validation set or the testing set. The GloVe [Pennington et al., 2014] is used to obtain the feasibility calibration of each possible composition for filtering out infeasible compositions. In model learning, we randomly select the attribute/object representations of the current training batch to obtain representation for a given unseen composition. Specifically, in model learning, we can obtain various unseen compositions based on a batch of Nbs training images. For simplicity, we also generate representations for Nbs unseen compositions, by randomly sampling primitives of the batch.

Constraints for feature composing. To obtain appropriate composed representations for unseen compositions, we enforce some constraints on the composed representations for unseen compositions and the primitive representations of training images, taking into consideration the disentanglement of the primitive representations, and the discriminability of the composed representations.

We first introduce a disentanglement loss to learn more disentangled primitive representations, considering the complex semantic entanglement of the attribute and the object in an image. We penalize the object representations for predicting the ground truth attribute labels, and attribute representations for predicting the ground truth object labels, and compute the loss as

where T o gt and T a gt are the textual representations for the ground-truth object label and ground-truth attribute label, respectively. Intuitively, the disentanglement loss can reduce the mutual information between attribute representations and object representations. Note that we directly optimize the similarities to make the model cannot predict the attribute label agt with the object representation vo , and vice versa. An alternative option is to use a negative cross-entropy classification loss, which will lead to the value of the loss becoming negative infinity quickly.

We further introduce a classification loss over all composed unseen compositions, to enforce the composed representations to be discriminative. For an unseen composition ui, the loss is calculated as

where, |C s,u | is all composed unseen compositions, and T c,u is the textual representations for these unseen compositions obtained via the textual encoder. Intuitively, we encourage the composed feature to be close to the textual presentations of unseen compositions at the semantic level.

Outlier exposure. To encourage the model to identify unseen compositions in testing, we introduce a loss as

where Puni is a uniform distribution over all seen compositions, and P u i is the predicted distribution of the composition classification module. CE() is the cross-entropy function. f() is to compute cosine similarities between a composed representation vu i and all seen compositions T c . Loe prevents composition classification from outputting confident predictions for OOD samples. Thus unseen compositions can be detected in testing via threshold comparison.

3.4 Optimization

We sum up all the above-mentioned losses to compute the overall loss to supervise the model learning. The overall loss is calculated as

where α1, α2, and α3 are hyper-parameters. We use a twostage training strategy for the proposed method. In the first stage, all the losses are used to train a outlier detector. In the second stage, we use only the classification loss to obtain the classifier. The model architecture is the same for two stages.

4 Experiments

4.1 Experimental Settings

Datasets. We conduct experiments on three widely used datasets, UT-Zappos [Yu and Grauman, 2014], MIT-States [Isola et al., 2015], and C-GQA [Naeem et al., 2021]. UTZappos is a synthetic fine-grained dataset consisting of 116 kinds of shoe classes composed of 16 attributes (e.g., rubber) and 12 objects (e.g. sandal). The dataset is split into 83 seen and 15/18 unseen compositions for training and validation/testing. MIT-States consists of 53, 753 crawled web images labeled with 1962 attribute-object. The dataset contains 1, 262 seen and 300/400 unseen compositions for training and validation/testing, respectively. C-GQA contains over 9, 000 common compositions and is split into 5, 592 seen and 1, 040/923 unseen compositions for training and validation/testing, respectively. Note that in validation/testing set of each dataset, the number of appeared seen compositions is equal to that of unseen compositions.

Metrics. We report the standard metrics of CZSL evaluation protocol in both closed-world and open-world settings, including the best seen accuracy (S), the best unseen accuracy (U), the best harmonic mean (HM) between the seen and unseen accuracy, and the area under the curve (AUC) of unseen versus seen accuracy. In the standard evaluation, to counteract biases of models for seen classes, the calibration bias, a scalar, is added to scores of unseen classes [Purushwalkam et al., 2019]. By varying the value of the calibration bias, the best seen/unseen accuracy, the best HM, and the AUC can be computed. Specifically, the AUC is the area under the curve of obtained seen accuracies and unseen accuracies. Considering that the best seen/unseen accuracy, and the best HM is obtained with a specific calibration bias, the AUC is thus able to better describe the overall performance of a model. In the open-world setting, the GloVe [Pennington et al., 2014] is used to filter out infeasible compositions.

Implementation details. We build our method with two backbones, the ResNet [He et al., 2016] and CLIP [Radford et al., 2021]. For the CLIP architecture, ViT-L/14 is used as the previous work [Lu et al., 2023]. For the ResNet backbone, we use a frozen ResNet-18 to obtain visual representations as [Yang et al., 2022]. We report the results of our method with both backbones on three datasets under two settings (Sec. 4.2). For other analyses, we use the CLIP backbone because the performance with it is superior. In outlier composing, we filter infeasible pairs by removing pairs with a Glove score lower than 0.5. In the first stage, we perform the grid search with the validation set to find the threshold to identify unseen compositions. In the second stage, we perform the grid search on the validation set by varying λu in {0.0,0.1,0.2,...1.0} and λs in {0.0,0.1,0.2,...1.0}. For the CLIP backbone, the training epochs for each dataset as 5/15 for the two stages, respectively. In the first stage, the hyper-parameters α1, α2, and α3 are set as (0.1, 0.1, 5.0) for the UT-Zappos, (0.01, 0.01, 1.0) for MIT-State, and (0.1, 0.5, 1.0) for C-GQA, respectively. For the ResNet backbone, the training epochs for each dataset as 50/100 for the two stages, respectively. The hyperparameters α1, α2, and α3 are set as (5.0, 0.1, 5.0) for the UT-Zappos, (5.0, 0.1, 1.0) for MIT-States, and (0.1, 1.0, 1.0) for C-GQA, respectively.

4.2 Main Results

We compare our method with various state-of-the-art methods, including both methods without CLIP and CLIP-based methods. All methods use a frozen visual feature extractor. Results of all methods on the test split of MIT-States [Isola et al., 2015], UT-Zappos [Yu and Grauman, 2014], and C-GQA [Naeem et al., 2021] under the standard closed-world setting are listed in Tab. 1. We observe that our method outperforms all methods in terms of AUC, which comprehensively evaluate the performance of CZSL models. The main reason is that our method can identify unseen compositions and separately classify seen and unseen compositions to alleviate the influence of distribution shift. Thus our method can achieve satisfactory performance for seen compositions and unseen compositions in testing with different values of the calibration bias. Note that we didn't compare with Troika [Huang et al., 2024], CAILA [Zheng et al., 2023], and RAPR [Jing et al., 2024], because they use adapter or LoRA [Hu et al., 2021] module in the transformer-based image encoder.

Tab. 2 shows the results on three datasets in the openworld setting. The proposed method outperforms all other methods, which demonstrates the effectiveness of our method for open-world compositional zero-shot learning. We observe that the performance gap between other methods and our method in the open-world setting is larger than that in the closed-world setting. A possible reason is in feature composing, we compose all possible compositions based on training images, which aligns with the open-world setting, where all possible compositions should be considered. Note that we use identical model weights, threshold, and fusion weights λs and λu for the two settings.

4.3 Ablation Studies

To study the effectiveness of several key designs of our method, we evaluate different variants of our model on the UT-Zappos in the closed-world setting with CLIP.

Effect of identifying images with unseen compositions. We first investigate the effectiveness of identifying unseen compositions in testing and separately classifying images with unseen compositions and seen compositions. In Tab. 3, we provide the results of our baseline model with only the two classification modules. The AUC of the model is much lower than our full model. The results of the composition classification module and the primitive classification module of the baseline model are also reported. Formally, we set both λu and λs as 0 to obtain the model "Composition", and set them as 1 to obtain the model "Primitive". Both models are inferior to the baseline models. We can also obtain an oracle model by using the ground truth binary label for whether an image contains a seen composition. We observe that the model significantly outperforms the full model in terms of AUC, which shows the two classification modules are complementary to each other and the idea of detecting images of unseen pairs is promising. Note that the best seen/unseen accuracies of the Oracle model don't always outperform other models. The reason is that it also searches fusion weights to softly combine the classification modules, rather than directly use a specific module for seen or unseen compositions.

Effect of each component. Then we study the influence of composing unseen attribute-object compositions as outliers. We ablate the outlier composing to obtain the model "w/o composing", which also uses the MSP to identify images with unseen compositions. As shown in Tab. 3, the model performs slightly better than the baseline model, but is much worse than our full model, which demonstrates that the composed outliers bring substantial improvements. We further ablate the outlier exposure loss, the disentanglement loss, and the discriminability loss, to obtain three models, respectively. The comparisons demonstrate the three losses are also beneficial on both datasets.

Effect of composing primitive representations via addition. In our implementation, we add two primitive representations to obtain representation for an unseen composition. To investigate the effect of this design, we use subtraction and multiplication to obtain two models ("sub" and "mul"), respectively. We also concatenate the two representations and use a trainable linear mapping to project it into a single representation, to obtain the model "cat". We observe that they are inferior to our full model, which demonstrates the simplicity and effectiveness of using the addition. A possible reason is that directly adding two representations can effectively maintain their information, and does not introduce any trainable parameters, which may affect the model learning.

Effect of using composed representations as outliers. To further study whether using composed representations as outliers is really beneficial for out-of-distribution detection, we conduct experiments on the UT-Zappos by using real images from the ImageNet dataset [Deng et al., 2009] and the images generated by the stable diffusion [Rombach et al., 2022]. The results are shown Table 3. Specifically, we randomly select 22, 998 images in the ImageNet or use the stable diffusion to generate over 2, 2998 images for the unseen compositions of the UT-Zappos, and require the model to output uniform distribution for these images. The obtained models ("Real images" and "Generated images") underperform our full model. The main reason is that by using composed features, we can obtain features of unseen pairs to better identify unseen compositions. These results demonstrate that adding primitive representations of training images to compose representations of unseen compositions is simple but effective.

4.4 Runtime Consideration

In this part, we analyze the computation time of the grid search. Table 4 shows the time for training one epoch, finding the best threshold T, finding the best fusion weights λs, λu, and a simplified grid search (SGS) to find the fusion weights, on all datasets. For the SGS, we only perform the grid search within the full range after the first epoch, and then perform the grid search in a small neighborhood around the fusion weights found in the last epoch. For example, if the found λu, λs are 0.3 and 0.7, the range of the next grid search should be [0.2, 0.3, 0.4] and [0.6, 0.7, 0.8], respectively. It is shown that finding the threshold takes almost no time while finding the fusion weights indeed takes some time. For the MITStates and C-GQA, finding the fusion weights with the SGS can save a lot of time. Besides, using the SGS leads to the same results with using vanilla grid search, because the fusion weights found of different epochs remain stable. Thus, the extra computations of the grid search are acceptable.

4.5 Qualitative Results

We visualize the feature distributions of our method with CLIP on the CGQA dataset in Fig. 3 to demonstrate the effectiveness of composing unseen compositions as outliers. Specifically, we select 10 most frequent seen/unseen compositions on the train/test set, respectively, randomly select 4 images for each composition, and visualize their image representations via the t-SNE tool [Van der Maaten and Hinton, 2008]. We also visualize 40 composed representations containing the 4 unseen compositions, by using primitive representations of training images. We observe that composed representations for unseen compositions are close to representations of testing images with unseen compositions, and thus can help our model to identify images with unseen compositions in testing. Note that we only expect composed representations to be beneficial for OOD detection, rather than matching the real distribution of unseen compositions.

5 Conclusion and FutureWork

In this work, we have presented COMO, a novel compositional zero-shot learning method. COMO composes representations of unseen compositions as outliers to identify unseen compositions. By combining composition/primitive classification modules, the method is able to separately recognize images with seen/unseen compositions. Extensive experiments show the effectiveness of our method.

In our implementation, the threshold and fusion weights are found via grid search on a validation set, which introduces extra computations. Besides, the fusion weights are the same for all samples with seen/unseen compositions. In the future, we will use several neural modules to predict sample-specific scalars based on each sample, and use reinforcement learning to enable end-to-end training.