Leveraging MLLM Embeddings and Attribute Smoothing for Compositional Zero-Shot Learning

1 Introduction

As for the study of compositional generalization ability inherent in human beings, compositional zero-shot learning

∗Corresponding authors

(CZSL) [Misra et al., 2017; Nagarajan and Grauman, 2018; Purushwalkam et al., 2019] is proposed to enable machines to recognize unseen compositions by leveraging knowledge of attributes and objects (i.e., primitives) learned from seen compositions. Specifically, in the training phase, the models are provided with images of seen compositions (e.g., ripe orange and peeled apple). During the testing phase, given an image that depicts a novel composition (e.g., peeled orange), models are assigned to correctly recognize it [Zhang et al., 2022].

Prior works [Misra et al., 2017; Nan et al., 2019; Naeem et al., 2021] focus on mapping the visual features and the word embeddings of compositions into a joint space. These methods have poor generalization capabilities on unseen compositions due to the recombination of primitives. Therefore, recent studies [Saini et al., 2022; Wang et al., 2023; Li et al., 2024; Zhang et al., 2024] consider visual disentanglement. Among them, some prominent works [Hao et al., 2023] deploy a triplet of images to disentangle visual features: a given image and two supplementary images, each sharing either the same attribute or object as the given image. The triplet of images is treated as two image pairs for subsequent analysis. These approaches aim to disentangle attribute and object by extracting the shared and exclusive features of the image pair, as well as aligning them with word embeddings (e.g., GloVe [Pennington et al., 2014]), as shown in Figure 1. Although these pioneering research studies have made great progress, they exhibit three limitations:

L1: Disentanglement is impeded due to the influence of the background and the intricate entanglement between attributes and objects in the same parts of images. On the one hand, models tend to extract the background features unique to one image in the pair as the disentangled exclusive features. On the other hand, some existing methods [Ruis et al., 2021; Saini et al., 2022] compute the similarity of the image pair for disentanglement at the spatial level. However, this paradigm is limited by the frequent entanglement of attributes and objects within the same image regions.

L2: Existing word embeddings lack the depth needed to capture complex multimodal semantic information. To begin with, word embeddings (e.g., GloVe [Pennington et al., 2014]) are grounded in word frequency and contextual co-occurrence, overlooking the high-level semantic nuances [Sarzynska-Wawer et al., 2021]. Moreover, the process of aligning visual features with word embeddings can be viewed as a form of cross-modal matching; however, these word embeddings are trained only in a single text modality, failing to capture multimodal information between images and texts.

L3: Existing methods display excessive confidence in seen compositions, impairing their ability to generalize towards novel compositions. Specifically, due to the one-hot label used during training, these approaches are limited by learning only one disentangled attribute and object, neglecting the fact that objects naturally exhibit multiple attributes [Xu et al., 2024]. Consequently, models exhibit overconfidence in the disentangled ground-truth attribute, treating other attributes that can describe the object as negative ones, which results in the diminished generalization to unseen compositions.

Being aware of these, we propose a novel framework named multimodal large language model (MLLM) embeddings and atTRibute smoothIng guiDEd diseNTanglement (TRIDENT), which consists of three major modules: visual feature extraction, attribute-object disentanglement, and feature alignment. The first module leverages feature adaptive aggregation (FAA) modules to mitigate the impact of background noise, and exploits learnable condition masks for multi-granularity feature learning at the dimensional level to improve subsequent disentanglement. The second module aims at leveraging shared and exclusive weights of image pairs to disentangle attributes and objects under the paradigm that apart from the shared features of the image pair, each image has its own exclusive features. The third module is intended to align the visual features of compositions and disentangled primitives with the last hidden states of MLLM (i.e., MLLM embeddings). This is inspired by prior works [Wang and Kuo, 2020; Muennighoff, 2022; Muennighoff et al., 2024], which reveal that the last hidden states of (M)LLM exhibit powerful representation capabilities in embedding tasks (e.g., retrieval and classification). Moreover, to tackle the issue that the overconfidence of the models regarding the ground-truth attribute hinders them from generalizing to unseen compositions, we exploit the large language model (LLM) to generate auxiliary attributes for compositions and perform label smoothing for attributes (i.e., attribute smoothing).

In summary, the contributions of our work are three-fold:

1. We propose novel feature adaptive aggregation modules to reduce the impact of background, and utilize learnable condition masks to capture multi-granularity features at

the dimensional level for disentanglement in CZSL.

1. We employ both LLM and MLLM to guide attributeobject disentanglement by generating auxiliary attributes and representing primitive words for CZSL, respectively.
2. Extensive experiments conducted on three challenging datasets (MIT-States [Isola et al., 2015], C-GQA [Naeem et al., 2021], and VAW-CZSL [Saini et al., 2022]) show that TRIDENT has achieved state-of-the-art performance.

2 RelatedWork

Compositional zero-shot learning (CZSL). Prior works in CZSL can be broadly divided into two main streams. One main stream is learning visual representations and textual labels of compositions in a joint space. SymNet [Li et al., 2020] aims to learn the symmetry property in compositions. Co-CGE [Mancini et al., 2022] leverages a graph convolutional neural network to learn composition representations. The other main stream aims at disentangling visual representations of primitives to reduce composition learning to primitive learning. SCEN [Li et al., 2022] leverages contrastive loss to excavate discriminative prototypes of primitives. CANet [Wang et al., 2023] learns the conditional attribute conditioned on the recognized object and the input image. More recent works [Nayak et al., 2023; Lu et al., 2023; Huang et al., 2024] leverage the encyclopedic knowledge of pre-trained vision-language models (VLM) like CLIP [Radford et al., 2021] to encode and align images and texts.

Large language model (LLM). LLMs have realized significant advancements thanks to the scaling up of training data and the increase in the number of parameters. Early models, such as GPT-2 [Radford et al., 2019], initially exhibit strong capabilities in understanding and generating humanlike language. Subsequently, GPT-3 [Brown et al., 2020] and LLaMA [Touvron et al., 2023] demonstrate great breakthroughs across numerous language benchmarks.

Expanding on LLMs, multimodal large language models (MLLM) incorporate a visual encoder for vision-language tasks. Flamingo [Alayrac et al., 2022] integrates Vision Transformer (ViT) [Dosovitskiy et al., 2021] and LLM by gated cross-attention. LLaVA [Liu et al., 2024b] and LLaVA v1.5 [Liu et al., 2024a] introduce visual instruction tuning to enhance the instruction following capability. The visual understanding part of LLaVA v1.5 consists of a ViT and an MLP cross-modal connector. We choose LLaVA v1.5 as our foundational MLLM for its state-of-the-art performance.

Recently, exploring the powerful embedding capabilities of (M)LLM to handle representation tasks (e.g., retrieval) has emerged as a prominent research domain. SGPT [Muennighoff, 2022] exploits the last hidden states of LLM for the input token sequence or a special learnable token to derive representational embeddings. Subsequently, GritLM [Muennighoff et al., 2024] applies mean pooling over the last hidden states of LLM to produce the textual embeddings.

3 Approach

3.1 Task Formulation

Compositional zero-shot learning (CZSL) aims at learning a model that can recognize unseen compositions of attributes and objects that are learned from seen compositions. Given an attribute set A and an object set O, the attributes and objects are composed to form a composition set C = A × O. The composition set C is divided into two disjoint sets: the seen composition set Cs and the unseen composition set Cu, i.e., Cs ∩ Cu = ∅. The model learns from a seen training set Dtr = {(xs, cs)}, where xs is an image of the seen composition label cs ∈ Cs. Following the Generalized CZSL [Purushwalkam et al., 2019], the model is evaluated on a predefined test set Dte = {(xte, cte)}, where xte is a test image from the predefined composition subset Cte of C, i.e., Cte ⊆ C, and cte ∈ Cte is the label of xte.

3.2 TRIDENT

As the major novelty, we propose a novel framework named MLLM embeddings and attribute smoothing guided disentanglement (TRIDENT), as shown in Figure 2. It consists of three modules: (1) visual feature extraction, (2) attributeobject disentanglement, and (3) feature alignment.

Visual Feature Extraction

As shown in Figure 2, we denote a given image with the attribute-object composition label (e.g. ripe apple) as the main image xm, and randomly sample an image with the same attribute xa (i.e., ripe orange) and an image sharing the same object xo (i.e., peeled apple) to form a triplet image set. For convenience of expression, we simply use ximg (where img ∈ {m, a, o}) to collectively denote the images as they are processed using the same module.

Visual backbone. As mentioned before, since LLaVA v1.5 is used as our fundamental MLLM, we directly leverage its visual encoder (i.e., ViT) and cross-modal connector (CMC) to extract visual features. Specifically, the image ximg is partitioned into n patch tokens, which are subsequently put into ViT along with the [CLS] token. Afterward, the output of patch tokens before the last layer of ViT is fed into the CMC module, as implemented in LLaVA v1.5. To align the dimensions of the patch tokens output by CMC and the [CLS] token produced by ViT, the patch tokens output by CMC is input into a linear layer. Consequently, we obtain one feature vector of [CLS] token f img cls ∈ Rd and a patch feature matrix of n patch tokens F img patch ∈ Rn×d .

Local features extraction. Intuitively, the composition (e.g., ripe apple) only occupies a few parts of the image. Since each patch token corresponds to one local region of the image, to filter out background noise and focus on related regions, we deploy p feature adaptive aggregation (FAA) modules to derive p relevant local features of ximg , where each FAA module is formulated as follows:

where Conv(·) represents the 1 × 1 convolution layer, agg ∈ Rn is the weight vector, and the k-th element of agg is the weight for k-th patch feature. ⊗ represents matrix product, and v ∈ Rd is the local feature obtained by an FAA module. We vertically concatenate the local features produced by p FAA modules to obtain the local feature matrix F img l ∈ Rp×d .

Global features extraction. Normally, the [CLS] token output by ViT is regarded as containing various global information of the image, which highly entangles both attribute and object together [Hao et al., 2023]. To disperse multigranularity global information into different representations at the dimensional level, q learnable condition masks are applied to f img cls to obtain q different global representations. Each global representation is computed as:

where u ∈ Rd denotes the global representation, c ∈ Rd refers to the learnable condition mask and ⊙ is the elementwise multiplication. We vertically concatenate q global representations to derive the global feature matrix F img g ∈ Rq×d .

Features concatenation. Finally, F img l and F img g are vertically concatenated to form the visual features of ximg , i.e., F img = [F img l , F img g ] ∈ Rh×d (where h = p + q), which is used for the following attribute-object disentanglement.

Orthogonal regularization. Different features should capture distinct and complementary aspects of the image. Therefore, we introduce the orthogonal regularization, i.e.:

where Ii denotes the identity matrix, and || · ||Fro refers to the Frobenius norm of the matrix.

Image embedder. Inspired by [Nagarajan and Grauman, 2018], for the input image ximg , we first use the average pooling Avg(·) on F img g and F img l , respectively, and horizontally concatenate them by Cat(·, ·) to aggregate both global and local features of ximg . Then the concatenated feature passes through a linear layer Lincomp(·) to derive the final visual feature f img comp that represents the composition. This module is formulated as follows:

Attribute-Object Disentanglement

As mentioned before, one of the key challenges for CZSL is to disentangle attributes and objects from visual features. To overcome the challenge, we propose a novel weighted disentanglement strategy, as illustrated in Figure 2. For brevity, the image pair (xm, xa ) from the triplet image set is taken as an example to elaborate on this strategy, while another image pair (xm, xo ) is processed in the same manner.

Weights computation. The features of xm and xa (i.e., F m and F a ) are vertically concatenated and fed into two MLP modules to derive their respective weights of shared attribute features relative to each other. Subsequently, we utilize them to compute the weights of their own exclusive object features as follows:

where wm2a attr , wa2m attr ∈ Rh represent the weights of the shared attribute features of xm relative to xa and xa relative to xm, respectively. wm2a obj and wa2m obj denote the weights of exclusive object features corresponding to xm and xa , respectively, which are derived by "1 - shared weights" paradigm as beyond the shared features of the image pair are the exclusive features of each image. Taking wm2a attr as an example, its k-th element refers to the shared attribute proportion of the k-th feature of xm relative to xa .

Disentangled features acquisition. We multiply the elements of each weight vector by the corresponding features and then calculate the average. The following takes the process of obtaining the shared attribute features of image xm relative to xa as an example:

where F mi,: denotes the i-th row of F m, i.e., the i-th feature of xm. wm2a attr i refers to the i-th element of wm2a attr .

For the image pair of xm and xa , four parts are obtained: the shared attribute features of xm relative to xa and xa relative to xm, as well as two exclusive object features of the two images, respectively. These four features are marked as f e pri , where e ∈ {m2a, a2m} and pri ∈ {attr, obj}. Then the shared attribute feature of xa and xm without relativity is obtained by an MLP layer, which is less dependent on the different objects of the two images. The process is as follows:

Similarly, we disentangle the attribute and object for the image pair (xm, xo ) and obtain the same visual features as (xm, xa ): f e pri , where e ∈ {m2o, o2m} and pri ∈ {obj, attr}, and the feature without relativity f mo obj .

Feature Alignment

Inspired by GritLM [Muennighoff et al., 2024] that leverages the last hidden states of LLMs as the representational embeddings, we consider the last hidden states of LLaVA v1.5 [Liu et al., 2024a] as our MLLM embeddings for primitive words. Moreover, to tackle the problem that the ineffective overconfidence exhibited by the model in terms of the ground-truth attribute hinders it from generalizing to unseen compositions, we employ GPT-3.5 [OpenAI, 2023] to generate auxiliary attributes for compositions and perform label smoothing during attribute alignment. The auxiliary attributes and MLLM embeddings are obtained offline before training TRIDENT.

Auxiliary attributes generation by LLM. Since only textual attributes need to be generated, the LLMGPT-3.5 [OpenAI, 2023], instead of an MLLM, is leveraged to generate t auxiliary attributes for each composition. Specifically, the following prompt is input to LLM: 'Please give me t adjectives that can describe the visual feature of a photo of a/an ... well.', where the attribute-object composition (e.g., peeled apple) is filled in '...'. Subsequently, the generated auxiliary attribute words form a set Aa. Therefore, the set of all words Y is obtained, including attributes, objects and auxiliary attributes:

MLLM embeddings acquisition. Each word y ∈ Y is fed into LLaVA v1.5 to get the last hidden states, i.e., LLaVAlhs(·). Specifically, y is tokenized into multiple subwords and passed through the MLLM; the final-layer output is averaged as the MLLM embedding of the word. Subsequently, it is passed through an MLP layer to obtain the embedding Eword(·) of the aligned dimension with visual features. And for a composed pair c of attribute a and object o, i.e., c = (a, o), we horizontally concatenate the MLLM embeddings for a and o and feed them into a linear layer Linco(·) to get the composed pair embedding Eco(·). The process is formulated as follows:

Word expanding. Previous works compute cosine similarities of disentangled features and word embeddings and apply cross-entropy only within the respective domains of attributes or objects, which results in the disentangled attributes and objects still retaining the information of each other. To address the problem, we propose a novel word expanding strategy, which computes cosine similarities of visual features and the embeddings of all words, including attributes and objects, and treats all words except the ground-truth word as the negative labels in subsequent cross-entropy.

Alignment by cross-entropy. Similar to [Mancini et al., 2021], we use cross-entropy to align the visual features and word embeddings. Assume that f is the visual feature and Eword(wd) is the word embedding for the word wd ∈ Y. The classifier logit from f to Eword(wd) is defined as follows:

where δ is the temperature variable, and cos(·, ·) denotes the cosine similarity function. Thus cross-entropy with/without label smoothing can be uniformly formulated as follows:

where α denotes the smoothing factor, t refers to the number of auxiliary labels, and z ∈ [0, 1] represents the target value of one-hot label or smoothing label. For cross-entropy without label smoothing, i.e., with one-hot label Hoh, α is set to 0. The cross-entropy with label smoothing is denoted as Hls.

For the disentangled attribute features of one image relative to each other, since a single object exhibits multiple attributes, we exploit attribute smoothing with auxiliary attributes to undermine the overconfidence in the ground-truth attribute and learn more related attributes. For the shared attribute features without relativity, the one-hot label is used to learn a pure attribute concept that is less conditioned on objects. The loss for disentangled attributes is as follows:

Concerning the disentangled object features, we use crossentropy with one-hot label to learn the object concept and the loss is as follows:

With respect to the visual feature of the composition from image embedder, we calculate the cosine similarities between the visual embedding and the composed pair embeddings of seen composition labels. The cross-entropy loss for the composition is as follows:

3.3 Training and Inference

During the training phase, the overall loss is as follows:

where γortho, γcomp, and γpri are weighting factors to balance the influence of different losses.

For inference, given an image from the test set, the cosine similarities of its visual feature obtained by image embedder and the composed pair embeddings of all candidate compositions are computed. The composition with the highest similarity is predicted by the model. Note that although the disentanglement branch is not used for inference, it still influences the formation of the composition feature space through the visual feature extraction module described in Section 3.2.

4 Experiment

4.1 Experiment Setup

Datasets. We evaluate our model on three challenging CZSL datasets: MIT-states [Isola et al., 2015], C-GQA [Naeem et al., 2021], and VAW-CZSL [Saini et al., 2022]. The common data splits are presented in Table 1.

Metrics. Our method is evaluated on seen and unseen compositions separately. Following the common generalized CZSL setting [Purushwalkam et al., 2019], a calibration bias trades off between the accuracies of seen and unseen compositions. We calculate the Area Under the Curve (AUC) using seen and unseen accuracies at different biases. The best Seen and Unseen accuracies of the curve are also reported. In addition, we calculate the Harmonic Mean of seen and unseen accuracies at different biases and report the best one (HM).

Implementation details. We use the visual encoder of LLaVA v1.5, ViT-Large-Patch14-336px as our frozen visual backbone. TRIDENT and all baseline models are trained with the batch size of 128 for 50 epochs under the PyTorch framework [Paszke et al., 2019]. The number of global features is set to 6, 2, and 4 for the three datasets, respectively, and the number of local features is twice that of the global features. The label smoothing factor is set to 0.09, 0.03, and 0.03 for the three datasets, respectively. The number of auxiliary attributes generated for each composition is set to 3. We train TRIDENT by Adam optimizer with the weight decay of 5e-5, learning rates of 1.5e-6 for word embedding and 2e4 for other modules. We decay the learning rate by a factor of 0.1 at epoch 30 and 40. The temperature variable of cosine similarity δ is set to 0.05. For weighting coefficients γortho, γcomp, and γpri, we set them to 0.1, 1, 0.25, respectively.

Baselines. We compare our TRIDENT with recent and prominent approaches in CZSL: SymNet [Li et al., 2020], CompCos [Mancini et al., 2021], Co-CGE [Mancini et al., 2022], SCEN [Li et al., 2022], OADis [Saini et al., 2022], INV [Zhang et al., 2022], CANet [Wang et al., 2023], and ProCC [Huo et al., 2024]. We replace their visual backbone with ViT-Large-Patch14-336px and retrain all models with the same number of epochs for the sake of fairness. In addition, although comparing TRIDENT with CLIP-based methods, which rely on the dual-tower architecture, is very unfair due to inadvertent exposure to unseen compositions, we still choose CLIP [Radford et al., 2021] and CoOp [Zhou et al., 2022] as baselines for their strong zero-shot abilities.

4.2 Results and Discussion

In this section, we compare TRIDENT with state-of-the-art methods. As shown in Table 2, our model surpasses other models by a substantial margin in general. TRIDENT boosts AUC from 13.6%, 5.7%, and 6.7% of the previous stateof-the-art method CANet to the new state-of-the-art performance of 14.2%, 8.0%, and 8.3% with 0.6%, 2.3%, and 1.6% improvement on three datasets, respectively. In addition, TRIDENT achieves 30.9%, 22.6%, 23.4% on the metrics of HM, providing 1.1%, 3.7%, and 2.4% improvement on CANet. For MIT-States, our model achieves competitive performance, despite considerable label noise [Atzmon et al., 2020]. The largest improvement is observed on the Unseen metric, indicating that attribute smoothing helps enhance the generalization ability of the model. We also observe that TRIDENT performs significantly better than CANet regarding all metrics on two more challenging and low-noise datasets C-GQA and VAW-CZSL, indicating the efficacy of our approach. These improvements arise from the utilization of MLLM embeddings and attribute smooth-

(a) image-to-text retrieval.

(b) text-to-image retrieval (successful cases).

(c) text-to-image retrieval (failure cases).

Figure 3: Qualitative analysis. (a) Top-5 image-to-text retrieval. The first two rows display successful cases, while the last row presents failure cases. For each image, the top row shows the ground-truth, followed by five rows of top-5 predictions. (b) Successful cases and (c) failure cases of top-5 text-to-image retrieval. In all cases, the successful and failure results are tagged in green and red, respectively.

ing, which enhance attribute-object disentanglement and consequently facilitate the recognition of unseen compositions while maintaining performance on seen compositions.

4.3 Ablation Study

We ablate the components of TRIDENT on MIT-States to evaluate their contributions. From the ablation results in Table 3, we gain the following observations.

1. TRIDENT outperforms the models without using patch and [CLS] features, indicating that both patch and [CLS] features are crucial, with patch features contributing more.
2. Both w/o FAAs and w/o condition masks models perform worse than TRIDENT, which validates the importance of filtering out the background noise and extracting the multigranularity features, respectively.
3. TRIDENT surpasses w/o word expanding model and w/o attribute smoothing model on the Unseen metric, yet falls short of them on the Seen metric. The difference between our model and the w/o word expanding model stems from its more thorough disentanglement, which enhances the recognition of unseen compositions while weakening the identification of seen ones. The disparity between our model and the w/o attribute smoothing model arises from attribute smoothing, which diminishes the overconfidence of the model in seen compositions, facilitating its generalization to unseen ones. However, the improvement of our model over these two models on AUC and HM indicates the effectiveness of the word expanding and label smoothing strategy.
4. TRIDENT outperforms w/o Lattr + Lobj model on all metrics, confirming that the attribute-object disentanglement module is highly advantageous for generalization from seen compositions to unseen compositions.
5. w/o Lortho model is inferior to TRIDENT, which suggests that the designed orthogonal regularization is helpful to guarantee that different features extract different visual information.

Impact of word embeddings. Our work leverages the last hidden states of LLaVA v1.5 as word embeddings, while Word2Vec [Mikolov, 2013] and FastText [Bojanowski et al., 2017] are the common word embeddings used for MIT-States in previous works. In Table 4, based on three models: SCEN [Li et al., 2022], CANet [Wang et al., 2023] and TRIDENT, we compare the performance of using the last hidden states of LLaVA v1.5 (LLaV Alhs) and the sum of Word2Vec and FastText (ft+w2v), respectively. The results indicate that the last hidden states of MLLM capture more complex multimodal semantic information than ordinary word embeddings.

4.4 Qualitative Analysis

In this section, we use TRIDENT to conduct both imageto-text retrieval and text-to-image retrieval experiments on the three datasets. We first consider image-to-text retrieval, shown in Figure 3a. For successful cases, such as the image labeled burnt house, we notice that the top four predictions can describe logs burning on a fire in the image. In terms of the image labeled green leaf, another successful case, all predicted attributes can describe the leaf, which is due to attribute smoothing learning more attributes for an object. For the failure cases, such as the image labeled colorful pillow, the prediction of orange pillow can also describe the image.

We then consider text-to-image retrieval. Successful cases are shown in Figure 3b, while failure cases are shown in Figure 3c. We observe that all retrieved images of peeled orange are correct. However, the retrieved images of green grapes are all wrong. This is due to the fact that the training images of green grapes are almost entirely filled with a single grape, which makes it difficult for the model to capture the contour features of a bunch of green grapes.

5 Conclusion

In this work, we propose a novel framework termed TRIDENT to address the challenging CZSL task. First, we leverage feature adaptive aggregation modules to mitigate the impact of background, and utilize learnable condition masks to capture multi-granularity features for attribute-object disentanglement. In addition, we exploit the last hidden states of MLLM to replace ordinary word embeddings, as they can capture complex multimodal semantic information. Moreover, we leverage LLM to generate auxiliary attributes and perform attribute smoothing to diminish overconfidence of the model in seen compositions, which enables it to generalize to unseen compositions more effectively. Extensive experiments conducted on three challenging datasets demonstrate the effectiveness of our method.

Acknowledgments

Our work was supported by the Beijing Natural Science Foundation (No. 4242046).