Exploring Transferable Homogenous Groups for Compositional Zero-Shot Learning

1 Introduction

Deep neural networks have achieved impressive results in object recognition tasks, yet their capabilities remain underexplored in recognizing abstract concepts and performing compositional reasoning. Humans, on the other hand, possess a remarkable ability to decompose and reorganize underlying visual knowledge to recognize new concepts. For instance, after recognizing a red car and a green truck, one can easily identify a green car and a red truck, even without having seen these specific visual samples before. To exploit the compositional recognition potential of models, Compositional ZeroShot Learning (CZSL) [Chen and Grauman, 2014][Misra et al., 2017] has been proposed, aiming to generalize to unseen compositions by learning from seen state-object sample pairs.

∗Corresponding author: Jingcai Guo.

Conditional dependency [Wei et al., 2019][Ge et al., 2022] is one of the most intractable problems in CZSL, which reflects the intrinsic connection between states and objects, often referred to as domain bias. The dependency results in the same state (or object) exhibiting vastly different visual features when paired with different objects (or states), making it difficult to generalize semantic information learned from seen compositions to unseen ones. To tackle this problem, existing approaches typically follow two fundamental paradigms, the all-to-one and one-to-one paradigms. The former focuses on learning a versatile, domain-invariant prototype, whereas the latter explicitly models the dependencies between objects and states to learn instance-specific representations.

Despite the promising results, it remains a challenge to strike a balance between transferability and discriminability [Chen et al., 2019]. The all-to-one paradigm overly emphasizes transferability, aiming for a pure and uniform central representation. This focus forces the model to sacrifice instance-specific semantic information. In contrast, the one-to-one paradigm prioritizes discriminability, leading the model to develop a bias toward the seen domain. Comparatively, humans are adept at analogizing and reasoning via hierarchical clustering to form group-specific conceptual cognition (Fig. 1 a). Such concepts maximize transferability among group members while preserving distinct and recognizable features. Thus, when recognizing an old dog, we would easily associate it with a bear or a cat to reduce the complexity of inference. Fortunately, despite the lack of guidance by hierarchical labels in CZSL, we find that homogeneous semantic structures are naturally maintained in the deep feature space (Fig. 1 b-c).

Motivated by this, we propose Homogeneous Group Representation Learning (HGRL). Analogous to the idea of hierarchical clustering, we cluster categories with similar properties together and call it a homogeneous group. The objective of HGRL is to identify unseen compositions by leveraging the high transferability among homogeneous group members. Our method integrates three critical components with CLIP as the backbone. Group-Aware Visual Representation

(GAVR) aims to perceive and capture homogeneous groups and to learn group-specific representations to preserve semantic richness and completeness. To mimic the human knowledge system, we introduce the text co-occurrence probability graph to guide the hierarchical clustering process. Decoupled Group Prompt (DGP) is a simple yet effective groupbased prompt representation. Traditional prompts provide a centralized prototype for each category. However, finding a common prototype representation is extremely challenging. To this end, DGP equips each group with an exclusive context prompt to learn distributed prototype representations. GroupAware Pair Enhancement (GAPE), which combines state and object features into pairs to facilitate joint recognition. To further improve the robustness and generalization of the model, we utilize the compatibility of homogeneous group members to enhance the state (object) features to obtain diverse pairs. We conduct experiments on three mainstream datasets, and the results demonstrate that our method achieves state-of-theart performance.

Our contributions are summarized below:

• We analyze that the limitation of existing paradigms dealing with conditional dependency lies in the zerosum game between transferability and discriminability. Inspired by the hierarchical clustering mechanism, we provide a new view to tackle this problem.
• We present a new framework, named HGRL, which integrates three essential components to enhance both visual and prompt representations. HGRL explores latent homogeneous groups and maintains group-specific semantic information with multi-expert networks and distributed prompts.
• We conduct experiments on three major benchmark datasets, and the results show that the proposed method achieves state-of-the-art performance.

2 RelatedWork

2.1 Compositional Zero-Shot Learning

The goal of Compositional Zero-Shot Learning (CZSL) is to recognize unseen state-object pairs by learning and reorganizing seen compositions. The primary challenge is the distributional bias between seen and unseen compositions, which is caused by the dependency relationship between objects and states. Some studies adopt the all-to-one paradigm, i.e., learning a versatile semantic representation for a single object or state. The dominant technique is decoupling representation, which is based on the idea that objects and states can exist and be represented independently, so that the two need to be divorced. To this end, prototype learning [Ruis et al., 2021][Lu et al., 2023] and attention-based decoupling techniques [Zhang et al., 2022][Li et al., 2023][Hao et al., 2023] have been introduced. In addition, some studies are inspired by causal representations to uncouple the potential connection between objects and states through decorrelation techniques [Atzmon et al., 2020]. Nan et al. [Nan et al., 2019] introduce contrastive learning to compress the intra-class semantic space to purify the representations. This paradigm is prone to sacrifice rich semantic features or learn pseudorelated features [Rao et al., 2024b].

Another prominent paradigm is one-to-one representation. The objective is explicitly modeling the dependency relations between states and objects. For example, CoT [Kim et al., 2023] uses object visual features as induced information to generate state visual features, while CANet [Wang et al., 2023] and Troika [Huang et al., 2024] leverage visual features to correct text embeddings. Other notable approaches include constructing conditional reconstruction networks [Nagarajan and Grauman, 2018][Li et al., 2020] and conditional classifiers [Liu et al., 2023][Huo et al., 2024]. This paradigm tends to bias the model in favor of the seen domain.

Contribution. The all-to-one and one-to-one paradigms explore solutions to the issue of conditional dependency in almost opposing forms, yet both reach an extreme of neglecting discriminability or transferability. We provide a new perspective on it-find a balance between the two thus maximizing the generalization ability of the model.

2.2 Hierarchical Clustering

Hierarchical clustering is a classic clustering idea that considers both merging and splitting, drawing inspiration from tree-like human knowledge system [Silla and Freitas, 2011]. It plays a crucial role in various structured data tasks, such as text classification [Chalkidis et al., 2020], speech classification [Dekel et al., 2004], and protein function prediction [Otero et al., 2009]. Recently, hierarchical clustering has shown promising potential in vision tasks. For example, B-CNN [Zhu and Bain, 2017] combines the inter-layer outputs of convolutional networks with tree labels, significantly improving classification efficiency. Detclipv3 [Yao et al., 2024] introduces hierarchical labels for open-vocabulary object detection, enabling the learning of multi-granularity semantic information. Additionally, some studies [Novack et al., 2023][Rao et al., 2024a] explore the potential of hierarchical clustering in zero-shot inference tasks.

Contribution. To the best of our knowledge, we are the first to apply the idea of hierarchical clustering to CZSL to solve the problem of imbalance between discriminability and transferability. Meanwhile, we introduce the text co-occurrence probability graph to cope with the problem that no hierarchical labels are available in CZSL.

3 Methodology

3.1 Preliminary

Suppose we have ns states S = {s1, s2, ..., sns } and no objects O = {o1, o2, ..., ono }, then there are ns × no compositions defined as C = {c1 = (s1, o1), c2 = (s1, o2), ...,Cns×no = (sns , ono )}, i.e., C = S × O. During the training phase, only partial compositions are available which are denoted as seen compositions C se . The goal of CZSL is to train a generalizable model thereby identifying unseen compositions C use . Note that C se ⋂ C use = ∅. Generally, |C se | + |C use | < |C| since some compositions in C do not exist in reality. So there are two settings in CZSL, closed-world CZSL where the label space is C se ⋃ C use and open-world CZSL where the label space is C. Formally, the training set is defined as T = {(xi, ci)|xi ∈ X, ci ∈ C se }, where X denotes the image space. An overview of our approach is presented in Fig. 2.

3.2 Group-Aware Visual Representation

The representations learned by paradigm all-to-one lose substantial key semantic information, while by paradigm one-toone tend to favor the seen domain. To solve this problem, we present Group-Aware Visual Representation (GAVR) to extract the state and object features, whose motivation is learning group-specific representations. The advantages is twofold: 1) Maximizing semantic transferability among group memebers; 2) Preserving semantic integrity. Since the two branches are symmetric, for ease of illustration, we next introduce the state branch only.

The immediate problem is how to extract group-specific features. Inspired by mixture-of-expert network [Jacobs et al., 1991], we introduce multiple expert sub-networks to learn the features of different groups. Given an instance (x, c) ∈ T , we obtain its semantic features via the image encoder ΦI. The feature ΦI(x) at this point are a mixture of state and object semantic information. To get its state features, we use a sub-network ΦS to map it to the state semantic space, which is denoted as ΦS(ΦI(x)). Previous methods would have used the feature to train directly, leading to a loss of group-specific information. We access a multi-expert network to extract group-specific semantic information. It consists of a route network ΦRS and ks expert sub-networks ΦSE = {ΦSE1 , ...,ΦSEks }. First, the feature ΦS(ΦI(x)) is routed through the route network to determine the belonging of the group, the confidence for expert i is denoted as:

We then assign features to the corresponding expert networks based on the route scores. To balance sparsity and smoothness, we use a top filtering mechanism, i.e., selecting K experts with the largest confidence to participate in the next process. The obtained feature can be expressed as a weighted sum of K experts:

Finally, we fuse the group-specific feature with the original feature:

where β is a learnable parameter.

How to Capture Homogenous Groups. Another problem is the lack of supervision of hierarchical labels makes the capture of homogeneous groups uncontrollable, despite the natural heterogeneity exploration capability of route network. To this end, we utilize a word embedding model, GloVe [Pennington et al., 2014], to evaluate compatibility between categories. The core idea of GloVe is to learn co-occurrence relationships between words from a large-scale corpus. We argue that the probability of co-occurrence between words largely matches the probability that they belong to the same superclass in reality, and thus can be used as a substitute for hierarchical labels. For state branch, we need to evaluate the heterogeneity of different objects with the same state. For any two object words, we compute cosine similarity to assess the co-occurrence probability of them, which is denoted as M o ij = cos(vo i , vo j ), where v denotes the word embedding extracted by GloVe.

Now the most straightforward approach is to assign hard group labels by clustering through similarity threshold filtering, which involves sensitive threshold effect and is prone to introduce noise. Inspired by graph representation learning [Scarselli et al., 2008], we encourage the model to adaptively discover homogeneous groups via local clustering. Specifically, we aggregate the feature of one sample with its nearest compatible samples. For example, for an old dog, we fuse it with old cat, old tiger, etc. to form a tiny homogeneous group. The model iteratively learns and summarizes the relationships of the tiny homogeneous groups to extend to the whole homogeneous group. Given a batch of instances {xi, ci = (si, oi)}B ∈ T , where B denotes the batch size. We can get their relation map A ∈ RB×B , and we have:

where I[·] denotes indicator function and ζ is a constant which is fixed to 0.5. After normalization for A, we can get the aggregated features which denoted as

where ⊗ denotes matrix multiplication and Λ denotes the unit diagonal matrix. Next the aggregated features are fed into route and expert networks for training.

3.3 Decoupled Group Prompt

Textual prompts are important components in visual language models (VLMs) [Radford et al., 2021]. In the classification tasks, they act as primitive representations of categories thus providing optimized targets for visual features. The common practice of designing prompts in the fixed form of a photo of [class], which has been proven to perform inferior to learnable prompts [Zhou et al., 2022]. The central idea of learnable prompts is to learn the task-specific context. To this end, previous CZSL approaches equip each branch with a separate contextual prompt, which are represented as:

where PSi ,POi ,PCi denote the state, object and pair prompts, respectively. And {ps1 , ..., psm}, {po1 , ..., pom}, {pc1 , ..., pcm} denote their contextual prompts and ws i , wo i denote the state and object words. For state and object branches, the learned prompts capture the global context yet ignore the discrepancies between heterogeneous groups. To this end, we add the corresponding prompt for each group, which are represented as:

where ks, ko represent the groups of state and object, gs , go represent the group prompts. Then these prompts are fed into the text encoder ΦT to obtain representations, which are denoted as:

Training Objectives. We adopt InfoNCE loss function to train the network. At the same time, we use the confidence scores in Eq. (1) as soft group labels for samples. For the state branch, the loss formula is:

where τ denotes the temperature coefficient. Since the object and state branches are symmetric, similarly, we can get the loss formula for the object branch as:

3.4 Group-Aware Pair Enhancement

To improve the compositional recognition ability of the model, we introduce a pair branch. It utilizes the outputs of state and object branches as inputs to learn semantic reorganization and interaction. The pair branch consists of a learnable network ΦP . We concat the state features xs and object features xo as inputs, and the outputs are features with the same dimension as ΦI (x). To enhance the performance of ΦP , we introduce group-aware feature enhancement to augment the input space. Although previous approaches have used similar enhancement techniques [Panda and Mukherjee, 2024][Jing et al., 2024], the difference is that we argue that not all enhancements are meaningful due to the heterogeneity of groups. For example, it is inappropriate to enhance an old dog with old semantics extracted from an old car. Therefore, we suggest that compatibility between instances should first be checked.

To realize this, we employ the group confidence scores in Eq. (1) to detect homogeneous groups. For one instance, the group confidence scores of its state features can be denoted as pSr (i) = [pS1 r (i), pS2 r (i), ..., pSks r (i)]. The compatibility of two instances with respect to the state can then be quantized as sim(i, j). Similar to Eq. (4), we can obtain an adjacency matrix ˆ As :

where sim(i, j) = cos(pSr (i), pSr (j)). Then the enhanced state features are formulated as:

Similarly, we can get the enhanced object features, denoted as ˆxo. Then we concat them to form the pair features, which are denoted as ¯xp = ˆxs ⊕ ˆxo, where ⊕ means concat. Then we can get the output of pair branch and pair prompt representations according to Eq. (8):

The training loss is formulated as:

3.5 Summarize

Total Training Loss. The basic loss that applies CLIP to the CZSL task is denoted as:

Then the total training loss of the propose method is denoted

Inference. We integrate the outputs of backbone, state, object, and pair branches for joint prediction. The output of backbone is represented as:

Similarly, the output of pair branch is denoted as ppair(c|xp). For output of state branch, it is denoted as:

Similarly, we can get the output of object branch, denoted as pobject(o|xo). The final prediction is:

where C tar is C se ⋃ C use for closed-world setting or C for open-world setting.

3.6 Theoretical Insights

We provide theoretical insights for our approach from the perspective of domain adaptation [Ben-David et al., 2006] since one state (object) on different objects (states) can be viewed as different domains [Zhang et al., 2022].

Definition 1 Given a seen domain, S, and an unseen domain, U. We have,

where ϵU(h), ϵS(h) is the expected risks for unseen and seen domains and h is the model. d(, ) is the distributional divergence and γ is a constant denotes the minimum of the risk sum of h over S and U.

Def. 1 reveals a loose upper bound which is determined by the distributional divergence between S and U. Let S ∗ ∈ S denotes the homogeneous group for U and we have S ♯ = S - S ∗ . Then d(S, U) could be decomposed into:

where g denotes group label. Obviously d(S ∗ , U) ≤ d(S, U) and d(S ∗ , U) ≤ d(S ♯ , U). To this end, our method first creates conditions for separating homogeneous group S ∗ by constructing a multi-expert network as well as distributed group prompts. Then, the text co-occurrence probability graph is utilized to guide the clustering of homogeneous categories thereby increasing the probability p(gu = gs∗).

4 Experiments

4.1 Experiment Settings

Datasets and Evaluation Metrics. We perform experiments on three commonly used datasets including MITStates [Isola et al., 2015], UT-Zappos [Yu and Grauman, 2014] and C-GQA [Naeem et al., 2021]. MIT-States has 115 states and 245 objects. The number of seen compositions is 1262 and unseen compositions is 400. UT-Zappos is a small footwear dataset with 16 states and 12 objects. There are 83 seen compositions for training and 18 unseen compositions for testing. C-GQA is a challenging dataset containing 413 states and 674 objects. There are 5592 seen compositions and 923 unseen combinations. We follow the evaluation protocol of previous work[Huang et al., 2024][Jing et al., 2024]. Accuracy curve is plotted by adjusting the calibration bias for the scores of unseen pairs. The area under the curve called AUC is recorded to measure model performance. Simultaneously the best seen accuracy S and the best unseen accuracy U are reported. We also report the best harmonic mean HM between seen accuracy and unseen accuracy.

Implementation Details. For a fair comparison, we adopt the pre-trained CLIP [Radford et al., 2021] ViT-L/14 model as the backbone. The MLP in GAVR is one-layer Multihead Attention and the expert is two-layer fully-connected networks. The MLP in GAPE is two-layer fully-connected networks. The learning rate is 5e-4 for UT-Zappos and 5e-5 for MIT-States and C-GQA. The batch size is 180 for UTZappos and 32 for MIT-States and C-GQA. We use Adam optimizer to train the model. The group number of state ks and object ko are set to 3 for UT-Zappos and 5 for MIT-States and C-GQA. The hyper-parameter λ is set to 1.0 for UT-Zappos and 0.1 for MIT-States and C-GQA.

be observed that removing any of the modules degrades the performance of the model, indicating the rationality of the design of the proposed method. Among them, GAVR and DGP have the greatest impact on the model's performance, indicating that they contribute greatly to the optimization of the visual and prompt representations, respectively. In addition, the results show that the introduction of the text co-occurrence probability graph (TCPG) improves the performance of the model, demonstrating that it indirectly facilitates the representational ability of GAVR, guiding it to discover and aggregate homogeneous groups.

Effect of group number ks and ko. Due to the lack of supervision by hierarchical labels, the potential number of

4.2 Main Results

For a fair comparison, we compare our methods with the state-of-the-art methods that employ the same backbone, including CLIP [Radford et al., 2021], CoOp[Zhou et al., 2022], CSP [Nayak et al., 2022], PromptCompVL [Xu et al., 2022], DFSP [Lu et al., 2023], PLID [Bao et al., 2025], Troika [Huang et al., 2024], CDS-CZSL [Li et al., 2024] and Retri-Aug [Jing et al., 2024]. The main results are presented in Table 1.

We compare the results under both closed-world and openworld settings. AUC and HM are the two most important metrics. From the table, we can observe that our method beats the other methods by a significant margin. Specifically, in both AUC and HM metrics, we achieve the best results on all three datasets. In particular, we lead the second place by 2.3% and 2.5% in AUC and HM in the closed-world setting on UTZappos, and by 4.2% and 2.8% in AUC and HM in the openworld setting. In the closed-world setting on the challenging dataset CGQ we lead the second place by 1.9% and 2.6% in AUC and HM. For MIT-States, which has relatively more noise. Our method leads the second place by 0.8% and 0.9% in AUC and HM in the closed-world setting. These results illustrate that the proposed method effectively improves the generalization of the model.

4.3 Ablation Study

We conduct extensive empirical analysis to assess the impact of the individual modules as well as the hyper-parameters.

Effect of keymodules. A series of ablation experiments are conducted to investigate the role of each module. The results of the experiments are shown in Table 2. From the table, it can groups is agnostic. We investigate the relationship between the setting of the group number and the model performance. The results are shown in Fig. 3 (a) and (b), which demonstrates the effect of the setting of the group number on the state accuracy and object accuracy on MIT-States. The results show that when the number of groups is set too small, e.g., 2, a substantial performance decay occurs, indicating that the weak expert panel has difficulty in handling the heterogeneous group problem. And when the number of groups gradually increases, the performance does not increase linearly, which may be caused by the saturation of the number of groups.

Effect of hyper-parameter λ. We investigate the effect of the coefficient λ in Eq. 19 on the performance of the model. The results are shown in Fig. 3 (c) and (d). It can be observed that the sensitivity of the model to λ is within a reasonable range. We suggest setting a larger value on a small dataset such as UT-Zappos to promote fast model fitting, and a smaller value on a large dataset such as C-GQA to avoid unstable training.

4.4 Visualization Analysis

To deeply investigate the role of the key components, we perform a series of visualization analyses.

Attention visualization for GAVR. To further explore the effect of GAVR, we perform a visual comparison. As can be seen in Fig. 4 (a), single network (w/o GAVR) is driven to learn semantic information shared between heterogeneous groups thus easily capture pseudo-related features. In contrast, GAVR has multiple expert sub-networks that can adaptively process data from different groups, and thus the captured features are more accurate (top row). Meanwhile, the expert networks complements the group-specific semantic knowledge, so the extracted features are richer and more discriminative (bottom row).

T-SNE Analysis for DGP. To study the context learned by group prompts, we map them to the visual space. Fig. 4 (b) and (c) show the feature distribution of the same state across different objects and the same object across different states, respectively. It can be observed that the significant group heterogeneity in the same state (object). Meanwhile, The proposed group prompts capture different contexts. These group prompts provide distributed centers so that visual features do not have to be compressed into a single prototype thereby maintaining group-specific semantic information.

5 Conclusion

In this paper, we explore the drawback of existing approaches to deal with conditional dependencies, i.e., the zero-sum game of transferability and discriminability. To solve this problem, we provide a new perspective, inspired by the human notion of hierarchical clustering, that formulates state (object) representation learning into homogeneous group representation learning. To this end, we propose a new framework, HGRL, which aims to adaptively discover homogeneous groups and extract group-specific semantic information. Our method consists of three key components to facilitate both visual and prompt representations. Experiments demonstrate the effectiveness of the proposed method.

Acknowledgements

This research was supported by funding from the Hong Kong RGC General Research Fund (152211/23E, 15216424/24E, and 152115/25E), the PolyU Internal Fund (P0056171), and the Huawei Gifted Fund.