TsCA: On the Semantic Consistency Alignment via Conditional Transport for Compositional Zero-Shot Learning

1 Introduction

Identifying new concepts from a set of seen primitives is one of the fundamental challenges for AI systems to imitate the human learning process. Imagine a new cuisine-Spicy Chocolate Cake. Although it may sound like an odd combination, one can still perceive its appearance and taste based on existing knowledge of 'spiciness' and 'Chocolate'. Likewise, compositional zero-shot learning (CZSL) [Misra et al., 2017;

∗Corresponding author: Jingcai Guo.

Figure 1: We represent each image as a set of patch embeddings and two sets of textual embeddings and employ semantic consistency conditional transport to align such cross-modal distribution trio.

Naeem et al., 2021] emerges as a compelling paradigm that seeks to endow machines with similar cognitive abilities, allowing them to understand the composed primitives (i.e., states and objects) during training and generalize to unseen compositions for inference.

Technically, studies on CZSL have revolved around achieving fine-grained alignment across the image and composed state-object text domains [Huynh and Elhamifar, 2020]. Pioneer methods typically load pre-trained image encoders and word embeddings to extract multi-modal features. Various alignment tricks are then applied to constrain the shared latent space, including concept learning [Xu et al., 2021], geometric properties [Li et al., 2020], semantic transformation [Nagarajan and Grauman, 2018], and graph embedding [Mancini et al., 2022]. Moving beyond these postprocessing alignments, a series of prompt-tuning methods have been developed to explore pre-trained vision-language models (VLMs) for the CZSL scenario [Xu et al., 2022; Nayak et al., 2023], achieving state-of-the-art prediction performance. Pre-trained on massive amounts of image-text pairs, VLMs (e.g., CLIP [Radford et al., 2021]) show impressive abilities to match the visual image with its textual label (prompt) in the multimodal space. Those models often employ a multi-path paradigm for fine-grained textual features, e.g., one composition and two primitive branches. One of the core challenges ensues: How to align the image with such multiple textual labels? Recent studies attempt to address it from different fields, such as hierarchical prompt searching [Wang et al., 2023], decomposed feature fusion [Lu et al., 2023], and multi-step observation [Li et al., 2023a]. Although showing attractive results, these methods focus primarily on image-to-composition (or to-primitive) alignments, ignoring intrinsic relationships within the composition and its primitives. This may fail to capture semantic consistency among the image-composition-primitive interactions, leading to suboptimal alignments.

To address the above issues, this paper proposes the Trisets Consistency Alignment framework (dubbed TsCA), which is built upon a novel Consistency-aware Conditional Transport (CCT) derived from the new view of CZSL. As shown in Fig. 1, we represent an image in three directions, i.e., P1, a distribution over all visual patches; P2, a distribution over the composition set; and P3, a distribution over the primitive set. Concretely, P1 captures the detailed visual features of an image, while P2 and P3 denote the global and local textual concepts of the same content. Therefore, the task of CZSL can be viewed as aligning these three discrete distributions as closely as possible. Accordingly, it is indeed key to properly measure the distance between empirical distributions with different supports. Fortunately, conditional transport (CT), well-examined in recent research, offers a bidirectional measurement of the distance from one distribution to another, given the cost matrix. Inheritedly, TsCA extends the minimization of CT to the alignment of our cross-modal distribution trio.

Specifically, TsCA gracefully facilitates the matching of image-composition-primitive triplets by meticulously crafting three pairs of CTs, thereby aligning the rich tapestry of cross-modal and intra-modal semantics. First, CT (P1,P2) measures the transport distance between the visual patch set and the composition set. On the one hand, it helps the label be transported to the compositional visuals with higher probabilities, and on the other hand, it regularizes the finetuning for better cross-modal alignments. Next, CT (P1,P3) focuses on discovering the corresponding attribute and object patches from the disentangle perspective. Last, unlike the above two, CT (P2,P3) aims at intra-modal interactions, which are often overlooked by previous studies. By minimizing the transport distance between the composition and its primitives explicitly, the textual outputs are assumed to present high semantic coherence. E.g., the composition vectors are closer to their primitives in the embedding space. Moreover, the learned transport plan in CT (P2,P3) further provides an efficient tool to filter out unfeasible compositions, which benefits the open-world setting.

More importantly, as discussed above, TsCA formulates the CZSL as the alignment of three distributions and seeks to minimize their transport costs. An intuitive solution is to run pairwise transportation and optimize the above three CT costs. Unfortunately, this case could not model the complex relations among these sets. To capture deeper interactions, we extend the naive CT to a well-designed consistency-aware CT (CCT) for the CZSL triplet case. Motivated by the fact that these three distributions describe the same semantics, CCT regularizes the product of three transport plans as a diagonal matrix from clockwise direction (shown in Fig. 1). In other words, the composition label will return to itself after a cycle of transport, which ensures semantic consistency across sets. In summary, our contributions are three-fold:

• We formulate the CZSL task as a transport problem and propose Trisets Consistency Alignment model (TsCA), which views an image as three discrete distributions over the patch, composition, and primitive spaces and tries to explore fine-grained alignments between these triplets.
• We develop consistency-aware CT (CCT), considering the semantic consistency across the image-compositionprimitive sets, improving the alignment robustness.
• Extensive comparisons and ablations on three benchmarks demonstrate the effectiveness of the proposed TsCA with competitive performance on all settings.

2 RelatedWork

2.1 Compositional Zero-Shot Learning

CZSL[Mancini et al., 2021] is a specialized case of zeroshot learning (ZSL) [Liu et al., 2023]. Given the same set of objects with associated states, it focuses on recognizing unseen state-object compositions by learning from seen compositions. In general, traditional CZSL can be divided into two streams, i.e., compositional classification and simple primitive classification. The former directly predicts compositions by aligning visual features and composed labels in a shared space, or resorting to a graph network for contextuality modeling [Tanwisuth et al., 2023; Saini et al., 2022; Li et al., 2022]. Conversely, the latter identifies states and objects independently and constructs the joint compositional probability distribution. They either disregard the contextuality between primitives or impose training-specific correlations that are detrimental to generalization.

Recently, equipped with prompt learning, large-scale VLMs like CLIP are empowered to adapt to CZSL by reducing domain shift and leveraging pre-trained knowledge .

Further, researchers attempt to explore merging those two typical paradigms to create an integrated multi-path paradigm. For instance, HPL [Wang et al., 2023] learns three hierarchical prompts by explicitly fixing the unrelated word tokens in the three embedding spaces at different levels. Troika [Huang et al., 2024] effectively aligns the branch-specific prompt representations and decomposed visual features with a crossmodal traction module. Our TsCA aligns closely with this paradigm, although with a greater emphasis on direct inquiries into semantic alignment within distributions.

2.2 Conditional Transport

Recently, CT has acted as an efficient tool to measure the distance between two discrete distributions [Zheng and Zhou, 2021]. Through learning bidirectional transport plans based on semantic similarity between samples from both source and target distributions, CT achieves fine-grained matching with the mini-batch optimization. More importantly, CT can integrate seamlessly with deep-learning frameworks and its effectiveness has been widely demonstrated in various applications [Li et al., 2023b; Wang et al., 2022; Tanwisuth et al., 2023]. For example, [Tian et al., 2023]

propose a novel CT-based imbalanced transductive few-shot learning model to fully exploit unbiased statistics of imbalanced query samples. Similarly, [Tanwisuth et al., 2021] employs a probabilistic bidirectional transport between target features and class prototypes for unsupervised domain adaptation, showcasing robustness against class imbalance and facilitating domain adaptation without direct access to the source data. For the first time, through empirical demonstration, we establish that CT, boasting attributes like lower complexity and better scalability, is equally viable for CZSL.

3 Methodology

The proposed model aims to solve the CZSL from fine-grained alignments under the CT framework, where images are viewed as three discrete distributions over the image, composition, and primitive spaces. A novel consistency-aware CT is further developed to explore deeper interactions of these three domains. In this section, we start with the task of CZSL, and then introduce how to formulate CZSL as a CT problem in detail. The framework of our approach is shown in Fig. 2.

3.1 ProblemFormulation

Given state set S = {s0, s1, ..., s|S|} together with object set O = {o0, o1, ..., o|O|} , we can define the label space with their Cartesian product, C = S × O. Then C can be divided into two disjoint label subsets such that C se ∈ C, C us ∈ C, and C se ∩ C us = ∅ where C se and C us are the set of the seen and unseen classes respectively. Specifically, during the training phase, C se are used to train a discriminative model M : X-→C se from the input image space to candidate composition label set. At inference time, the model is expected to predict unseen compositions in the test sample space, i.e.,M : X-→C test . In this paper, we follow the setting of Generalized ZSL [Xian et al., 2017], considering testing samples contain both seen and unseen compositions. In general, in the closed-world evaluation, only the known composition space of test samples is required as C test = C se ∪C us . For the open-world evaluation [Karthik et al., 2022], the model has to consider all possible permutations of the state-object pairs, i.e., C test = C.

3.2 Image as Three Sets

Built upon the pre-trained CLIP and with the soft prompt tuning technique, our TsCA represents an image as three sets: visual patch set , textual composition set , and primitive set . These sets capture the multimodal features of the same content from different semantic domains, acting as a fundamental role in the fine-grained alignments.

Patch Set. For an input image x, CLIP first splits it into N non-overlapping patches evenly and then feeds them (with a [CLS] token inserted) into the image encoder to extract the embedding of the [CLS] token xcCLS and the patch features x = {xn|Nn=1 } ∈ Rd×N , where xn ∈ Rd denotes the embedding of patch n with the embedding dimension being d. Naturally, the discrete probability distribution over the patch set can be formulated as:

where δ denotes the Dirac delta function. We view all the patches equally and employ the Uniform distribution to model the patch weight θ. Note that P1 collects the detailed visual features of the local region, thereby bringing benefits to discriminative representation learning, especially when different concepts require emphasis on distinct areas within the input images. Besides, the [CLS] visual token xcCLS is also learned, serving as the global representation, which is often used as the image feature to downstream tasks [Zhou et al., 2022]. Here, we consider it as the visual prior that constructs subsequent textual sets.

Composition Set. In the textual domain, we build composition labels into a learnable soft prompt [v1][v2][v3][state][object], where [v] is the prefix vectors, and [state][object] denote the state and object name, respectively. Then, the prompt will be fed into the text encoder to obtain textual representations y = {ym|Mm=1 } ∈ Rd×M , where M is the number of training compositions. To embrace the variety of content and minimize the cross-modal disparities, we follow previous work [Huang et al., 2024] and incorporate a residual component obtained through a cross-attention mechanism [Vaswani, 2017] with patch embeddings (we here still use y as the output compositions):

where yin denotes the input compositions. Cross-Att is the cross-attention layer with the query yin , key x, and value x as inputs, and outputs the fused features. As a result, the composition set can be viewed as:

where σ denotes the softmax function. We calculate the composition weights α via the semantic similarity of the composition label and visual feature. This helps P2 to focus on compositions that describe the input image well.

Primitive Set. Unlike the composition set that focuses on global textual features, the primitive set aims to explore the disentangled state and object representations. Motivated by the recent multi-path paradigms, we learn the corresponding primitive representations through [v1][v2][v3][state] and [v1][v2][v3][object], with the outputs denoted as zs ∈ Rd×|S| and zo ∈ Rd×|O| respectively. The corresponding probability weights are calculated as:

where xsCLS and xoCLS are two adapted visual features, which can be obtained via a lightweight adapter g:

where g takes the image feature xcCLS as input, and outputs the state/object-relevant features, deriving unique visual representations. To complete the primitive set, we concatenate the feature points from both state and object labels as z = {zk|Kk=1 } ∈ Rd×K , where K = |S| + |O| is the total number of state and object labels. Like compositions that fuse the patch features with Eq. 2, z is fed into the same cross-attention layer:

Figure 2: The overall framework of the proposed TsCA (zoom-in for more details).

Finally, the primitive set can be expressed as:

where ⊕ denotes the concatenation operation. Together with P2, these two sets provide primitive-composition textual knowledge for downstream alignment tasks.

3.3 Semantic Consistency Alignment

Based on the three carefully constructed sets of the input image, our TsCA formulates the CZSL task as fine-grained alignments under the consistency-aware CT framework, which consists of pairwise CT distance and cycle consistency regularization.

Pairwise CT Distance. Given the source distribution P1 and the target distribution P2, CT measures the distance by calculating the total transport costs bidirectionally, leading to the forward CT and backward CT, respectively. Denoting c(xn, ym) ≥ 0 as a cost function to define the difference between points xn and ym, the forward CT is measured as the expected transport cost of transporting all points fromP1 to the target P2 , and the backward CT reverses the transport direction. Mathematically, the CT distance between P1 and P2 can be expressed as:

where we employ the cosine distance as the cost function, i.e., the closer the patch xn and composition ym are, the lower the transport cost. 1M is the M dimensional vector of ones. T is called the transport plan, which is to be learned to minimize the total distance. The forward transport plan, e.g., -→t nm in -→T describes the transport probability from the source point xn to target point ym:

where sψ denotes the similarity function with learnable parameter ψ, and we specify it as sψ(xn, ym) = xTn ym exp(ψ) . That is, the patch xn will transport with higher probability if the composition ym shares similar semantics with it. Similarly, we have the backward transport expressed as:

More interestingly, the definition in Eq. 9- 10 naturally satisfies the constraint in Eq. 8, simplifying the CT optimization.

By combining P1, P2, and P3 pairwise, we extend CT to the CZSL scenario and denote the total CT distances as:

where the first two terms facilitate cross-domain alignments but focus on different semantic levels. CT (P1,P2) takes the patch set and composition set as inputs and aims to find compositional visuals that match its textual label. While CT (P1,P3) pays more attention to primitive alignment. Intuitively, the primitive set provides decoupled textual guidance, and it helps the model extract patches that contain the state or object-relevant visuals, which will improve the fine-grained predictions. Moving beyond the vision-language alignments, the last term attempts to explore the intra-modal interactions. CT (P2, P3) optimizes the composition and its primitives to be semantically close in the embedding space, showing linguist coherence.

Cycle Consistency Regularization. Due to the independent operation in Eq. 11, pairwise CTs may lead to potential inconsistencies and inefficiencies, as they fail to fully capture the global relationships among all sets involved. Cycle consistency, which has been extensively explored in domains such as image matching [Bernard et al., 2019], multi-graph alignment [Wang et al., 2021], and 3D pose estimation [Dong et al., 2019], offers a natural solution to this problem. By enforcing a closed-loop structure, cycle consistency can enhance the coherence of semantic relationships across all sets, ensuring more consistent and efficient alignment. Here, we denote the probability of the composition set being self-mapping back to its initial state as:

where ←- T21 from CT (P1,P2) denotes the transport plan from P2 to P1. This guarantees each label is transferred with a probability of 1 during transportation among three sets. For a given input, the corresponding point from the composition set first interacts with key patches in the patch set, then moves to the relevant state and object points within the primitive set, and finally returns to the composition set. Accordingly, we derive the cycle-consistency constraint by:

where ycm denotes the one-hot binary label vector of the input image.

For one thing, such a closed-loop transport constraint establishes a close link between independent pairwise CTs, which supports maintaining semantic consistency throughout the CT transport process. For another, it facilitates more robust composition representation learning, enhancing the model's accuracy.

Primitive Decoupler. In light of the intricate entanglement between attributes and objects within an image, we devise a decoupling loss with the idea that visual representation xsCLS and xoCLS can be seen as state-expert and object-expert if their paired sub-concepts fail to be inferred:

where zo gt and zs gt are textual representations for the groundtruth object class and attribute class, respectively. By mitigating the entanglement among visual representations, TsCA enhances its capacity to pinpoint correlative image representations that align with specific knowledge, thereby offering guidance on the textual sets prior and playing a complementary role in deriving a more accurate composition during inference.

3.4 Tranining and Inference

Training Objectives. Recalling that the probability weights α, βs , and βo , calculated during the construction of the three sets, capture the semantic similarity between the visual image and the corresponding textual features, this naturally defines the probability for predicting the labels of state s, object o, and composition c for the image x:

Then the classification losses are given by:

Let Lbase = Lc +Ls +Lo, the overall training loss is defined as follows:

where λ- are hyper-parameters to balance the losses. Like the previous works, the first term is our base classification loss. It will be used to predict the final label via a combined strategy during the inference. In addition, it helps to construct increasingly coherent textual sets as the loss decreases. The last three terms act as semantic regularization, which guides the learning process from various domain experts.

Inference. With multi-path union, the prediction results of states and objects can be incorporated to assist the composition branch. Formally, the integrated composition probability can be denoted as:

where γ balances the contributions of primitive prediction and composition prediction in multi-path learning.

Additionally, we apply a single CT computation to filter out infeasible compositions that might be present in the openworld setting. Concretely, we calculate the bidirectional transport plans between primitives and compositions as their feasible scores, and discarded less relevant pairs by a threshold T

This strategy reduces the search space and increases performance simultaneously.

4 Experiment

4.1 Experimental Settings

Datasets. The proposed TsCA is evaluated on three realworld CZSL benchmark datasets: MIT-states[Isola et al., 2015], UT-Zappos[Yu and Grauman, 2014] and C-GQA [Naeem et al., 2021]. MIT-States comprises images of naturally occurring objects, with each object characterized by an accompanying adjective description. It comprises 53,753 images depicting 115 states and 245 objects. UT-Zappos is a fine-grained dataset consisting of different kinds of shoes with texture attributes, totaling 16 states and 12 objects. CGQA, derived from the Stanford GQA dataset [Hudson and Manning, 2019], features 453 states and 870 objects with over 9,500 compositions, making it the most pairs dataset for CZSL. We follow the split suggested by the previous work [Purushwalkam et al., 2019] to ensure fair comparisons.

Metrics. Following the common practice of prior works [Lu et al., 2023], we utilize the standard evaluation protocols and assessed all results using four metrics in both closed-world and open-world scenarios. Concretely, S measures the best seen accuracy when calibration bias is +∞ and U denotes the accuracy specifically for unseen compositions when the bias is -∞. To provide an overall performance measure on both seen and unseen pairs, we also report the area under the seen-unseen accuracy curve (AUC) by varying the calibration bias from +∞ to -∞ and identify the point that achieves the best harmonic mean (H) between the seen and unseen accuracy. Among these, H and AUC are the core metrics for comprehensively evaluating the model.

Implementation Details. The proposed TsCA and all baselines are implemented with a per-trained CLIP ViT-L/14 model in PyTorch [Paszke et al., 2019]. For the UT-Zappos, the hyper-parameters λ0, λ1, λ2, λ3 in losses are set as 1, 0.1, 10, and 0.1. For the MIT-States, the hyper-parameters are set as 1, 0.01, 0.1, and 0.01. For the C-GQA, the hyper-parameters are set as 1, 0.01, 0.3, and 0.01. During inference, γ is set to 0.8 for UT-Zappos and 0.4 for MIT-States and C-GQA in the closed-world scenario and is adjusted to 0.4, 0.3, and 0.2 for each of the datasets in the open-world setting. The primitive adapters consist of two individual single-layer MLPs. All experiments are performed on a NVIDIA RTX A6000 GPU.

4.2 Comparision with State-of-the-Arts

We compare our TsCA with the most recent CZSL methods. The results are shown in Tab. 1 and Tab. 2. On the closedworld setting, TsCA exceeds the previous SOTA methods on all datasets in core metrics. Specifically, it yields improvements of 0.6% on MIT-States, 2% on UT-Zappos, and 1.1% on C-GQA in H over the second-best methods. It also attains considerable gains in AUC with increases of 0.5%, 1.6%, and 0.8%, respectively. Notably, the advantage of UT-Zappos is more significant, suggesting that the fine-grained nature of UT-Zappos can better align with TsCA's ability to achieve precise alignment across local visual features and textual representations. Similarly, the results in the open-world setting are also promising. Our TsCA attains the highest AUC across all datasets , with the scores of 8.7% , 37.1% and 4.5%. It should be noted that the CT-based filtering strategy contributes to the improvements in this challenging setting. All numerical results substantiate our motivation to empower the model to capture semantic consistency among the image-compositionprimitive interactions.

4.3 Ablation Study

We empirically verify the effectiveness of each component in TsCA on UT-Zappos by comparing it against five variants. The results, seen in Tab. 3, illustrate several observations: 1) The baseline model, i.e., removes the consistency-aware CT module and decoupler loss, shows the lowest performance. 2) Introducing either pairwise CT loss or decoupler loss on top of the baseline model significantly boosts all metrics. Notably, both primitive decoupler and cycle-consistency constraint positively contribute to pairwise CT. The former improves the quality of the textual sets, while the latter ensures semantic consistency during the transport chain, each leading to further enhancements in the model's effectiveness. 3) Combining the strengths of all components, our complete model achieves the best performance in terms of H and AUC.

To demonstrate the effectiveness of cycle-consistency, we draw the self-mapping matrix of the composition set for our full model on test data in Fig. 4. Though degradation occurs in the unseen pair positions, the results still exhibit a close similarity to the identity matrix.

4.4 Qualitative Results

We further provide some visualization examples of transport plans learned in our TsCA. Recalling that the columns ←- T21 and ←- T31 depict how likely the corresponding textual semantics are transported to each visual patch. We convert each transport plan into heatmaps and resize them to combine with the raw image at Fig. 5. We observe that different prompts from multi-path tend to align different patch regions, each of which contributes to the final prediction. For example, in the first row, the state branch emphasizes the wet fur, the object branch highlights the nose and mouth as key features for recognizing a cat, while the composition branch provides a more comprehensive focus on both aspects.

Moving beyond the visualization of cross-modal alignment, ←- T32 and -→T23 also grant us access to intra-modal components, prompting an exploration of the interplay between compositions and primitives in an interpretable form. Fig. 3 presents the top-3 transport retrieval results for bidirectional transfers between the composition and primitive sets. Note that our model can both retrieve the correct composition from the primitive set, and also adeptly performs the inverse mapping of conceptual pairs. Besides, all top-3 retrieval results not only ensure the rationality of the state-object combination but also conform to the description of the images. We also present some failure cases. Interestingly, even when incorrect, the retrieved labels still capture the content of the given images, demonstrating the effectiveness of our TsCA.

5 Conclusion

In this paper, we revisit compositional generalization in CZSL through a conditional transport perspective. We explore pairwise CTs between the local visual features of images and two textual label sets within a multi-path paradigm. We then introduce cycle-consistency as a link to bond all sets, promoting robust learning. The primitive decoupler further improved accuracy and decoupled global primitive visual representations during prediction. Benefited from transport plan between primitive set and composition set, this approach successfully narrowed the composition search space by excluding unfeasible pairs during inference. Extensive experiments across three benchmarks consistently demonstrate the superiority of TsCA. Comprehensive ablation studies and visualizations confirm our motivation and the essential role of each component. With its inherent flexibility and simplicity, we hope our work inspires innovative ideas for future research.

Acknowledgments

This research was supported by funding from the Hong Kong RGC General Research Fund (No. 152211/23E, 15216424/24E, and 152115/25E), the PolyU Internal Fund (No. P0056171), and the Huawei Gifted Fund.