VS-Boost: Boosting Visual-Semantic Association for Generalized Zero-Shot Learning

1 Introduction

Recently, zero-shot learning has made great progress and attracted increasing attention. Conventional zero-shot learning (CZSL) [Lampert et al., 2013] aims to only recognize objects of unseen classes through a classifier learned from seen classes and semantic embeddings e.g. attributes and word embeddings. Unlike CZSL with the strong assumption that the query objects are only from unseen classes, generalized zero-shot learning (GZSL) [Xian et al., 2018a] aims to recognize both seen and unseen classes, which is more challenging.

∗Corresponding authors.

To better transfer knowledge from seen classes to unseen classes, for each category, zero-shot learning introduces a corresponding semantic embedding as prior knowledge e.g. manual annotated attributes [Lampert et al., 2013], word embeddings extracted by language models [Reed et al., 2016], etc. The current mainstream solutions for GZSL are semantic embedding methods [Huynh and Elhamifar, 2020] [Xie et al., 2019] and feature generation methods [Xian et al., 2018b] [Li et al., 2019a] [Xian et al., 2019]. The semantic embedding methods project features into the semantic space and perform metric learning in the semantic space to learn a visual-tosemantic inference, and finally perform classification in semantic space using nearest neighbors. Due to the absence of unseen classes, embedding methods are usually biased towards seen classes and performance is inferior to generation methods. The feature generation methods first train a generator to synthesize unseen features conditional on unseen semantic embeddings and Gaussian noises, then the synthetic features and real seen features are used to train a GZSL classifier in a supervised way. Recently, to enhance feature discriminability, some methods [Han et al., 2020] [Han et al., 2021] [Chen et al., 2021a] introduce metric learning into feature generation methods, which use triplet loss [Wen et al., 2016] or contrastive loss [Hadsell et al., 2006] and their variants to increase inter-class distance and decrease intra-class distance. However, as illustrated in Figure 1, these featurerefined methods and embedding methods only perform metric learning in feature space alone or in semantic space alone, ignoring the association between the feature space and the semantic space. As is known to all, GZSL uses semantics as prior knowledge to transfer visual knowledge from seen classes to unseen classes and there is a gap between visual and semantic information, thus the association between vision and semantics becomes a crucial problem. To boost visualsemantic association for GZSL, we propose a new feature generation-based method termed VS-Boost and introduce a novel relational metric learning which can bridge the metric learning between two different spaces.

VS-Boost first uses a semantic embedding network to constrain the visual features, where the features extracted by ResNet101 [He et al., 2016] (parameter freezing) are first fine-tuned by the encoder and then projected into the semantic space, and metric learning is performed in the semantic space. The fine-tuned features will be more relevant to the semantics and more discriminative through the constraint of the semantic embedding network. After obtaining semanticrelevant features, the proposed relational metric learning is used to further enhance the consistency between the visual and semantic spaces. Concretely, relational metric learning measures the similarity between instances in the feature space and the semantic space and aligns the similarity of the same categories between the two spaces. We use binary crossentropy loss to align the similarity between the two spaces and give proof of the validity of the loss function. It is well known that the APY [Farhadi et al., 2009] dataset is the least generalized dataset due to the huge difference between seen and unseen classes, and VS-Boost greatly improves the SOTA level on APY, which indicates that boosting the association between vision and semantics is an effective way to solve the GZSL problem.

In this paper, our contributions are as follows:

• We propose a novel relational metric learning, which can relate the metric learning of two different spaces and enhance the consistency of the distribution of the two spaces.
• Based on the feature generation method and relational metric learning, we propose a novel framework for GZSL, termed VS-Boost, which effectively enhances the association between visual space and semantic space, thus greatly improving the generalization of the model to unseen classes.
• We evaluated our method on five GZSL benchmark datasets and experimentally find that ours achieves competitive results with significant gains.

2 RelatedWork

2.1 Conventional Zero-Shot Learning

Early zero-shot learning methods focused on the conventional zero-shot learning (CZSL) problem, where the testing set only contains unseen classes. Semantic embedding models [Frome et al., 2013] [Akata et al., 2015a] [Akata et al., 2015b] [Romera-Paredes and Torr, 2015] [Kodirov et al., 2017] [Xian et al., 2016] learn a mapping from an image feature space to a semantic space. The classic semantic embedding methods DAP and IAP [Lampert et al., 2013] make use of the semantic embeddings within a two-stage approach to infer the label of an image that belongs to one of the unseen classes. In addition, other hybrid models [Zhang and Saligrama, 2015] [Norouzi et al., 2013] [Changpinyo et al., 2016] embed both images and semantic embeddings into another intermediate space to perform classification. These embedding methods have achieved good results on CZSL task.

2.2 Generalized Zero-Shot Learning

The concept of generalized zero-shot learning (GZSL) [Xian et al., 2018a] has received significant attention since its proposal. In GZSL, the testing set contains both seen and unseen classes, due to the overfitting of seen classes, the existing CZSL methods decline dramatically in performance and suffer from a very serious strong-bias problem. In order to solve the problem of shortage of unseen-class samples, the generative adversarial networks (GAN) [Goodfellow et al., 2014] [Mirza and Osindero, 2014] [Arjovsky et al., 2017] and variational auto-encoding (VAE) [Kingma and Welling, 2013] were introduced for GZSL, where a generator was trained to synthesize unseen-class visual features conditional on corresponding semantic embeddings. Most of the current feature generation methods [Xian et al., 2018b] [Felix et al., 2018] [Li et al., 2019a] [Sariyildiz and Cinbis, 2019] [Xian et al., 2019] [Narayan et al., 2020] attempt to learn an inference from semantic embeddings to visual features and some methods [Verma et al., 2020] [Liu et al., 2021] introduce the meta-learning strategy into the feature generation method to improve the generalization of the model. The common space methods [Ma and Hu, 2020] [Schonfeld et al., 2019] [Chen et al., 2021b] propose to learn a common space into which both visual features and semantic embeddings are projected for effective knowledge transfer. In addition to the feature generation methods, the prototype generation methods [Li et al., 2019b] [Yu et al., 2020] [Liu et al., 2020] also achieved good results in GZSL, where the semantics-to-prototype mapping is trained and the synthetic prototypes are used as a classifier for different classes. The attention-based methods [Xie et al., 2019] [Huynh and Elhamifar, 2020] [Min et al., 2020] [Chen et al., 2022] usually use attention mechanisms to extract visual features which fit better with the semantics and design the new loss functions to balance the predictions between seen and unseen classes in GZSL task. There are also some open-set classification methods [Yue et al., 2021] [Chou et al., 2020] applied in zero-shot learning, which first separate the unseen classes from the seen classes, and then classify them separately.

Recently, in order to enhance the discriminability of features, some methods [Han et al., 2020] [Han et al., 2021] [Chen et al., 2021a] introduce metric learning e.g. triplet loss [Wen et al., 2016] [Schroff et al., 2015] and contrastive loss [Hadsell et al., 2006] to the generation method, but these methods only perform metric learning in the feature space without considering linking features to the semantic space, which is not conducive to model generalization.

3 Method

3.1 ProblemDefinition

The zero-shot learning problem is defined as follows: a training (seen classes) dataset Dtr = {(xi, yi)|xi ∈ X, yi ∈ S}, where xi is the visual feature and yi is its corresponding label, S is the label set of seen classes. In CZSL task, the testing set is denoted as Dte = {(xj , yj)|xj ∈ X , yj ∈ U}, where U is the label set of unseen classes and S ∩ U = ∅. While in GZSL task, the testing set is denoted as Dte = {(xj , yj)|xj ∈ X, yj ∈ S ∪ U}. In zero-shot learning, each seen and unseen class has its own corresponding semantic embedding ak ∈ A, ∀k ∈ S∪U. Given Dtr and A, the task of CZSL is to learn the classifier fczsl : X → U, and the task of GZSL is to learn the classifier fgzsl : X → S ∪U. Due to the strong-bias to seen classes, GZSL task is more challenging than CZSL.

3.2 Method Overview

The architecture of VS-Boost is illustrated in Fig. 2, and it contains two streamlines: the feature generating network and the metric learning module. In the feature generating network, we train a generator G to synthesize the visual features from the semantic embeddings. And in the metric learning module, a feature encoder E is trained for refining original features. Different from the existing metric learning used in GZSL which only focuses on metric learning in the semantic space or feature space, our VS-Boost introduces a novel relational metric learning to relate the measures of semantic spaces and feature spaces. The metric learning module contains the classical semantic embedding network and the proposed relational metric learning. The visual features extracted by ResNet101 [He et al., 2016] are refined by encoder E as h = E(x), and the h is mapped to the semantic space, and semantic metric learning is completed by InfoNCE loss

[Van den Oord et al., 2018]. Moreover, we enforce the relational metric learning to constrain encoder E. As illustrated in Figure 3, relational metric learning first calculates the similarity between refined features by a learnable function F and then measures the similarity of semantic embeddings. Cross-entropy loss is employed to bridge the similarity between features and their corresponding semantic embeddings. Through relational metric learning, the distribution of feature space and semantic space becomes more consistent, which is greatly conducive to the inference of visual tasks with semantic embedding as a cue.

In classification, the trained feature generator G will be used to synthesize features of unseen classes, then the synthetic unseen features and real seen class features are refined by the encoder E as the input to a classifier.

3.3 Feature Generating Network

The feature generating network [Xian et al., 2018b] introduces GAN into GZSL for the first time and achieves outstanding results than previous methods. GAN learns a feature generator G to synthesize the visual features ˜ x = G(a, ϵ) conditioned on a class-level semantic embedding a and Gaussian noise ϵ ∈ N(0, 1). At the same time, the discriminator of generator D is cross-iteratively trained with the generator to discriminate between a real pair (x, a) and a synthetic pair (˜ x, a). The generator tries to generate a more realistic synthetic feature ˜ x with its corresponding semantic embedding a. The generative model adopts the Wasserstein Generative Adversarial Networks (WGAN) [Arjovsky et al., 2017] and introduces the gradient penalty term [Gulrajani et al., 2017] to train GF and D, the adversarial training loss of WGAN can be formulated as:

where E indicates expectation, ˜ x = G(a, ϵ), ˆx = αx + (1 - α)˜ x with α ∼ U(0, 1) and γ is the penalty coefficient. As suggested in [Gulrajani et al., 2017], we fix γ = 10.

3.4 Semantic Metric Learning

The semantic embedding network [Frome et al., 2013] [Akata et al., 2015a] [Akata et al., 2015b] [Romera-Paredes and Torr, 2015] [Kodirov et al., 2017] [Xian et al., 2016] was originally used in CZSL to learn a mapping function R that maps a visual feature x into the semantic space denoted as R(x). The commonly-used semantic embedding methods rely on a structured loss function [Akata et al., 2015b][Frome et al., 2013] formulated as below:

where a+ is the semantic embedding corresponding to class of x, a- ̸= a is a randomly-selected semantic embedding of other classes, and δ > 0 is a margin. The structured loss is of the same form as triplet loss [Wen et al., 2016] [Schroff et al., 2015; Zhang et al., 2022], allowing the model to perform metric learning in semantic space. Recently, semantic embedding networks is used by many generation methods [Felix et al., 2018] [Narayan et al., 2020] [Chen et al., 2021a] as a reconstructor to give synthetic features a consistency constraint guaranteed that the features synthesized from semantic embeddings can be reconstructed back to semantic embeddings. In this paper, to boost the associations between features and semantics, we introduce the semantic embedding network to impose a semantic measure constraint on original features through training an encoder E to refine features. Unlike some methods[Han et al., 2020] [Han et al., 2021] [Chen et al., 2021a; Hu et al., 2021] that perform metric learning directly in visual space, mapping the visual features to the semantic space and performing metric learning makes the visual features more relevant to their semantics. Furthermore, since semantics have excellent discriminability, using semantic metric learning also makes the model learn to represent more discriminative visual features. Concretely, as illustrated in Fig. 2, the original features x are refined by encoder E, and the refined features h are mapped to the semantic space to obtain the mapped semantic embeddings ˜a = R(h). In order to ensure the discriminability of ˜a in the semantic domain, we employ the current popular infoNCE loss [Van den Oord et al., 2018] instead of structure loss as the objective function for semantic metric learning, which is formulated as:

where τ > 0 is the temperature parameter for infoNCE loss, N is the total number of semantic embeddings. Although semantic metric learning can make features and semantics more correlated, it is still not enough, because it only performs metric learning in a separate space without really bridging the semantic space and the feature space together.

3.5 Relational Metric Learning

Recently, some methods [Han et al., 2020] [Han et al., 2021] [Chen et al., 2021a] have proposed to refine visual features by triplet loss [Wen et al., 2016] or its variants, which effectively improve the performance of GZSL through feature augmentation. However, these methods only perform metric learning in the feature space, which can effectively improve the discriminability of features but cannot enhance the association between features and semantics. In this subsection, we introduce a novel relational metric learning for instancelevel constraint. Unlike conventional metric learning simply pulls intra-class instances closer and inter-class instances farther, relational metric learning effectively relates the metrics in the semantic space with the metrics in the feature space, thus making the distribution in the feature space more consistent with the distribution in the semantic space. Specifically, relational metric learning is based on the learnable similarity function F (see Fig. 3), which learns to predict similarity probability between two features. F is achieved through a learnable inner product similarity and activated by a sigmoid activation function σ, which is formulated as:

where wF is a [2048, 1] fully connected layer, 2048 is the dimension of h and ◦ indicates element-wise multiplication. By scoring the similarity of two instances, F can be modeled as a probability prediction problem. We take the cosine similarity between semantic embeddings as the ground truth and the cross-entropy loss is as follows:

where hi, hj are the refined features of pair-wise instances and ai, aj are their corresponding semantic embeddings. cos(ai, aj) = ai·aj ∥ai∥∥aj∥ indicates the cosine similarity.

Theorem. Since the semantic embedding space does not make any changes, supposed that ζ = cos(ai, aj) ∈ [0, 1] is a constant after the calculation and ξ = F (hi, hj ) ∈ (0, 1) is an independent variable, Equation (5) is expressed as Lbce(ξ) = -ζlogξ-(1-ζ)log(1-ξ), and the partial derivation is formulated as follows:

where Lbce(ξ) achieves a minimum value when and only when ξ = ζ i.e. the distribution relation of feature space is consistent with the distribution relation of semantic space.

Furthermore, imitating triplet loss [Schroff et al., 2015], each instance h is compared with its positive sample h+ and negative sample h- through F, where h+ and h- are sampled at random. Based on Eq.(5), the relational metric loss is as follows:

Through relational metric learning, on the one hand, the discriminability of visual features is improved by the inter-class and intra-class metric learning, on the other hand, it enables a more consistent distribution between semantics and features. The distribution of the visual space is consistent with the feature space in favor of the classification of unseen classes because unseen class classification is guided by semantic cues.

3.6 Optimization

The full model of our VS-Boost optimizes G, D, E, R, and F simultaneously with the following objective function:

where no hyper-parameters are needed to balance the different losses to achieve the desired results.

3.7 Classification

After the proposed VS-Boost has been well trained, the seen features extracted by ResNet-101 [He et al., 2016] and unseen features synthesized by the generator are refined through encoder E as h = E(x). Suppose that the refined feature sets of seen classes and synthetic unseen classes are Hs and ˜Hu, which can be used to train a standard classifier through minimizing the cross-entropy loss:

where θ is the parameter of the classifier, and P(y|h) is the softmax prediction. We denote in CZSL, h ∈ ˜Hu,y ∈ U while in GZSL h ∈ ˜Hu ∪ Hs, y ∈ S ∪ U.

In the testing, the testing features xt are also refined as ht = E(xt). The classification function is:

where in CZSL, y ∈ U and in GZSL,y ∈ S ∪ U.

4 Experiments

Dataset. We evaluate our method on the five benchmark datasets for zero-shot learning: Attribute Pascal and Yahoo (APY [Farhadi et al., 2009]), Animals with Attributes (AWA [Xian et al., 2018a]), Caltech-UCSD Birds-200-2011(CUB) [Welinder et al., 2010], Oxford Flowers (FLO) [Nilsback and Zisserman, 2008] and SUN Attribute (SUN) [Patterson and Hays, 2012]. Among them, AWA, APY, and SUN use class attributes as semantic embeddings, and CUB and FLO use word embeddings extracted by CNN-RNN [Reed et al.,

2016] as semantic embeddings. APY is annotated with 64dimensional attributes and combines datasets a-Pascal and a-Yahoo, which has 30 and 12 classes respectively. AWA is a coarse-grained animal dataset with manually annotated 85-dimensional attributes. While CUB and FLO are two fine-grained datasets with 1,024-dimensional word embeddings. And SUN is a scenario dataset with annotated 102dimensional attributes. Table 1 shows the detailed statistics of the five datasets. Similar to the state-of-the-art generation methods, we extract the 2048-dimensional visual features for five datasets with the backbone ResNet-101 [He et al., 2016] pre-trained on ImageNet [Krizhevsky et al., 2012] without finetuning. In addition, we adopt the Proposed Split(PS) [Xian et al., 2018a] to divide all classes on each dataset into seen and unseen classes.

Evaluation Protocols. Following the evaluation strategy in [Xian et al., 2018a], we compute the average per-class Top-1 recognition accuracy (Acc) as the criteria. We evaluate Acc of unseen classes (noted as U) and seen classes (noted as S). And the performance of GZSL is measured by their harmonic mean: H = 2 × S × U/(S + U).

Implementation Details. We implement our model by using PyTorch based on Python 3.7 platform. The the proposed model is trained and evaluated on one GeForce RTX 3090 GPU. As a pre-processing step, we normalize the visual features like [Li et al., 2019a]. Feature generator G, discriminator D, and semantic regressor R are multilayer perceptrons that contain a 4,096-unit hidden layer with LeakyReLU activation. The feature encoder E is a [2048, 2048] Linear layer with LeakyReLU activation. Finally, we use the task with N way, K shot (N-K) random sampling for training, and use a random mini-batch size of 8-64 for APY and AWA, 4-16 for CUB, 1-32 for FLO and 64-2 for SUN in our method.

4.1 Compared with State-of-the-arts

To evaluate the performance of our method, we compare VSBoost with fourteen SOTA methods: GXE[Li et al., 2019b], DVBE[Min et al., 2020], DAZLE[Huynh and Elhamifar, 2020], AREN[Xie et al., 2019], MSDN[Chen et al., 2022], f-CLSWGAN[Xian et al., 2018b], LisGAN[Li et al., 2019a], RFF-GZSL[Han et al., 2020], TF-VAEGAN[Narayan et al., 2020], TGMZ[Liu et al., 2021], FREE[Chen et al., 2021a], SDGZSL[Chen et al., 2021c], CE-GZSL[Han et al., 2021], and ICCE[Kong et al., 2022]. From Table 2, it is observed that our VS-Boost achieves competitive results. In the harmonic mean H, the main criteria of GZSL, VS-Boost achieves the best results on APY, AWA, CUB, and FLO, and the gains are 10.1%, 1.3%, 0.3%, and 0.8% against TGMZ [Liu et al., 2021], ICCE [Kong et al., 2022], MSDN [Chen et al., 2022] and FREE[Chen et al., 2021a] respectively. VSBoost makes significant gains on GZSL, especially on APY which is recognized as a challenging dataset due to the huge difference between seen and unseen domains. On SUN, the result of the CE-GZSL[Han et al., 2021] is 0.6% higher than ours and VS-Boost is inferior to CE-GZSL[Han et al., 2021] and TF-VAEGAN[Narayan et al., 2020] in terms of S and U. We speculate that the disadvantage is that SUN has 727 classes but only 102-dimensional semantic embeddings that provide very limited information than visual features, which degrades the performance of our VS-Boost. Furthermore, in terms of unseen classes, VS-Boost achieves the best results on APY, AWA, and FLO and achieves second place on other datasets. The gains for unseen classes are 4.6%, 2.7%, and 0.1% on APY, AWA, and FLO, respectively. The performance improvement achieved on GZSL fully validates the effectiveness of our proposed VS-Boost. And the best results on four datasets (especially on APY) indicate that the generalization of our method is more excellent than other methods.

4.2 Conventional Zero-Shot Learning

In addition to the GZSL task, we implement our method on the conventional zero-shot learning (CZSL) task, where the test set contains only unseen classes. We compare VSBoost with nine CZSL methods: DAP&IAP [Lampert et al., 2013], SSE [Zhang and Saligrama, 2015], LATEM [Xian et al., 2016], DEVISE [Frome et al., 2013], SJE [Akata et al., 2015b], ALE [Akata et al., 2015a], ESZSL [RomeraParedes and Torr, 2015], SYNC[Changpinyo et al., 2016], and four resent GZSL methods: LisGAN [Li et al., 2019a], TF-VAEGAN [Narayan et al., 2020], CE-GZSL [Han et al.,

2021], ICCE [Kong et al., 2022]. As documented in Table 3, our VS-Boost achieves significant gains on five benchmark datasets, whether compared with the CZSL methods or the GZSL methods. In detail, VS-Boost made 16.7%, 1.5%, 1.4%, and 0.4% improvements on APY, AWA, CUB, and FLO, respectively. Although the results of VS-Boost are not as satisfactory as some GZSL methods on SUN, it is still 4.3% better than the best CZSL method. The competitive results achieved under CZSL verify the superior capabilities of our VS-Boost.

4.3 Ablation Study

To provide further insight into VS-Boost, we conduct ablation studies to evaluate the effects of semantic metric learning (SML) and relational metric learning (RML). Based on the feature generating network, we introduce SML and RML independently and analyze the results. The results of the ablation study are shown in Table 4, after using SML individually, the results (in terms of H) on the five datasets are improved by 6.3%, 8.3%, 7.1%, 6.8%, and 1.6%, respectively. While using RML individually, the results on the five datasets are improved by 8.7%, 7.8%, 6.0%, 7.0%, and 1.3%, respectively. The great improvements on five datasets fully verify RML and RML both have a significant impact on the VSBoost. Concretely, after using SML, the recognition accuracy of both seen and unseen classes is greatly improved, while, RML has more significant enhancements for unseen classes. We conjecture that it is because RML associates the semantic space with the feature space, which is very beneficial for the generalization of unseen classes. After using both SML and RML, the results (in terms of H) are greatly improved on the five datasets by 12.6%, 10.7%, 9.6%, 9.3%, and 2.5%, respectively.

4.4 Quantitative Analysis

Figure 4(a) and 4(b) show the visualization results of the unseen features synthesized from the semantic embeddings and the real unseen features, with different numbers representing different category centers. It can be seen that after refinement by VS-Boost, the gap between real features and synthetic features becomes significantly smaller. We speculate that this is due to the better connection between the feature space and the semantic space after fine-tuning by VS-Boost, which allows the generator to do the inference from semantic embeddings to visual features more efficiently. Furthermore, as shown in Figure 4(c) and 4(d), we visualize all seen and unseen features, with different colors representing different classes. After the VS-Boost refinement, it can be observed that the refined features are vastly improved in both inter-class discriminability and intra-class aggregation, which shows that our proposed VS-Boost can effectively enhance the discriminability of visual features while enhancing visual and semantic consistency.

5 Conclusion

In this paper, we propose a novel GZSL method termed VSBoost, where a stronger association between visual features and semantic embeddings can be built. VS-Boost first uses a semantic embedding network to extract semantic-relevant visual features and then relates the visual feature space with the semantic embedding space by the proposed relational metric learning. The experimental results on five benchmark datasets demonstrate that VS-Boost improves SOTA performance on four datasets, in particular on APY with a 10% improvement in the harmonic mean. The huge performance improvement indicates that boosting the association between vision and semantics is a very effective solution to the GZSL problem.

Acknowledgements

This work is supported by the National Key Research and Development Program of China No.2020AAA0108301, the National Natural Science Foundation of China under Grant No.62176224, 62222602, and 62176092, Shanghai Science and Technology Commission No.21511100700, and CCFLenovo Blue Ocean Research Fund.