Semantic Compression Embedding for Generative Zero-Shot Learning

1 Introduction

The exciting performance of recent supervised deep learning relies on the massive amount of manually labeled visual samples [Scholkopf and Smola, 2002; He et al., 2016; Cheng et al., 2017]. However, these methods fail to recognize the objects whose classes are not in the training set. Zeroshot learning (ZSL) has been proposed to solve this problem [Palatucci et al., 2009]. Based on the side information shared by seen and unseen classes, ZSL models try to transfer knowledge from seen to unseen classes. Thus, ZSL can Semantic Compression Embedding

∗Equal contribution.

†Corresponding authors.

(b) Our proposed SC-EGG.

Figure 1: Motivation illustration. (a) Existing generative ZSLs learn a mapping from the class-level semantic space to the CNN backbone visual space which lacks attribute-level semantic information. Consequently, the two seen classes are confusing, and the knowledge transfer from seen to unseen classes is also limited. (b) SCEGG learns a mapping from the class-level semantic space to the more discriminative dense-semantic visual space learned by semantic compression embedding, thus synthesizing the high-quality unseen visual features.

recognize the novel unseen classes. According to the classification range, the ZSL task can be grouped into conventional ZSL (CZSL), which merely aims to predict unseen classes, and generalized ZSL (GZSL), which aims at predicting both seen classes and unseen classes.

Recently, generative methods in ZSL (i.e., generative ZSLs) has achieved significant progress [Xian et al., 2018; Xian et al., 2019; Narayan et al., 2020; Chen et al., 2021a; Chen et al., 2021c]. Generative models usually learn an implicit mapping from the class-level semantic space to the visual space using seen data and synthesize unseen visual features from the semantic vectors of unseen classes. Thus, ZSL is converted into a standard supervised classification task, which is helpful for tackling the problems of data imbalance between seen and unseen classes, as well as visual-semantic domain gaps.

However, existing generative ZSL methods simply learn a mapping function from the class-level semantic space to the CNN backbone visual space (represented by the visual features directly extracted by a pre-trained CNN backbone, e.g., ResNet101 [He et al., 2016]), which lacks attributelevel semantic information (e.g., "bill color yellow" and "head pattern plain") that is critical for distinguishing seen classes as well as transferring knowledge from seen to unseen classes in ZSL [Xie et al., 2019; Sylvain et al., 2020; Chen et al., 2021c]. For example, as shown in Figure 1(a), the pre-trained CNN backbone may only extract visual features containing semantic information about "Bird" for three samples of different fine-grained bird categories, which are not discriminative enough. As such, the seen classes are confusing, and the knowledge transferred from seen to unseen classes is also limited.

To address the above problems, we propose a novel Semantic Compression Embedding Guided Generation (SC-EGG) model, which cascades a semantic compression embedding network (SCEN) and an embedding guided generative network (EGGN). The SCEN consists of a local embedding network (LEN) and a global embedding network (GEN). The LEN extracts a group of local visual features whose semantics correspond to attributes. Considering the curse of dimension for feature synthesizing, we introduce a new semantic consistent regression loss to enable the GEN to learn the lowdimension global visual feature whose semantic is consistent with the local visual features for each sample. Thus, we compress attribute-level semantic representations into the global visual features and obtain a dense-semantic visual space, as shown in Figure 1(b). The EGGN employs a generative model to learn a mapping from the class-level semantic vectors to the dense-semantic visual space for better knowledge transfer. To alleviate model overfitting to seen classes, we further introduce an embedding guided synthesis loss that takes the trained classifier in GEN to constrain the visual feature synthesizing for both seen and unseen classes in EGGN. Finally, we take the trained EGGN to synthesize an amount of unseen visual features to train a classifier, which is used for ZSL classification.

Our main contributions are summarized as follows:

• We propose a novel ZSL method, termed semantic compression embedding guided generation (SC-EGG) model, to address the problem of attribute-level semantic missing of the CNN backbone visual features, thus further boosting the performance of generative ZSL.
• We propose a new semantic consistent regression loss for semantic compression and an embedding guided synthesis loss for alleviating generative models overfitting to seen classes.
• Extensive experiments on three challenging benchmark datasets, i.e., CUB [Welinder et al., 2010], SUN [Patterson and Hays, 2012] and AWA2 [Xian et al., 2017],

demonstrate the significant performance gain of SCEGG over current state-of-the-art methods and its baseline.

2 RelatedWork

Embedding-based ZSL. Embedding-based ZSL methods usually learn a mapping from visual to semantic domains [Lampert et al., 2014; Akata et al., 2016], performing the ZSL classification using nearest-likely strategy according to the distance between itself and class-level semantic descriptors. However, most existing methods are based on global visual features that are not attribute-level semanticrelated, resulting in poor discriminative and transferable feature representations. Recently, attention-based local embedding methods [Xie et al., 2019; Huynh and Elhamifar, 2020b; Xu et al., 2020; Han et al., 2021; Chen et al., 2022a; Chen et al., 2022b] were used to enhance the visual feature representations for seen and unseen classes. Unfortunately, these methods learn the ZSL model only on seen classes, inevitably overfitting to seen classes.

Generative ZSL. In order to overcome the limitations of embedding-based ZSL methods, generative ZSL methods employ a generative model to synthesize the unseen visual features for feature augmentation [Xian et al., 2018; Xian et al., 2019; Narayan et al., 2020; Chen et al., 2021b]. Felix et al. [2018] proposed a conditional Wasserstein GAN (WGAN) that synthesizes visual features by optimizing the Wasserstein distance regularized by a classification loss. In [Xian et al., 2019], the authors introduced an f-VAEGAN framework that combined a WGAN and a VAE to take advantage of the strength of both. Later, lots of generative ZSL methods follow up the f-VAEGAN framework because of its excellent performance [Narayan et al., 2020; Chen et al., 2021a; Yan et al., 2021]. However, these generative methods learn the mapping from semantic vectors to global visual features which are directly extracted by the pretrained CNN backbone. Since the global visual features are not semantic-related to the pre-defined attributes in a specific dataset, the learned generative model is poor. As such, the synthesized unseen visual features are not discriminative and transferable.

3 ProposedMethod

Problem Definition. Let x ∈ X denotes an input image, y ∈ Y denotes the corresponding label. Y = Y s ∪ Y u , where Y s = {ys 1 , · · · , ysM } denotes the set of M seen classes, Y u = {yu 1 , · · · , yuN } denotes the set of N unseen classes, and Y s ∩ Y u = ∅. Both seen and unseen classes have the class-level semantic description vector a(yj) = [a1, . . . , aA]T ∈ A, ∀yj ∈ Y s ∪ Y u , which encodes the relationships between all classes, and are available during training. The semantic description vector a(yj) of class yj has A elements, each of which corresponds to an attribute. The tasks in CZSL and GZSL are to learn the classifiers fczsl : X →Y u and fgzsl : X →Y s ∪Y u , respectively.

Overview. As shown in Figure 2, the proposed semantic compression embedding guided generation (SC-EGG) model cascades a semantic compression embedding network (SCEN) and an embedding guided generative network (EGGN). Specifically, the SCEN employs a local embedding network (LEN) and global embedding network (GEN) to learn a local-global consistent embedding for semantic compression. The EGGN contains a generative model (TFVAEGAN [Narayan et al., 2020]) to learn semantic-to-visual mapping guided by a global classifier, which shares parameters with the global classifier in GEN. Preliminarily, we represent the semantic attribute vector of A attributes as {vn}An=1 using a language model (e.g., GloVe [Pennington et al., 2014]), ∀vn ∈ Rda . In addition, we use the pre-trained CNN backbone to extract a set of region features {ri}Ri=1 with R regions for each input image x, ∀ri ∈ Rdv .

3.1 Semantic Compression Embedding Network

Semantic compression embedding network (SCEN) learns a dense-semantic visual space by local-global semantic consistent feature learning, in which the local embedding network (LEN) and the global embedding network (GEN) learn local and global visual features, respectively. The SCEN is constrained by a local embedding cross-entropy loss, a global embedding cross-entropy loss, and a semantic consistent regression loss.

Local Embedding Network

Local embedding network (LEN) firstly locates attributelevel semantic-related image regions to represent a group of local visual features whose semantic is aligned to attributes, using attribute-based attention mechanisms [Huynh and Elhamifar, 2020b]. Thus, the attribute-level semantic-related local features {hn}An=1 are formulated as:

where hn ∈ Rdv denotes the local visual feature whose semantic is aligned with the n-th attribute, Wα ∈ Rda×dv is a learnable matrix.

Then, LEN embeds {hn}An=1 into the semantic space:

where el ∈ RA denotes the feature embeded in semantic space, Wβ ∈ Rdv×da is a learnable embedding matrix, and diag(·) represents the diagonalization operation. After learning the visual-to-semantic embedding, the class prediction result pl of LEN can be computed based on the similarity between the embeded vector el and the class-level semantic vectors a(yi):

where a(yi) ∈ RA, ∀yi ∈ Y s ∪Y u , and pl ∈ RM+N denotes the class prediction score vector.

Global Embedding Network

Global embedding network (GEN) consists of a semantic compression module SC and a global visual feature classifier CLSg. The SC compresses the attribute-level semanticrelated local visual features {hn}An=1 to the dense-semantic low-dimension global visual feature:

where g ∈ Rdv denotes the global visual feature. The SC, which can be a multilayer perceptron (MLP) with A input units and one output unit, compresses the local features along the dimension of attributes to remove redundant semantic. Then the CLSg directly embeds the global visual feature g to the class-label space:pg = CLSg(g), (5)

where the pg ∈ RM+N denotes the class prediction score.

Local Embedding Cross-Entropy Loss

Aiming at ZSL classification, we take the local embedding cross-entropy loss LLCE to train the LEN. The LLCE is formulated as:

where N denotes the batch size, pl(y′ ) is the score predicted by LEN that the x belongs to class y′ .

Global Embedding Cross-Entropy Loss

Simultaneously, we take the global embedding cross-entropy loss LGCE to train the GEN. The LGCE can be formulated as:

where N denotes the batch size and pg(y′ ) denotes the score predicted by GEN that the x belongs to class y′ .

Semantic Consistent Regression Loss

In order to guarantee the low-dimension global visual feature g containing attribute-level semantic the same as local visual features {hn}An=1 (i.e., compressing semantic of local features to the global feature), we take the semantic consistent regression loss LSCR as a constrain for both LEN and GEN:

where pl and pg represent the prediction score vector of LEN and GEN, respectively.

3.2 Embedding Guided Generative Network

We introduce the embedding guided generative network (EGGN), which learns a mapping from the class-level semantic vector as to the dense-semantic visual feature gs (learned by SCEN) under the constraint of the embedding guided synthesis loss. Based on the semantic compression, the densesemantic visual feature gs is low-dimension and attributelevel semantic-related. Thus, EGGN can synthesize discriminative unseen features ˆ gu.

The EGGN is based on the TF-VAEGAN [Narayan et al., 2020], including a variational auto-encoder (VAE) and a generative adversarial network (GAN). The VAE consists of an encoder E and a decoder G. The GAN consists of a generator G, which shares the decoder in VAE, and a discriminator D. In addition, a semantic embedding decoder Dec and a

Algorithm 1 The algorithm of SC-EGG.

Input: the training set {X s , Y s }; the testing set {X u , Y u }; class-level semantic vectors A; the pre-trained CNN backbone ResNet101; the pre-trained language model GloVe; loss weights (i.e., λg, λw, λr, λs, λu); training epoches for each stage: ES1, ES2, ES3 and EC.

Output: the predicted label c∗ for the test samples.

• 1: Take ResNet101 to extract the region features {ri}Ri=1 for each sample x in X .
• 2: Take GloVe to extract the semantic attribute vectors {vn}An=1 for A attributes.
• 3: while iter1 < ES1 do
• 4: Update parameters of SCENwith Equation (16).
• 5: end while
• 6: while iter2 < ES2 do
• 7: Update parameters of GENwith Equation (17).
• 8: end while
• 9: while iter3 < ES3 do
• 10: Update parameters of EGGNwith Equation (18).
• 11: end while
• 12: Using the SCEN to extract gs for seen samples.
• 13: Using the EGGN to synthesize ˆ gu for unseen classes.
• 14: while iterc < EC do
• 15: Update parameters of fczsl (or fgzsl) with gs and ˆ gu as inputs.
• 16: end while
• 17: Predict the label c∗ of the test samples using the trained classifier fczsl (or fgzsl).
• 18: return the predicted label c∗ for the test samples.

feedback module F are introduced to enhance feature synthesizing collectively. The basic loss LVG of TF-VAEGAN can be formulated as:

in which λw and λr are loss weights and:

where KL is the Kullback-Leibler divergence, p(z|a) is a prior distribution (assumed to be N(0, 1)), -logG(z, a) is the visual feature reconstruction loss, g′ = τg+(1-τ)ˆ g with τ ∼ U(0, 1) and λ is the penalty coefficient. More details of the TF-VAEGAN can be refer to [Narayan et al., 2020].

Embedding Guided Synthesis Loss

In order to encourage EGGN to synthesize the discriminative features and avoid the synthetic unseen features overfitting to seen classes, we introduce an embedding guided synthesis loss based on a classifier, which shares parameters with the global classifier CLSg in GEN. Specifically, the embedding guided synthesis loss LEGS includes a synthetic seen crossentropy loss LSCE and a synthetic unseen cross-entropy loss LUCE, formulated as:

where λs and λu are loss weights. LSCE and LUCE are formulated as:

where N denotes the batch size, ps = CLSg(ˆ gs) and pu = CLSg(ˆ gu), ˆ gs and ˆ gu represent the synthetic seen and unseen features, respectively. p(y′ ) denotes the score of the x belonging to class y′ predicted by the CLSg.

3.3 Model Optimization

We train the SC-EGG in three stages:

i) Stage 1: We first train the SCEN by the local and global embedding cross-entropy loss to learn attribute-level semanticrelated local features. The loss of stage 1 is formulated as:

ii) Stage 2: At this stage, we train the SCEN by freezing the parameters of LEN and updating the GEN with the semantic consistent regression loss:

iii) Stage 3: We use the trained SCEN to extract the densesemantic visual features. Then we train the EGGN to learn a mapping from the class-level semantic space to the densesemantic visual space. The total loss of stage 3 is:

3.4 Classification

After training, we use the SCEN to extract dense-semantic visual features gs of seen samples and the generator G of EGGN to synthesize unseen features ˆ gu. Then, we use the gs and ˆ gu to train a CZSL classifier fczsl : X → Y u and a GZSL classifier fgzsl : X →Y s ∪Y u . During inference, test features xt are processed by SCEN to obtain dense-semantic visual features gt, which are further used as input of classifier fczsl or fgzsl for ZSL prediction. The full procedure of SCEGG is presented in Algorithm 1.

4 Experiments

Dataset. We evaluate the proposed SC-EGG on three standard ZSL benchmark datasets: Caltech-UCSD-Birds (CUB) [Welinder et al., 2010], SUN Attribute (SUN) [Patterson and Hays, 2012] and Animals with Attributes2 (AWA2) [Xian et al., 2017]. CUB is a fine-grained dataset which consists of 11,788 images of 200 bird classes. SUN is a fine-grained dataset including 14,340 images from 717 scene classes. AWA2 is a coarse-grained dataset which contains 37,322 images from 50 animal classes.

Evaluation Protocols. During testing, we measure the top1 accuracy both in the CZSL and GZSL tasks following [Xian et al., 2017]. In CZSL, we only predict the unseen classes Yu to compute the accuracy of test samples (denoted as Acc). In GZSL, we compute the accuracy of the test samples from both seen classes Ys (denoted as S), unseen classes Yu (denoted as U) and their harmonic mean H = (2 × S × U)/(S + U).

Implementation Details. We use the training splits proposed in [Xian et al., 2018]. ResNet101 [He et al., 2016] (pre-trained on ImageNet) is used for image feature extraction without fine-tuning. We use the Adam as optimizer with lr = 10-4 and batchsize = 64. We use a single layer FC as the final CZSL or GZSL classifier. Hyperparameters λg, λw, λr, λs and λu are respectively set to 1.0, 10.0, 0.01, 0.1 and

0.1. In addition, the setting of other hyperparameters in TFVAEGAN follows [Narayan et al., 2020]. We train SC-EGG for 20 epochs in stage 1, 20 epochs in stage 2, and at most 200 epochs in stage 3. The code of SC-EGG is available at the online page 1 .

4.1 Comparison with State of the Arts

SC-EGG is an inductive method, so we only compare it with other inductive methods for fair comparisons. Table 1 shows the comparison with state-of-the-art methods on CUB, SUN, and AWA2 both in the CZSL and GZSL settings. In the CZSL setting, our SC-EGG achieves the best accuracies of 69.2% and 78.2% on SUN and AWA2, respectively, and obtains significant gains of over 3.2% and 4.6% when compared with all of other methods. On CUB, SC-EGG still obtains competitive performance with a top-1 accuracy of 75.1%. These are benefitted from the transferable representation of the semantic compression embedding learned by our SC-EGG. In the GZSL setting, the SC-EGG achieves the best performances with harmonic mean (H) of 68.5%, 44.3% and 72.4% on CUB, SUN and AWA2, respectively. Compared with other methods that achieve good performance on either seen classes or unseen classes, our SC-EGG achieves a good balance between seen and unseen classes. This is because SC-EGG is capable of learning the discriminative dense-semantic visual representations that are attribute-level semantic-related. Thus, the quality of visual features in both seen and unseen classes is enhanced.

1 https://github.com/HHHZM/SC-EGG

4.2 Ablation Study

In this section, we provide further insights into SC-EGG by conducting ablation studies to evaluate the effects of LLCE, LGCE, LSCR and LEGS. We also show the baseline of the generative model in SC-EGG (i.e., TF-VAEGAN [Narayan et al., 2020]). Results of the ablation study are shown in Table 2. The results of the baseline, which is without semantic compression embedding, are significantly worse than the full SC-EGG, with the Acc/H drops by 6.8%/8.2% and 2.0%/1.1% on CUB and SUN, respectively. If we incorporate the SCEN without local classification constrain (i.e., SC-EGG w/o LLCE), the model achieves poor ZSL performance against the full SC-EGG with the Acc/H drops by 4.1%/4.3% and 1.4%/1.9% on CUB and SUN respectively. When we remove the global classification constrain of SCEN (i.e., SC-EGG w/o LGCE), the performance of SC-EGG will drop slightly. Moreover, removing the semantic consistent regression loss LSCR results in 2.2%/2.9% and 1.9%/0.9% drops of Acc/H on CUB and SUN. If LEGS is not used, SCEGG inevitably overfits to seen or unseen classes, leading to inferior results. Our full SC-EGG effectively learns the compressed semantic embeddings for discriminative and transferable feature representations, achieving the best results.

4.3 Qualitative Results

Visualization of AttentionMaps. In Figure 3, we show the visualization of attention maps of the SCEN of two random selected samples. The first column shows the input images, the second column shows the global attention maps of GEN, and the other columns are local attention maps of LEN with top-8 attention scores. As shown in Figure 3, local visual features precisely focus on the regions corresponding to the attributes. The global visual feature focuses on the complete instance in the image, avoiding the influence of useless scene information.

t-SNE Visualizations. As shown in Figure 4, we present the t-SNE visualizations [Van der Maaten and Hinton, 2008] of real seen/unseen and synthetic unseen visual features in CNN backbone visual space and dense-semantic visual space. Results show that the visual features in the dense-semantic visual space are more discriminative in real seen, real unseen, and synthetic unseen classes than the CNN backbone ones.

5 Conclusion

In this paper, we propose a novel semantic compression embedding guided generation model (termed as SC-EGG) for ZSL. SC-EGG cascades a semantic compression embedding network (SCEN) and an embedding guided generative network (EGGN). The SCEN learns a discriminative densesemantic visual space by semantic compression embedding, and the EGGN learns an implicit mapping from the classlevel semantic space to the dense-semantic visual space to synthesize high-quality unseen features, under the guidance of SCEN. Extensive experiments on three popular benchmark datasets demonstrate the superiority of SC-EGG for ZSL.

Acknowledgements

This work is partially supported by NSFC (61772220, 62172177, 62006244), Special projects for technological innovation in Hubei Province (2018ACA135), Key R&D Plan of Hubei Province (2020BAB027), Natural Science Foundation of Hubei Province (2021CFB332), Young Elite Scientist Sponsorship Program of China Association for Science and Technology (YESS20200140).