Learning Prototype via Placeholder for Zero-shot Recognition

1 Introduction

Inspired by the human cognitive system, zero-shot learning (ZSL) was proposed to identify unseen classes by utilizing semantic embeddings (e.g., attributes [Lampert et al., 2009] or text descriptions [Reed et al., 2016] ) to transfer knowledge from seen domain to unseen domain. The ZSL can be categorized into conventional and generalized settings according to the different classes that a model sees in the test phase. In conventional ZSL, the test images from the unseen domain will be recognized. And the more challenging generalized ZSL (GZSL) [Xian et al., 2017] aims to predict the test images belonging to both the seen and unseen domains.

∗Corresponding author

Domain shift [Fu et al., 2015] is an intractable problem in ZSL, due to the underlying difference between data distributions of seen domain and unseen domain. Only built on seen domain, the visual-semantic alignment is often distorted and loses desired discrimination especially for semantically similar categories of unseen domain. Early ZSL methods [Bucher et al., 2016] learn projecting visual features into semantic space to improve semantic representative capability of the features and eliminate the domain shift. However, some pioneers argue that taking semantic space as the projection space is less discriminating due to the hubness problem [Radovanovic et al., 2010]. When high-dimensional visual features are mapped to a low-dimensional semantic space, the shrink of feature space would aggravate the hubness problem that some instances in the high-dimensional space become the nearest neighbors of a large number of instances [Liu et al., 2020]. To tackle these problems, CVCZSL[Li et al., 2019] proposes mapping semantic embeddings to visual space and treats the projected results as class prototypes. Though the prototypes of seen classes are highly discriminative, it is still suboptimal due to the lack of unseen classes in the training phase.

Since the deep neural network (DNN) tends to predict with a subset of the most predictive features [Huang et al., 2020], embedding-based methods are limited to learning visualsemantic alignment of seen classes and generally obtain prototypes of seen classes located in a confined space as shown in Figure 1 (left). Being constricted in such a limited space, prototypes of unseen classes can easily lose the discrimination due to the disruption of the domain shift and cannot be well adapted to recognize unseen classes in ZSL setting or distinguish the unseen from seen classes in GZSL setting. As a result, to cope with the domain shift, the prototypes of unseen classes require more space for highly separable arrangements.

In this work, we propose to learn prototypes via placeholders for zero-shot recognition (LPL) to mitigate the domain shift. Figure 1 indicates the motivation of our method. Building upon the idea that the unseen class usually share semantics with several seen classes, e.g., zebra is black and white animals (as gaint panda) with four legs (as tiger) and horse shape body (as horse). LPL utilizes the combination of seen classes to hallucinate both visual and semantic embeddings for new classes scattering among seen classes. As shown in Figure 1 (right), taking hallucinated classes as placeholders for unseen classes, LPL learns highly dispersed prototypes of seen classes. Thus, more space is spared to insert prototypes of unseen classes and tackle the impact of domain shift. Specifically, an effective two-steps hallucination strategy is proposed. First, we blend visual and semantic embeddings of multiple seen classes on a similarity graph respectively and control the hallucination classes to distribute around seen classes without significantly deviating from the original data. Second, to obtain abundant classes as placeholders for unseen classes, we further interpolate between elementary hallucinated classes obtained in the previous step and the original seen classes. To prevent the semantic ambiguity of the hallucinated classes from weakening the effect of placeholders, a semantic-oriented fine-tuning strategy is proposed for preliminary visual-semantic alignment which promotes the feasibility of placeholders.

Our contribution is three-fold: (1) We propose placeholder-based prototype learning for zero-shot recognition (LPL), which hallucinates new classes playing as placeholders of unseen classes and encourages learning wellseparated prototypes for unseen classes recognition. (2) With the proposed semantic-oriented fine-tuning, we prevent generating substandard hallucinated classes that trigger semantic ambiguity, which promotes the faithfulness of the placeholders. (3)With extensive experiments and ablation study on five benchmarks, we demonstrate that the proposed LPL achieves new state-of-the-art ZSL performance.

2 RelatedWorks

2.1 Visual-Semantic Gap

Zero-shot learning (ZSL) transfers knowledge from seen classes to unseen by class semantic embeddings. Visual and semantic are two kinds of modal embeddings located on different manifold structures. Thus, there is typically a gap between visual and semantic domains. Thus the crucial task of ZSL is to learn a visual-semantic alignment. Embeddingbased methods [Xian et al., 2017; Li et al., 2019] learn a common space to bridge the gap. Specifically, CVC-ZSL[Li et al., 2019] thinks visual space has highly precious discriminative power and proposes classifying visual features based on prototypes projected from semantic embeddings. Our work also learns classes prototypes by semantic→visual mapping for classification.

2.2 Projection Domain Shift

The problem of domain shift in ZSL is proposed by [Fu et al., 2015] and known as the projection domain shift. The same attributes may have very different visual appearances in terms of seen and unseen classes of ZSL. Thus, the visualsemantic alignment, i.e., the projection function learned from the seen classes, is often distorted when directly applied to the unseen classes. SAE [Kodirov et al., 2017] takes the encoderdecoder paradigm to enforces the reconstruction constraint on seen classes. However, it is less discriminating to project visual features into semantic space. As the baseline of our work, CVC-ZSL[Li et al., 2019] projects semantic embeddings to visual space and treats the projected results as classes prototypes. Due to the lack of unseen classes, the domain shift still cannot be well managed. Differently, LPL alleviates the domain shift by improving the separability of classlevel prototypes. Dispersed prototypes are learned by reserving placeholders for the unseen classes. Placeholder is implemented by class hallucination which has been explored in [Zhang and Wang, 2021] to mitigate the lack of samples for few-shot detection scenario. In this work, with different motivation and implementation, we leverage class hallucination to play a bridge between seen and unseen classes.

3 Method

3.1 Preliminaries

We denote the seen dataset as S = {xi, yi}|S| i=1 , where xi is an image, yi is its class label in seen classes set Y s and |S| is the number of seen images. Let U = {xui , yu i }|U| i=1 denotes the unseen dataset, in which the class label yu i is in unseen classes set Y u and |U| is the number of unseen images. Y u is disjoint from the seen classes set Y s . Given an image xi, through a visual backbone, the extracted visual feature is denoted as vi ∈ RC. The number of seen classes and unseen classes are denoted as Ls and Lu, respectively. For the common semantic space transferring information between seen and unseen classes, A = {ak}Ls+Lu k=1 is consisted of classlevel semantic attribute vectors, where ak ∈ RD . In ZSL, only the seen dataset S is used during training and the task is to predict the labels of images from unseen classes Y u . For a more realistic and challenging scenario, GZSL is to predict images from both seen and unseen classes, i.e., Y s ∪ Y u .

3.2 Overview

The pipeline of LPL is shown in Figure 2. LPL includes two learnable parts: semantic-oriented fine-tuning (SoF) and placeholder-based prototype learning (PPL). SoF includes the CNN backbone and a visual→semantic mapping network (denoted as visual mapping) which are jointly optimized and aim to provide a preliminary visual-semantic alignment for credible classes hallucination. After semantic-oriented finetuning, the parameters of the backbone are frozen. Based on seen classes, PPL obtains hallucinated classes that play as placeholders of unseen classes. Then the placeholders are leveraged to learn well-separated classes prototypes by mapping from semantic embeddings.

3.3 Placeholder-based Prototype Learning

Besides seen classes, we also hallucinate new classes as input of the semantic mapping network. Playing as placeholders of unseen classes, the hallucination classes prompt dispersed prototypes of seen classes and spare more space for inserting unseen ones. In detail, classes hallucination includes two steps: embedding propagation (EP) and embedding interpolation (EI). EP respectively blends visual/semantic embeddings of multiple classes by implementing propagation on the graph. EI further interpolates the elementary hallucinated visual/semantic embeddings and original ones for more hallucinations, which support placeholder-based prototype learning. We will introduce each operation in the following.

Embedding Propagation. We propose to hallucinate new classes with the visual and corresponding semantic embeddings of seen classes. Thus, embedding propagation is implemented on undirected fully connected graphs Gv and graph Ga corresponding to visual and semantic two spaces respectively as shown in Figure 2. The nodes of the two graphs are respectively visual and corresponding semantic embeddings. The edges represent the propagation weight. Given a batch of data {v, a} = {{v1, a1}, {v2, a2}, · · · , {vk, ak}} from different seen classes, embedding propagation outputs a batch of hallucinated classes {v′ , a′ } = {{v′1 , a′1 }, {v′2 , a′2 }, · · · , {v′k , a′k }} where visual/semantic embeddings are formulated as follows:

where N ′ i is a subset randomly chosen from all neighbor nodes set Ni. The number of chosen classes n is a hyperparameter. Empirically, combining too many neighbor nodes would cause semantic ambiguity and produce less reliable hallucination classes. Considering that visual and semantic embeddings are located in different manifold structures, it is necessary to synchronize the hallucination in visual and semantic spaces. Thus, we apply the same propagation weight wij on two graphs.

To determine the propagation weight, for the visual space, we firstly compute the raw distance dvij = δ(vi, vj) for each pair of visual embeddings. Here δ denotes a distance function, e.g, cosine distance. To guarantee the convex combinations, we apply softmax function to adjust the distance dij ∈ [0, 1] as follow:

where σ is the scaling factor and Ni denotes the set composed of neighbor nodes from the same batch. For semantic embeddings space, there are similar operations and we can get the node distance wa ij in semantic space. To keep the embedding propagation in a synchronized manner in visual and semantic spaces, we harmonize the propagation weight in two spaces as the final propagation weight wij = (wv ij + wa ij )/2.

Embedding Interpolation. By embedding propagation we obtain hallucination classes located around the pivot of multiple classes. Considering that the distribution of unseen classes is unknown, we should generate as diverse placeholders as possible. Based on elementary hallucination classes {v′ , a′ } = {{v′1 , a′1 }, {v′2 , a′2 }, · · · , {v′k , a′k }} in the previous step, it is possible to produce more reliable and diverse new classes as placeholders.

Inspired by manifold mixup [Verma et al., 2019], we propose to interpolate between the elementary hallucination classes data and original seen classes data on the graph as shown in Figure 2. Given a set of original classes data {v, a} = {{v1, a1}, {v2, a2}, · · · , {vk, ak}} and hallucinated classes data {v′ , a′ } = {{v′1 , a′1 }, {v′2 , a′2 }, · · · , {v′k , a′k }} , we mix up them with very little computational effort in both visual and semantic spaces as follows:

where β ∈ [0, 1] is sampled from Beta distribution Beta(α1, α2). Especially, when β = 1 the final hallucination classes are just the original seen classes without any semantic ambiguity. Instead when β = 0 the output classes are just the previous elementary hallucination classes with some semantic ambiguity. On the one hand, using elementary hallucination classes as a pivot, the abundance of placeholders is substantially improved. On the other hand, the ratio factor β of original data adjusts the semantic authenticity of hallucination classes.

Visual Feature Classification. Adopting episodic training fashion, for each batch of data composed of k seen classes, we can obtain corresponding k hallucinated classes {v′′ , a′′ } = {{v′′ 1 , a′′ 1 }, {v′′ 2 , a′′ 2 }, · · · , {v′′ k , a′′ k }} , which play as placeholders of unseen classes and are used for learning visual feature classification. In fact, according to our hallucination strategy, the hallucinated classes are likely to be original seen classes. Following CVC-ZSL [Li et al., 2019], we learn class prototypes by a semantic→visual mapping network denoted as semantic mapping in Figure 2. For hallucination classes, the classification loss is defined as:

where h(·) denotes the semantic→visual mapping function. Also, we measure cosine distance between projected classes prototypes and visual samples.

3.4 Semantic-oriented Fine-tuning

According to our motivation, hallucinated classes play as the placeholders of real unseen classes. Due to the visual and semantic embeddings located in two different manifold spaces, a favorable visual-semantic alignment is absent. Especially, the visual representations generally contain lots of semanticunrelated components and show poor intra-class compactness. As a result, the hallucinated classes easily trigger semantic ambiguity and appear unreliable compared to genuine classes. With substandard hallucination, the desired effect of placeholders is weakened.

To mitigate the above challenges, we propose semanticoriented fine-tuning (SoF) for preliminary visual-semantic alignment. Especially, SoF takes semantic attributes as supervision and fine-tune the backbone followed by a plain visual→semantic mapping network denoted as visual mapping in Figure 2. The mapping network is implemented by a linear full-connected layer. Denote the weight matrix W ∈ RD×C , where D and C respectively denote the dimension of semantic and visual space. Given the input image xi, visual features are extracted by backbone F(·) and then projected to semantic space by the transformation matrix W for classification. The loss is formulated as:

which jointly optimizes the backbone and the visual mapping network.

3.5 Training and Inference

The training of our model includes two stages as described in Section 3.2. The backbone is fine-tuned guided by semantic attributes in the first stage and frozen in the second stage. For the second stage, we finish learning prototypes with the benefit of hallucination classes. It should be noted that the whole training does not use any seen classes visual samples or semantic embeddings. For the inference, we use the learned semantic mapping network to obtain unseen classes prototypes for zero-shot recognition as follow:

For GZSL, to mitigate the bias towards seen classes, we apply calibrated stacking (CS) to reduce seen class scores by a calibration factor δ as follow:

4 Experiment

4.1 Datasets and Evaluation Protocol

We conduct extensive experiments on five benchmark datasets: AWA2 [Xian et al., 2017], CUB [Welinder et al., 2010], SUN [Patterson and Hays, 2012], FLO [Nilsback and Zisserman, 2008], and APY [Farhadi et al., 2009]. For class semantic embeddings, we use manually-labeled attribute vectors for AWA2, CUB, SUN, and APY, and use 1024-dimensional semantic embeddings extracted from text descriptions [Reed et al., 2016] for FLO dataset.

We follow the datasets split and evaluation protocol proposed by [Xian et al., 2017]. Under the conventional ZSL setting, we evaluate the per-class Top-1 accuracy on unseen classes, denoted as T. Under the GZSL setting, we evaluate the Top-1 accuracy on seen classes and unseen classes, respectively denoted as S and U. The holistic performance of GZSL is measured by their harmonic mean: H = (2 × U × S)/(U + S).

4.2 Implementation Details

We adopt ResNet-101 pre-trained on ImageNet as the backbone. For training of SoF, we adopt the SGD optimizer with batch size of 16. During training of semantic mapping network, we use Adam optimizer. We also adopt an episodebased training fashion to sample M categories and N images for each category in a mini-batch. For all datasets, we obtained hyper-parameters by grid search on the validation set [Xian et al., 2017]. We set M = 20 and N = 4 and set hyperparameters σ = 0.2, (α1, α2) = (5, 1). During the inference of GZSL, we use the calibrated stacking for balanced performance. Refer to Appendix A in supplementary file for more details.

4.3 Comparison with SOTA

To further demonstrate the superiority of our model, we compare LPL with recent methods under inductive settings as shown in Table 1. We categorize the compared methods into generative and non-generative methods. Obviously, LPL achieves significant improvement. Especially on CUB dataset, LPL outperforms all the compared methods with a large margin, e.g., 7.4%, 6.0% and 6.3% for ZSL comparative to RGEN [Xie et al., 2020], GEM-ZSL [Liu et al., 2021], CEZSL [Han et al., 2021]. Considering CUB is a fine-grained dataset and requires discriminative prototypes for recognition, the impressive performance sufficiently demonstrates the effectiveness of learning prototypes via placeholders. For AWA2, LPL can also achieve an amazing result, which indicates that our model is applicable to datasets with different granularity. Besides, LPL does not achieve significant gains on SUN and APY datasets. SUN owns a large number of categories and too few samples in each category. Also, the two datasets include much background information which restricts hallucinated placeholders from playing a effective role.

4.4 Ablation Studies

Component Analysis. The ablation results are shown at the bottom of Table 1. Firstly, we reproduce CVC-ZSL, i.e, a plain semantic→visual (S2V) mapping network, as the baseline. To validate the effect of placeholders, we directly hallucinate classes built on raw visual embeddings. It could be seen that consistent improvement is achieved. Then, we finetune the backbone by semantic-oriented fine-tuning (SoF) for refined visual representations. S2V obtains obvious gains especially 4.0% on AWA2 and 9.4% on CUB. Next, built on refined visual representations, we introduce placeholders by two steps. First, we preliminarily hallucinate classes as placeholders by embedding propagation (EP). Then more hallucinated classes are produced by embedding interpolation (EI). We observe obvious improvement on five ZSL benchmarks respectively. Overall, introducing placeholders (EP & EI) contributes to learning discriminating prototypes and boosts the ZSL accuracy by 7.7% (AWA2), 5.5% (CUB), and 8.8% (FLO). The ablation experimental results fully validate the effectiveness of our ideas.

Discussion about SoF. We fine-tune CNN backbone by semantic embeddings to extract semantic-related visual representations. Previous work [Xian et al., 2019; Narayan et al., 2020] also fine-tune backbone by class labels, named vanilla fine-tuning (VF) here. For fair comparison, in Table 2, we evaluate performance of TF-VAEGAN [Narayan et al., 2020], TCN [Jiang et al., 2019], CN [Skorokhodov and Elhoseiny, 2020] with refined visual representations by SoF. Despite showing consistent improvement, LPL still achieves SOTA performance and demonstrates the superiority of semantic-oriented fine-tuning and learning prototypes via placeholders. Besides, when LPL hallucinates with visual representations refined by VF, it shows a slightly decreased performance, which demonstrates the necessity of

，

Figure 5: The t-SNE visualization of raw features (a) and refined features (b). For the latter, less semantic ambiguity is produced for hallucinated classes (with black color).

prior visual-semantic alignment instead of a plain clustering.

4.5 Visualization Analysis

Visualization of Well-separated Prototypes. To showcase the effectiveness of LPL in learning well-separated class prototypes, we calculate cosine similarity between class prototypes and visualize the similarity heat-maps. As shown in Figure 4, (a)-(b) display the similarity between all seen classes prototypes on CUB and (c)-(d) display the similarity between all unseen ones. We could find that the similarity, not only seen but also unseen, is significantly reduced, which means that the prototypes have a more discrete distribution as our declared motivation. The results shown in Figure 3 agree with the quantitative results in Table 1. When more space is spared for the placement of unseen classes prototypes, the decision boundaries built on unseen domain tend to be less ambiguous when facing semantically indistinguishable categories and lead to better classification accuracy.

Visualization of Refined Features by SoF. We visualize the features refined by SoF on CUB dataset in Figure 5. After fine-tuned by SoF, visual features obviously show better intraclass compactness with removing many semantic-unrelated features. Another important point is that semantic ambiguity is prevented as much as possible during hallucinations. As shown in the bottom right of Figure 5 (a)-(b), hallucination classes data (with black color) have much overlap with genuine classes, which leads to semantic ambiguity, while the overlap is greatly avoided when visual embeddings are refined by SoF. More analysis is presented in Appendix B.

4.6 Hyper-parameters Analysis

We evaluate the effect of different numbers of classes for hallucinations. We set number n in {0, 1, 2, 3, 4, 5, 6, 7, 8}. The model is generally robust to the value of n. When n is too small, the performance is limited by the diversity of placeholders. When too large, the performance shows a slight decrease considering too many classes would increase the extent of semantic unreliability compared to authentic categories. In order to balance diversity and faithfulness, we set n = 4 for our all experiments.

We also evaluate the influence of the scaling factor σ. We consider σ in {0.1, 0.2, 0.5, 1.0, 5.0, 10.0}. With the different σ values, the Top-1 accuracy of ZSL and harmonic mean results on CUB datasets change slightly, indicating that our method is robust to the scaling factor. Specially, for all datasets in our work, we set σ = 0.2. Corresponding figures for display and more hyper-parameters analysis are presented in Appendix C.

5 Conclusion

In this paper, we propose an effective ZSL model termed LPL. With hallucination classes playing as placeholders of the unseen domain, the learned prototypes are encouraged to be well-separated. The hallucination method can obtain abundant classes with less cost by embedding propagation and interpolation on graphs composed of visual and semantic embeddings of seen classes. Furthermore, semantic-oriented fine-tuning is adopted to decrease substandard hallucination data. Small to medium-sized datasets are considered in our work, larger data scenarios will be explored in the future.

Acknowledgments

This study is partially supported by National Natural Science Foundation of China (62176016), the National Key R&D Program of China (No.2021YFB2104800, No. 2018YFB2101100 and No.2019YFB2101600). Guizhou Province Science and Technology Project: Research and Demonstration of Sci.& Tech Big Data Mining Technology Based on Knowledge Graph (supported by Qiankehe[2021] General 382).