Zero-Shot Logit Adjustment

1 Introduction

In recognition tasks, it is challenging when classes for training and test are different, known as Zero-Shot Learning (ZSL) problems. The goal of ZSL is to correctly recognize unseen samples with a classifier trained on seen classes. Bridging between training and test domains counts on the semantic class priors, i.e., the human-annotated attribute [Lampert et al., 2009] or the word embedding [Reed et al., 2016]. Through similarities between the class semantic descriptors, ZSL can transfer knowledge from seen to unseen classes without accessing the unseen class data.

* Equal contribution.

� Corresponding author. This work was supported by the National Natural Science Foundation of China (NSFC) under Grants No. 61872187, No. 62077023 and No. 62072246.

-20

-35

-35 -20

-20

-40

-40 -20

Figure 1: t-SNE visualization of the synthetic-real unseen data (left) and the real train-test seen data (right) in AWA2, we sample the same number for each class pair. The synthetic unseen data shows obvious domain bias from real test unseen data, while the training domain of the seen classes is consistent with the test domain.

Early ZSL works focus on end-to-end classifier learning, particularly the embedding models [Lampert et al., 2013; Li et al., 2019] which match visual features and semantic descriptors in a shared embedding space. These methods appear effective in ZSL but perform poorly in the more challenging Generalized Zero-Shot Learning (GZSL) setting [Chao et al., 2016; Xian et al., 2017]. More recent efforts [Xian et al., 2018; Shen et al., 2020] aim to improve GZSL performance by decomposing the learning process into generator learning and classifier learning. Such a two-stage strategy compensates for the feature expression of unseen classes during the classifier learning by the generated samples. A typical family of studies has focused on the improvement of the generator, exploring alternative architectures, or introducing various inductive biases. On the contrary, there is a scarcity of research on classifiers under the generative paradigm.

A principled classifier design is required to uncap the performance of the two-stage approach. The intuitive notion is introducing semantic information to classifier learning. However, both semantic and visual embedding strategies learn the same knowledge as generative models, i.e., semantic-visual links. It implies that, by directly replacing the classifier, the generated unseen samples can only serve to deliver the generator's learned links to the classifier, with little effect. As a result, classic ZSL classifiers perform worse than vanilla softmax classifiers in the generative setting [Xian et al., 2018].

To determine design direcions for the classifier, we first investigate the distribution of synthetic unseen data. As shown in Figure 1, the sythetic unseen samples deviate severely from the real distribution. This bias is theoretically inevitable since the domain shift problem [Fu et al., 2014], which leads to the misalignment of training and test domains. Nevertheless, we further explore the working mechanism of generated samples in Table 1, finding that the information contained in various synthetic unseen samples is quite homogenous for the classifier. Existing methods employ a large sampling size from generated unseen distribution to magnify the influence of modest valid information included in synthetic unseen samples on feature expression. However, such a resampling strategy has been proven to result in overfitting of the classifier on certain feature patterns (the biased generated unseen distribution in this setting) in other fields [Buda et al., 2018; Menon et al., 2020], which harms the recognition of both seen and unseen classes.

These findings imply that there is a need to restrict the generation number of unseen samples. Can we directly transmit class imbalance information to the classifier during training rather than inefficiently increasing the expression of unseen classes by resampling? In this work, we regard the bias and homogeneity of generated unseen samples as special prior knowledge. In light of the success of loss modification in class imbalance research [Lin et al., 2017; Menon et al., 2020], we incorporate this seen-unseen prior into the classifier training process in the form of logit adjustment. Specifically, we establish the lower bound of the seenunseen balanced accuracy [Xian et al., 2017] with variational Bayesian inference and obtain an adjusted posterior. Then the prior takes part in training via adjusting the logits of vanilla cross-entropy loss. This approach, termed Zero-Shot Logit Adjustment (ZLA), allows for a lower number of generated unseen samples while producing more balanced results. By establishing a semantic-prototype mapping, we further introduce the semantic information to the classifier. Notably, our proposed ZLA allows the generated unseen samples to play an adjustment role rather than supplying unseen class information, which overcomes the embedding-based classifier's previous ineffectiveness in the generative scenario (see Sec. 4.5). Our contributions are summarized as follows:

• We mathematically derive the lower bound of the seenunseen balanced accuracy, allowing us to include generated unseen samples's bias and homogeneity as a seenunseen prior in cross-entropy via logit adjustment.
• Based on ZLA, we break the previous classifier's inconsistency in training objectives and test metrics and the inability to exploit semantic priors in classifier learning.
• Our proposed classifier enables greatly reducing the generation number of unseen samples. It outperforms SoTAs when combined with the base generator, and can be a plug-in to augment various generation-based methods.

2 RelatedWork

2.1 Zero-Shot Learning

Zero-Shot Learning (ZSL) [Lampert et al., 2009; Farhadi et al., 2009] has become a popular research area in recent years. Classic ZSL excludes seen classes during the test, while Generalized Zero-Shot Learning (GZSL) [Chao et al., 2016;

Xian et al., 2017] considers both seen and unseen classes, attracting more current interest. The typical embedding-based ZSL methods [Li et al., 2019; Skorokhodov and Elhoseiny, 2021] learn the semantic-visual links for classification, but with little effectiveness in the GZSL scenario. The progress of GZSL was once driven by the development of generative models [Kingma and Welling, 2013; Goodfellow et al., 2014], which allowed converting the GZSL problem to common supervised classification using a generator-classifier architecture. Until recently, research on generators [Shen et al., 2020; Han et al., 2021] gradually saturated, whereas classifier design is rarely examined in such a two-stage framework. To break the bottleneck of generation-based approaches, the principle design of a classifier is required.

2.2 Posterior Modification

Posterior modification, which has been deeply studied in class imbalance learning [Lin et al., 2017; Menon et al., 2020], aims at producing a class-balanced prediction. Post-hoc correction [Collell et al., 2016], loss re-weighting[Menon et al., 2013], and logit adjustment [Menon et al., 2020] are its representative strategies. A similar procedure has been adopted in certain ZSL research. DCN [Liu et al., 2018] utilizes entropy regularization to calibrate the predictions of seen and unseen classes. The post-hoc correction, known as calibrated stacking [Chao et al., 2016] in ZSL, is also employed. However, a more general strategy in generation-based GZSL is to sample a large number of unseen class samples from the generated distribution. Although the re-sampling technique [Chawla et al., 2002] has been proven to produce overfitting in long-tail learning [Collell et al., 2016], its shortcoming in generation-based GZSL is a lack of exploration. In this paper, we mathematically introduce the more advanced logit adjustment strategy into GZSL for a better balance between seen and unseen predictions.

3 Methodology

Considering two disjoint label sets, Y s and Y u , GZSL aims at recognizing instances that belong to Y s ∪ Y u , while only accessing samples with labels in Y s during training. Define the visual space X ⊆ Rdx and the semantic set A = {ay|y ∈ Y s ∪ Y u } ⊆ Rda , where dx and da are dimensions of these two spaces. Then the goal of GZSL is to learn such a classifier, i.e., fgzsl : X → Y s ∪ Y u given the training set Dtr = {x, y|x ∈ X , y ∈ Y s } and the global semantic set A.

The two-stage framework typically processes this problem with two main components: the generator G and the classifier C . The generator G, defined as an arbitrary generative model [Kingma and Welling, 2013; Goodfellow et al., 2014], is first trained with the seen visual features and their corresponding semantics for conditionally mapping the Gaussian noise to fit the real visual distribution. The instances generated by unseen class semantics and the real seen instances are then fed into the classifier C together to fit the posterior probability:

where ˆx denotes the synthetic instances, z represents random Gaussian noises, and ˜x is either real or synthetic instances. p(y|˜x) denotes the posterior probability derived from the classifier. Then the model predicts the class label by taking Cout = argmaxy(p(y|˜x)). In this work, we focus on the design of the classifier under the GZSL setting.

3.1 Preliminary: Logit Adjustment

Logit adjustment strategy is commonly employed in class imbalance tasks [Lin et al., 2017; Menon et al., 2020], which weights the logit in softmax cross-entropy, i.e.,

where Cy′(·) is the logit corresponding to class y′ , and δ(y, y′ ) represents the adjustment weight. The larger δ(y, y′ ) results in the network focusing more on optimizing the logit of y′ , allowing control of the prediction probabilities of different categories. Existing class imbalance works typically associate the weights with the class prior of y or y′ [Cao et al., 2019; Tan et al., 2020; Menon et al., 2020].

3.2 Empirical Analysis on Generated Samples

Regardless the bias problem (Figure 1) of generated unseen samples, generation-based methods achieve a certain success in GZSL. Thus, we empirically investigate the working mechanism of the generated unseen samples. As shown in Table 1, we compare the single class center point (semantic to visualcenter mapping trained with MSE loss) resampling technique with two generative models, i.e., VAE [Kingma and Welling, 2013] and WGAN [Gulrajani et al., 2017] (detailed in supplementary material). Two phenomena can be intuitively observed by comparing the results in Table 1: (1) the key success of generation-based models in GZSL relies on unseen class feature expression enhancement by a large number of generated unseen samples; and (2) the more diversified samples (generated from generative models) produce a limited performance improvement compared to replicating a single point. We forego a deeper study due to its orthogonal nature to our work, but these modest findings imply that the synthetic unseen samples are rather homogenous than real ones.

Despite the difficulty of eliminating bias and homogeneity, can we include them in classifier training as the seen-unseen prior? Below, we'll illustrate how, by changing the classifier's optimization target, we can integrate this prior into the learning process in the form of the logit adjustment.

3.3 From the Statistical View

The nature of GZSL is an extreme case of class imbalance, as measured by a class balanced metric AH (detailed in Sec. 4.1). Assuming the class space has been completed by a set of pseudo unseen class samples generated by the trained generator, existing classifiers optimize the global accuracy AG by modeling the base posterior probability (Eq. 1):

where p(x) is defined as a uniform distribution over all data, yx is the true label of x, and q(Cout = y|x) is the probability to predict class y with the classifier C. However, Eq. 3 neglects the imbalance between seen and unseen domains, which is inconsistent with the test metric AH . To find balanced results across classes, we in turn employ evaluation indicators to guide the design of the classifier. Indeed, let A(y) denote the accuracy in class y, we have [Collell et al., 2016]

where p(y) represents the statistics frequency of class y, and p(y|x) denotes the real posterior probability. Then the average accuracy of seen classes is

where p(y|y ∈ Y s ) denotes the frequency of class y in Y s , and p(Y s ) is a theoretically derived data-independent probability which will be served as a hyperparameter. Analogously, the average accuracy of unseen classes is

Then we consider the harmonic mean, AH , which serves the target of attaining high accuracy for both seen and unseen classes, and empirically reaches its maximum when the accuracy of seen and unseen classes is balanced [Xian et al., 2017]. We have

With the convexity of the inversely proportional function, we have the upper bound of 1/AS with Jensen Inequality:

Analogously deriving AU , we get a lower bound of AH :

where Y is Y s (Y u ) when y belongs to seen (unseen) classes. To simplify the symbols, we designate |Y s ∪Y u |p(Y)/|Y| as p0(Y) in the following.

Despite the difficulty in determining the Bayesian optimal of AH , maximizing its lower bound achieves an approximate effect, which is equivalent to minimizing its reciprocal. Intuitively, the denominator term of Eq. 9 is minimized if

for each x in p(x). So, given a datum (x, yx), we change the modeling objective in Eq. 1 to the adjusted posterior probability, i.e.,

where p0(Y) refers to the seen-unseen prior (Sec. 3.2) which reflects the bias and homogeneity of pseudo unseen samples. Eq. 11 theoretically gives a more balanced predicted probability distribution than the base posterior, as shown in Figure 2. Next, we will show the estimation of the adjusted posterior.

3.4 ZLA-Based Classifier

AdjustedCross-Entropy. The base posterior probability in Eq. 1 is typically estimated with the cross-entropy loss, i.e.,

Referring to researches on class imbalance [Tan et al., 2020; Menon et al., 2020], we directly model Eq. 11 with C(·) and the posterior becomes

This enables integrating the conditional class prior p(y|y ∈ Y) and the seen-unseen prior p0(Y) into the softmax crossentropy in a logit adjustment manner. Then the weights in Eq. 2 are replaced with

The final adjusted cross-entropy is

In contrast to the standard cross-entropy form, we consider the specific prior information in the generation-based GZSL setting, which contributes to the balancing results across classes. In practice, we make p0(Y s ) much bigger than p0(Y u ). This is intuitively explainable from two perspectives: first, small p0(Y u )/p0(Y s ) promotes seen samples to focus on learning decision boundaries between seen classes; and second, large p0(Y s )/p0(Y u ) encourages large prediction probabilities for unseen classes, which serves the same purpose as a large generation number of unseen samples [Xian et al., 2018; Han et al., 2021].

Semantic Prior Inclusion. Embedding-based methods [Li et al., 2019; Skorokhodov and Elhoseiny, 2021] work by learning a semantic-visual direct link. In this case, an extra generator is hard to aid in embedder learning (see Sec. 4.5 for experimental results) because the overlap between the knowledge (i.e., semantic-visual link) learned by the generator and the embedder results in semantic information crucial for knowledge transfer not being fully exploited. The proposed ZLA, in contrast, allows for supporting the learning of the semantic-based classifier through an adjustment mechanism. No longer teaching the net the semantic-visual link of unseen classes, the pseudo unseen samples adjust the decision boundaries between seen and unseen classes by weighting the logits, avoiding knowledge overlapping to an extent. Specifically, we adopt a prototype learner P [Li et al., 2019; Skorokhodov and Elhoseiny, 2021] to directly map semantics to visual prototypes, and then the adjusted posterior probability of a datum x is estimated through cosine similarity, i.e.,

where τ is the temperature [Hinton et al., 2015]. In the test phase, the prediction class y∗ corresponds to the prototype that achieves the maximum similarity.

4 Experiments

4.1 Datasets andMetrics

Benchmark Datasets. We study GZSL performed in Animals with Attributes 2 (AWA2) [Lampert et al., 2013], Attribute Pascal and Yahoo (APY) [Farhadi et al., 2009], Caltech-UCSD Birds-200-2011 (CUB) [Wah et al., 2011], and SUN Attribute (SUN) [Patterson and Hays, 2012], following the common split (version 2) proposed in [Xian et al., 2017]. AWA2 includes 50 animal species and 85 attribute annotations, accounting 37,322 samples. APY contains 32 classes of 15,339 samples and 64 attributes. CUB consists of 11,788 samples with 200 bird species, annotated by 312 attributes. SUN carries 14,340 images from 717 different scenario-style with 102 attributes.

Visual Representations and Semantic Descriptors. We follow [Xian et al., 2017] to represent images as the 2048-D ResNet-101 [He et al., 2016] features. Moreover, we regard the artificial attribute annotations that come with the datasets as the semantic descriptors of AWA2, APY, and SUN, and the AS

#Accuracy(%)

Table 2: GZSL performance comparisons with state-of-the-art methods. AU and AS denote the per-class accuracy (%) on unseen and seen classes, respectively, and AH is their harmonic mean. The best results are bolded, and the underlines indicate the second-place results. † and ‡ represent whether a generator is employed to obtain the pseudo unseen samples, respectively (‡ indicates yes, and † is the opposite). ZLAP is the proposed zero-shot logit adjustment prototype learner. 1 5 10 20 50 100 15 30 Num AU AH Figure 1: figure name

1024-dimensional character-based CNN-RNN features [Reed et al., 2016] generated from textual descriptions as the semantics of CUB.

Evaluation Protocol. We calculate the average per-class top-1 accuracy for unseen and seen classes, respectively, denoted as AU and AS . Then the metric AH for GZSL is represented as their harmonic mean.[Xian et al., 2017].

4.2 Implementation Details

Since our work focuses on studying a plug-in classifier, we test it on WGAN [Gulrajani et al., 2017], with the same generator and discriminator structures as [Xian et al., 2018]. Our prototype learner P consists of a multi-layer perceptron (MLP) with a single 1024-D hidden layer activated by LeakyReLU and no activation function in the output. The Adam optimizer is employed with a learning rate of 1×10-3 , and the batch size is set at 512 for evaluating our design. When plugging into CE-GZSL [Han et al., 2021], we employ a batch size of 256 in SUN and 512 in other datasets instead of the default 4096 (due to device limitations) while maintaining all other settings in the published paper.

4.3 Comparison with State-of-the-Arts

We apply the proposed classifier to the vanilla WGAN (compare to f-CLSWGAN [Xian et al., 2018]) and the more advanced CE-GZSL [Han et al., 2021] to show its effect and compatibility in different generative frameworks. The baseline results are obtained from the official codes. As shown in Table 2, the combination of the proposed classifier and the basic generative framework (WGAN) outperforms SoTAs, demonstrating the excellent class balancing capability of ZLA. We note that the performance gain provided by our module is not as significant in the highly fine-grained SUN dataset as it is in other datasets. It is mainly due to the minor bias problem of generated unseen samples in the case of multiple classes and modest feature variations that it takes the limited strengths of our approach. Moreover, our classifier improves the performance of both f-CLSWGAN and CE-

1 GZSL, even though CE-GZSL has already produced decent results, proving its plug-in ability in two-stage frameworks.

4.4 Hyperparameters

2 Three key factors are involved in our work, i.e., the generation number (per unseen class) Ng, p0(Y) in Eq. 11, and the temperature τ in Eq. 16. Following [Skorokhodov and Elhoseiny, 2021], τ is fixed at 0.04 for all experiments, since it has a slight bearing on our study. We first examine p0(Y) in the form of the ratio of p0(Y s ) to p0(Y u ), which more intuitively reflects the seen-unseen dichotomy. The ratio is denoted as σ, and its effect is plotted in Figure 3 (left), where a reverse variation of seen and unseen accuracy can be observed, demonstrating ZLA's capacity to moderate the prediction between classes. Figure 3 (right) depicts the influence carried by Ng. Although Ng also posses the ability to affect the accuracy, a large generation number lowers the performance cap (72.8 with 10 generated vs. 69.7 with 1000 generated), indicating the harm of re-sampling the biased samples. In the major experiment (Table 2), we generate 10 samples per unseen class for all datasets to contrast with f-CLSWGAN [Xian et al., 2018], and the best results are obtained when σ is set to 1000, 30, 60, and 300 for AWA2, CUB, SUN, and APY, respectively (better results are possible by trading off between Ng and σ). In comparison to CE-GZSL [Han et al., 2021], we keep the published generated number and take the value of σ as 28, 9, 7, and 920 for the above datasets.

4.5 Ablation Study

In this section, we perform ablation studies to validate our design and display the key elements in our implementation.

Function of ZLA. Figure 3 (right) shows the function of ZLA, where log σ = 0 means p0(Y s ) = p0(Y u ), i.e., without major adjustments. The effect of ZLA is reflected in the comparison of different σ values. Intuitively, an adequate σ value produces considerable performance gains in varying generation numbers, especially when the number is minimal. Furthermore, ZLA enables the prototype learner to be effective in generative scenarios, as illustrated in Table 4 (second lines in two baselines reflect previous ineffectiveness), which further reduces the reliance on the generation number and thereby resolving the previously discussed bias problem.

Beyond the Standard Prototype Learner. Existing prototype leaners [Li et al., 2019; Skorokhodov and Elhoseiny, 2021] are simply established on the semantic-visual links of seen classes, generalizing to unseen classes based on semantic similarities. In this paper, we find the last ReLU layer is crucial to these approaches' hitherto unseen class performances. To investigate the effect of the ReLU function, we compare the latest pure prototype learner [Skorokhodov and Elhoseiny, 2021] with the WGAN-based ZLAP in Table 3. When the ReLU layer is removed, as shown in Table 3, the seen accuracy improves in both baselines, whereas the unseen accuracy decreases if pseudo unseen samples are not accessible. Zeroing the negative output layer values intuitively affects (seen class) prototype expression. However, it provides a regularization which narrows the gap between the unseen class prototypes and the instances when (pseudo) unseen instances are unavailable in training, since the visual feature components are likewise larger than or equal to zero [Xian et al., 2017]. In this sense, our proposed ZLA allows the model to remove the ReLU layer by adjusting the unseen class expression using the synthetic unseen instances, resulting in a win-win situation for both seen and unseen class accuracy.

Beyond the Vanilla Softmax Classifier. In Table 4, we compare the prototype-based classifier to the vanilla softmax classifier to validate its necessity. Results reveal that the prototype learner beats the vanilla softmax classifier thoroughly when ZLA is employed. The explanations are as follows:

(1) the prototype learner enables further regularization on unseen class prototypes by learning semantic-prototype relations with real-world data, whereas the vanilla softmax classifier learns to distinguish unseen classes solely by generated samples; and (2) classifier weights mapped from semantics focus more on category-distinctive information, i.e., semantic information. Specifically, in coarse-grained datasets like AWA2, the generated unseen class samples meet a more serious bias problem, causing the classifier to incorrectly detect unseen classes. The prototype learner, on the other hand, can provide supplemental information to unseen class weights by comprehending the semantic-visual links in seen classes. In fine-grained datasets such as CUB, samples from different classes are relatively close together, making it challenging to separate them correctly. The semantics-based classifier, which contains intrinsically category-distinctive information, aids in increased discrimination.

4.6 Time Complexity Analysis

We note that some recent generation-based methods [Han et al., 2021; Chen et al., 2021b] also mine the semantic discriminant information of samples. However, these methods are typically built on class contrast during generator training, which yields a time complexity of O(N|Y s |) in the training phase (N is the data size). In comparison, the proposed classifier combined with the vanilla WGAN achieves a comparable result with O(N) time complexity. Moreover, the proposed method allows for a much smaller (10 vs. 4000 in AWA2) synthetic number in the classifier training phase, resulting in further time savings.

5 Conclusion

In this work, we theoretically include the logit adjustment tech in the generation-based GZSL. We begin by examining the bias and homogeneity of the generated unseen samples, which build the seen-unseen prior. Then we derive an adjusted posterior from the seen-unseen balanced metric, which enables integrating the seen-unseen prior into the original cross-entropy via logit adjustment. Considering the zero-shot setting, we call our approach Zero-Shot Logit Adjustment. Based on ZLA, we inject the semantic information into the classifier, which always fails in existing two-stage methods.

Our work explores the underutilized potential of the generation-based GZSL by breaking the previous inconsistency between the classifier's training objectives and testing metrics. This approach allows for greatly fewer generated unseen samples, achieving SoTA with little time consumption.