Hyperbolic Knowledge Transfer with Class Hierarchy for Few-Shot Learning

1 Introduction

Relying on vast quantities of labeled samples, deep Convolutional Neural Network (CNN) has shown remarkable and promising performance on image classification tasks [He et al., 2016]. However, collecting so much labeled data is very time-consuming, laborious, and even unrealistic, on some applications such as cold-start recommendation [Zheng et al., 2021] and medical diagnosis [Prabhu et al., 2019]. To overcome this challenge, Few-Shot Learning (FSL) has been proposed and has become a hot research topic recently. Just like humans learn new concepts through knowledge transfer, FSL

∗Corresponding author

aims to learn transferable knowledge from base classes with abundant labeled samples and then transfers the knowledge to quickly recognize novel classes with few instances.

The core challenge of FSL is learning what kind of knowledge to alleviate the data scarcity issue for recognizing novel classes. To solve the challenge, a large number of FSL methods have been proposed from the perspective of metalearning. Their main idea is constructing a large number of FSL tasks from base classes to learn task-agnostic knowledge (called meta-knowledge), and then leveraging the metaknowledge to quickly learn/infer a classifier for novel classes. The meta-knowledge can be a good initial model [Finn et al., 2017], optimization algorithm [Zhang et al., 2022], embedding network [Snell et al., 2017], or label propagation strategy [Liu et al., 2019b], etc. Though these existing methods has achieved promising performance on FSL, most of them focus on learning implicit knowledge (beyond human comprehension), but ignore the explicit prior knowledge summarized from human past experiences such as class text description, class attributes, or class hierarchy. Such explicit prior knowledge has been proved to be effective on zero-shot learning [Wan et al., 2019]. However, it is not fully explored on FSL, thus the performance improvement is limited.

Recently, several studies attempt to explore the explicit prior knowledge, e.g., class name [Li et al., 2020], description [Schwartz et al., 2019], attributes [Zhang et al., 2021b], and hierarchy [Peng et al., 2019], for further improving FSL performance. Among them, the class hierarchy-based method, proposed in [Peng et al., 2019], delivers more promising FSL performance, due to its good knowledge transfer characteristics. In the method, they focus on the idea of prototype classifier to perform class prediction, i.e., assigning the label of each sample to its nearest class prototypes. In particular, they regard the class hierarchy (i.e., the hierarchy relations between classes) as an undirected graph, and then design a graph convolution-based Knowledge Transfer Network (KTN) to learn to leverage it to infer the class prototypes for novel classes. Such design can fully exploit the class semantics and hierarchy relations for knowledge transfer, such that the FSL performance can obtain significant improvement.

In this paper, we also focus on exploring the class hierarchy for FSL but argue the class hierarchy is not fully explored and utilized in [Peng et al., 2019]. The reasons are: 1) regarding the class hierarchy as an undirected graph is not entirely accurate, which more likes a tree structure actually (see Figure 1a); 2) for such tree-like structure, exploring it in Euclidean space can not obtain the powerful geometrical representations, since the volume of the Euclidean space grows in a linear trend (see Figure 1b) such that the volume can not match the exponential growth of nodes with the tree depth.

To address the drawback, we propose a novel Hyperbolic Knowledge Transfer framework for FSL, namely HyperKT, by exploring the hierarchical relations between classes in hyperbolic space, i.e., the space with constant negative curvature. As shown in Figure 1c, in the hyperbolic space, the volume grows exponentially with respect to its radius, which is analogous to the exponential growth of the number of nodes in a tree with respect to its depth. Such characteristic allows for sufficient room to represent the tree-like structure so that the class hierarchy can be well preserved for the knowledge transfer (see Figure 1d). To this end, in the HyperKT, we first design a hyperbolic representaton learning module to map the features of each image to the hyperbolic space. Then, a hyperbolic prototype inference network is designed to learn to leverage the class semantics and their relations to infer the prototypes for each class/superclass in the hyperbolic space. Besides, a hierarchical classification loss and a relation reconstruction loss is designed to fully leverage and learn the treelike class hierarchy for FSL. Finally, the class prediction can be performed in a nearest-prototype manner for novel classes. Our main contributions can be summarized as follows:

• We propose a novel hyperbolic knowledge transfer framework for FSL by exploring the class hierarchy in hyperbolic space. Resorting to the exponential growth of hyperbolic space volume, the class hierarchy can be explored and utilized with arbitrarily low distortion.
• A hierarchical classification loss and a relation reconstruction loss is carefully designed for FSL, which offer our framework the excellent ability to leverage and learn such tree-like structure of the class hierarchy.
• We conduct comprehensive experiments on three data sets, which verify the effectivenss of our method.

2 RelatedWork

2.1 Few-Shot Learning

Few-shot learning aims to recognize novel classes from very few examples. A mainstream method is meta-learning [Wertheimer et al., 2021; Baik et al., 2021], which focuses on learning task-agnostic knowledge (called meta-knowledge)

from a large number of similar FSL tasks. For example, [Finn et al., 2017] attempts to learn the task-agnostic initial parameters of model, such that the model can be quickly adapted to new FSL tasks via few gradient updates. [Snell et al., 2017; Rizve et al., 2021] regard the task-agnostic metric space or embedding space as meta-knowledge, where the novel classes can be predicted via a simple nearest neighbor classifier with Euclidean or cosine distance. [Rodr´ıguez et al., 2020; Yang et al., 2020] learn a task-agnostic label propagation strategy (i.e., meta-knowledge), such that the labels of unlabeled samples can be predicted in a label propagation manner from few labeled samples to unlabeled samples. From the perspective of meta-knowledge, these methods mainly focus on learning implicit knowledge. Different from them, our method aims at exploring the the explicit knowledge of class hierarchy for improving FSL, which can effectively alleviate the data scarcity issue by feat of human prior knowledge.

2.2 Semantics Knowledge

Semantics knowledge refers to the prior knowledge that human beings derive from past experiences such as text description or knowledge graph. Such type of knowledge have been successfully leveraged in various domains, e.g., zeroshot learning (ZSL) [Wan et al., 2019]. Recently, several FSL techniques relying on semantics knowledge have been proposed [Xing et al., 2019; Zhang et al., 2021b; Peng et al., 2019]. For example, in [Zhang et al., 2021b], the authors explored the priors of visual attributes to complete prototypes for FSL. Zhang et al. [Zhang et al., 2021a] introduce concept graph as FSL prior and improve FSL performance from the perspective of weak-supervised information. In [Peng et al., 2019], the class hierarchy is introduced as priors to enhance prototypes. In this paper, we also focus on exploring the class hierarchy but in hyperbolic space instead of Euclidean space, which provides sufficient room to preserve the tree-like class hierarchy.

2.3 Hyperbolic Geometry

Recently, hyperbolic geometry has received increasing attention due to its powerful representation on complex tree-like structure data. Specifically, [Nickel et al., 2017] first proposes to learn the node hierarchy of a graph in hyperbolic space and showed powerful representation performance. [Tifrea et al., 2018] embeds the words in hyperbolic spaces to learn their hierarchical semantics representations. Besides, some researchers began to apply hyperbolic geometry to various domains, such as ZSL and FSL. For example, [Liu et al., 2020b; Khrulkov et al., 2020] propose to explore image hierarchy for better knowledge transfer on ZSL and FSL, respectively. In [Qi et al., 2021], the oblique manifold space is explored and an oblique distance-based classifier is proposed for FSL. Gao et al. [Gao et al., 2021] propose to learn a task-aware geometrical structure by making use of the hyperbolic geometry for FSL. Different from these existing works, we regard the class hierarchy as a tree-like structure and propose to learn hyperbolic representations and prototypes that capture such tree-like structure, such that better knowledge transfer can be achieved for novel classes.

3 ProblemDefinition and Preliminaries

3.1 ProblemDefinition

For the N-way K-shot FSL problem, three data sets are given: an auxiliary data set Dbase with abundant base classes, a training data set S (called support set) with N novel classes, and a test data set Q (called query set) of novel classes. Let's call the set of base class and novel class as Cbase and Cnovel, respectively. In particular, a large number of labeled samples are available in Dbase, but there are only K labeled samples per novel class in the support set S where K is very small (e.g., K = 1 or 5). In this paper, we introduce a class hierarchy G as FSL prior knowledge, which refers to what kinds of superclass the base/novel class should share, e.g., the "persian cat" and "ragdoll cat" share the same superclass "cat". Our goal is learning a good classifier of novel classes for the query set Q by leveraging the auxiliary data set Dbase, the support set S, and the class hierarchy G.

3.2 Preliminaries

In this paper, we mainly focus on learning transferable representation in the hyperbolic space for FSL, by resorting to the prior knowledge of class hierarchy. Thus, we first review some preliminaries of hyperbolic geometry in this subsection.

Hyperbolic Space. The hyperbolic space is a Riemannian manifold with a constant negative curvature. In this space, there are five isometric model to model embedding space, including the Hyperboloid model, the Klein model, the Hemisphere model, the Poincare ball model, and the Poincare halfspace model. In this paper, we select the Poincare ball to represent our embedding space, due to its abundant operations such as vector addition and subtraction. Formally, the d-dimensional Poincare ball is defined as Dn = {x ∈ Rn | ‖x‖ < 1} where ‖ · ‖ denotes the Euclidean norm. Since the tangent space on Hyperbolic space always is Euclidean, the vector v in Euclidean and Hyperbolic space can transform to each other, which is achieved by using an exponential map expx(v) or logarithmic map log0(v). The exponential map exp0(v) aims to map the Euclidean vector v to the Hyperbolic space and the logarithmic map log0(v) map the Hyperbolic vector v back to the Euclidean space.

Hyperbolic Distance. In the d-dimensional Poincare ball space Dn , the distance between two vectors x ∈ Dn and u ∈ Dn can be defined as the geodesic distance. That is,

where ‖ · ‖ denotes the Euclidean norm.

4 Methodology

Our main idea is that regarding the class hierarchy as a tree-like structure and then leveraging it to learn transferable (i.e., satisfing such tree structure) image representations and class prototypes, such that better knowledge transfer can be achieved for FSL. Next, we elaborate on our HyperKT.

4.1 HyperKT Framework

In this paper, different from the existing FSL method exploring the class hierarchy [Peng et al., 2019; Zhang et al., 2021a], we regard the class hierarchy as a tree-like structure and then present a novel hyperbolic knowledge transfer framework for FSL, namely HyperKT, which explores the class hierarchy in hyperbolic space. As shown in Figure 1d, the intuition of such design is that by using the exponential growth characteristic of hyperbolic space volume, the treelike hierarchy relation of base/novel classes and their superclasses can be well preserved, so that transferable image representations and class prototypes can be achieved for FSL.

As shown in Figure 2, the HyperKT framework consists of four key components, i.e., a CNN-based feature extractor fθf () with parameters θf , an exponential map exp0(), a hyperbolic representation learning module (HRLM) fθr () with parameters θr, and a hyperbolic prototype inference module (HPIM) fθp () with parameters θp. The feature extractor fθf () aims to obtain a good representation for each image in Euclidean space, which can be obtained by following representation learning methods [Rizve et al., 2021]. Note that the feature extractor is freezed in our framework. Then, the exponential map exp0() transforms the Euclidean representation of each image to the hyperbolic space. After that, the HRLM fθr () is employed to learn a hyperbolic representation for each image and the HPIM fθp () accounts for inferring the prototypes for each classes. At last, we perform the class prediction in a nearest-prototype manner. The main details of the HRLM and the HPIM will be elaborated in Sections 4.2 and 4.3. Here, we first introduce the workflow depicted in Figure 2, which consists of a pre-training and a fine-tuning phase.

Pre-Training Phase. In this phase, our goal is pre-training the HRLM fθr () and the HPIM fθp () on all base classes and then transfering it to novel classes for FSL. Based on this class hierarchy G, the main idea is to learn the representations and prototypes satisfing such tree structure from base classes and then apply this meta-knowledge to novel classes to alleviate the data scarcity issue of FSL.

Specifically, as shown in Figure 2, on the one hand, given a sample (x, y) ∈ Dbase from base classes, we first leverage the feature extractor fθf () to represent each image in the Euclidean space. Then, the representation is feeded to an exponential map, followed by the HRLM fθr (), which is transformed to the hyperbolic space. As a result, the hyperbolic representation z can be obtained for each image x. That is,

On the other hand, we consider the class hierarchy G as a tree structure and then feed it into the HPIM fθp () to infer the prototypes pk for each base/novel class or superclass k by leveraging the class semantics and their hierarchy relations. Finally, based on the hyperbolic representation z and prototypes pk, we propose a novel meta-objective to learn HRLM fθr () and HPIM fθp (), consisting of a hierarchical classification loss Lc and a relation reconstruction loss Lr. That is,

where λr is a hyper-parameter trading off the loss Lc and Lr. The Hierarchical Classification Loss Lc aims to learn good represenations and prototypes to classify each sample x ∈ Dbase to the base class y ∈ Cbase and its superclasses. Specifically, based on the hyperbolic representation z and the prototypes pk, the class probability P(k|x, θr, θp) that each image x ∈ Dbase belongs to each base class or superclass can be estimated by calculating their hyperbolic distance. That is,

where γ is a temperature parameter. Then, the hierarchical classification loss Lc is defined as the negative log likelihood loss on base class datasets Dbase. That is,

where λc is a hyper-parameter and τy denotes the set of superclass containing class y. The Reconstruction Loss Lr accounts for guiding the HRLM fθr () and the HPIM fθp () to preserve the hierarchy structure of class hierarchy for improving knowledge transfer of FSL. Specifically, based on the prototypes pk, the edge probability Pk1,k2 of classs k1 and k2 in the class hierarchy G can be calculated with their hyperbolic distance by using the Fermi-Dirac distribution [Liu et al., 2019a]. That is,

where β and α are the hyper-parameters. Following [Liu et al., 2019a], β = 2 and α = 1 are used. Then, the relation reconstruction loss Lr is defined as the negative log likelihood loss on all edges of class hierarchy G. That is,

where V denotes the set of edges in the class hierarchy G.

Fine-Tuning Phase. In the phase, our main goal is finetuning the HRLM fθr () and the HPIM fθp () to adapt to novel classes with few labeled samples. The workflow is similar to the pre-training phase. As shown in Figure 2, the difference is that we replace the base class samples x ∈ Dbase with the support samples x ∈ S of novel class. After fineuning, we directly estimate the novel class probability for each query sample x ∈ Q by following Eqs. 2 and 4 and assign its label with the novel class with highest class probability.

4.2 Hyperbolic Representation LearningModule

In the subsection, we introduce how to design the hyperbolic representation learning module fθr () (HRLM), which is used to transform the Euclidean feature representation into the hyperbolic space. Our main idea is employing a hyperbolic fully connected network [Ganea et al., 2018] to achieve it. Spefifically, the transform at layer k can be expressed as:

where wk and bk denote the parameters of each layer.

4.3 Hyperbolic Prototype Inference Module

The key challenge of FSL is how to learn the class prototype pk when only few labeled samples are available. To address the challenge, we regard the class hierarchy G as inputs and then design a hyperbolic graph convolution-based inference network fθp () to learn the map from class hierarchy G to the class prototype pk. It advantage is the class semantics and hierarchy relation can be fully leveraged for learning prototype.

Specifically, the propagation rule of hyperbolic graph convolution [Chami et al., 2019] at layer k for node m is:

where Wk and bk are the parameters of hyperbolic graph convolution layer, I(m) denotes the set of neighbour nodes of node m, and A′ is the normalized adjacency matrix.

5 Experiments

In this section, we conduct extensive experiments to evaluate the effectiveness of HyperKT on three public datasets.

5.1 Datasets and Settings

miniImagenet. This is a subset of the ImageNet, which contains 100 classes and 600 images per class. The size of each image is 84 × 84. Following the setting of [Peng et al., 2019], we split the data set into 64 classes for training, 16 classes for validation, and 20 classes for test, respectively.

tieredImagenet. This dataset is also derived from the ImageNet dataset. Different from miniImagenet, 1) the data set is a larger dataset, which consists of 608 classes and each class collects about 1200 images with size of 84 × 84; and 2) the dataset is splited by following high-level semantic categories [Gao et al., 2021], i.e., 20 high-level semantic classes for training, 6 high-level semantic classes for validation, and 8 high-level semantic classes for testing, respectively.

CIFAR-FS. This dataset is constructed from CIFAR100, which consists of 100 classes and each class contains about 600 images of size 32×32. Following [Gao et al., 2021], we split the data set into 64 classes, 16 classes, and 20 classes for training, validation, and testing, respectively.

5.2 Implementation Details

Network Details. Following [Rizve et al., 2021], we use ResNet12 as our backbone, which delivers a 640-dim vector to encode each image. For the HRLM, the dimension of its each layer is set to 640, 640 ×4, 640 ×4, and 640, respectively. For the HPIM, the dimension of each layer is set to 300, 640, 640, and 640, respectively. In the HRLM and HPIM, the hyperbolic ReLU is used as the activation function except for the output layer. These hyper-parameters, γ = 1/640, α = 2, β = 1 are used in our all experiments. For hyper-parameters λc and λr, λc = 1 and λr = 2 are used for miniImagenet, λc = 2 and λr = 4 are used for tieredImagenet, and λc = 1 and λr = 2 are used for CIFAR-FS.

Training details. In the pre-training phase, we use the Riemannian Adam optimizer with a weight decay of 0.001 to pre-train the HRLM and HPIM, and train it with 100 epochs, where we set the initial learning rate to 0.00001 and then decay it by 0.1 at epochs 50, 80 and 90, respectively. In the finetuning phase, the HRLM and HPIM are further fine-tuned with 50 epochs for adapting to novel classes. During finetuning, the Riemannian Adam optimizer with a learning rate of 0.00001 and a weight decay of 0.001.

Evaluation. Following the 5-way 1-shot/5-shot setting, we evaluate our HyperKT on 600 randomly sampled episodes from the novel class dataset and report the mean accuracy together with the 95% confidence interval. In each episode, 15 images are randomly sampled as the query set.

5.3 Discussion of Results

In the subsection, we evaluate the performance of our HyperKT on miniImagenet, tieredImagenet, and CIFAR-FS.

Results on miniImagenet and tieredImagenet. Table 1 shows the results of our HyperKT and some state-of-the-art methods on miniImagenet and tieredImagenet. It can be observed that our HyperKT achieves superior and competitive performance over the state-of-the-art methods on 1-shot and 5-shot few-shot image classification tasks, respectively. In particular, the performance improvement is significant on 1shot few-shot image classification tasks, around 1% ∼ 5%, which means that our HyperKT is effective, especially when only very few labeled sampels are available.

Specifically, compared with the methods without using external knowledge, our HyperKT additionally introduce a class hierarchy as prior and attempts to leverage the concept graph to learn transferable image representations and class prototypes on the hyperbolic space for FSL. The experimental results verify the effectiveness of introducing the external knowledge of class hierarchy on FSL. As for these FSL methods using external knowledge, they also introduce the external knowledge as priors. Different from these methods, our method explore the class hierarchy as prior and focuses on utilizing the hierarchy relations between classes to learn transferable image representations and class prototypes for FSL, instead of to fuse modality or learn a feature extractor. It worth noting that our method also outperforms the FSLKT method, which also introduces the class hierarchy as prior and is our key baseline method. Different from FSLKT, we focus on learning transferable image representations and class prototypes for FSL. This is achieved by exploring the class hierarchy in the hyperbolic space instad of Euclidean space, so that the class hierarchy relation can be well preserved for the knowledge transfer. The result validates the superiority of our manner to incorporate the class hierarchy on FSL. Finally, our HyperKT outperforms existing hyperbolic FSL methods by a large margin, around 1% ∼ 5%. This is because we introduce the class hierarchy as prior knowledge and attempt to explore the class hierarchy instead of image hierarchy, which can well enable knowledge transfer for FSL.

Results onCIFAR-FS. Table 2 reports the experimental results on CIFAR-FS dataset. Similar to the results of Table 1, we find that our HyperKT methods also achieve the superior performance over state-of-the-art methods, exceeding these methods by around 1% ∼ 3% classification accuracy on 1shot tasks. This further verifies the effectiveness of our HyperKT, which can effectively boost the FSL performance by exploring the class hierarchy in the hyperbolic space.

5.4 Ablation Study

In this section, we conduct ablation studies on miniImagenet and CIFAR-FS to analyze the effectiveness of all components, which aims to answer the following three questions.

Is the meta-objective defined in Eq. 9 effective? We conduct an ablation study on the 5-way 1-shot/5-shot tasks of the miniImagenet, to verify the effectiveness of the proposed two losses, i.e., a hierarchical classification loss Lc and a relation reconstruction loss Lr. Specifically, 1) we use all losses to train our model, i.e., our HyperKT; 2) we remove the superclass terms of hierarchical classification loss Lc on 1); 3) we remove the reconstruction loss Lr of 1). The results are shown in Table 3. We can see that the FSL performance decreases 1% ∼ 5% when removing the superclass terms of hierarchical classification loss, and reconstruction loss, respectively. This suggests that our meta-objective is effective.

Are the two key components of HRLM fθr () and HPIM fθi () effective? To verify the effectiveness of the two key components of HRLM fθr () and HPIM fθi (), we conduct an ablation study on the 5-way 1-shot/5-shot tasks of the miniImagenet. Specifically, 1) we report the classification performance of HyperKT, which includes the two key components of HRLM and HPIM; 2) we remove the component of HRLM fθr () on 1) and map the Euclidean features to tangent space, which is directly regarded as the hyperbolic representations; and 3) we replace the component of HPIM fθi () on 1) with a hyperbolic fully connect layers. These results are reported in Table 4. From the resluts of 1) ∼ 3), we can see that the

• (b) Prototype visualization.

Figure 3: Performance anaysis and visualization on miniImagenet.

performance of 1) outperforms 2) and 3), by around 1% ∼ 10%. This means that the two key components (i.e., HRLM fθr () and HPIM fθi ()) are beneficial for our HyperKT.

How does the number of support samples affect the performance of HyperKT? In Figure 3a, we analyze the impacts of the number of support samples on the performance of 5-way tasks. The experiments are conducted on miniImagenet by varing the number of support samples from K = 1 to K = 5. We find that our HyperKT outperforms Cosine Classifier and the performance improvement decreases gradually as the number of support sample increases. This may be reasonable since the data scarcity issue is alleviated gradually as the number of support sample increases.

How does our HyperKT work? We visualize the prototypes of all base/novel classes and their superclasses in Figure 3b. Note that these prototypes are obtaine by applying our meta-objective to a 2-dimensions Poincare ball. As shown in Figure 3b, the tree-like class hierarchy relations between classes are well preserved in the Poincare ball space, which is beneficial for improving knowledge transfer of FSL.

6 Conclusion

In this paper, we present a hyperbolic knowledge transfer framework, which effectively explores the class hierarchy in the hyperbolic space for alleviating the data scarify issue of few-shot learning. Furthermore, a hyperbolic classification loss and a relation reconstruction loss are proposed to learn transferable image representations and class prototypes with tree-like class hierarchy. In this way, our framework can be trained to preserve the tree-like structure and further enhance the knowledge transfer for FSL. The experimental results on three datasets demonstrate that our method achieves superior accuracy with previous state-of-the-art FSL methods.

Acknowledgments

This work was supported by the Shenzhen Science and Technology Program under Grant No. JCYJ20180507183823045, Grant No. JCYJ20200109113014456 and Grant No. JCYJ20210324120208022, and National Natural Science Foundation of China under Grant No. 61972111.