Few-shot Classification via Ensemble Learning with Multi-Order Statistics

1 Introduction

Few-shot Classification (FSC) is a promising direction in alleviating the labeling cost and bridging the gap between human intelligence and machine models. It aims to accurately differentiate novel classes with only a few labeled training samples. Due to limited supervision from novel classes, an extra base set with abundant labeled samples is often used to improve the classification performance. According to the adopted training paradigms, FSC methods can be roughly divided into meta-learning-based [Finn et al., 2017; Snell et al., 2017] and transfer-learning-based [Chen et al., 2019; Liu et al., 2020; Afrasiyabi et al., 2020]. The first type takes the form of episodic training, in which subsets of data are sampled from the base set to imitate the meta-test setting. Since sampling does not cover all combinations, this paradigm cannot fully utilize the information provided by the base set. In contrast, the transfer-learning takes the base set as a whole, so it avoids the drawback of meta-learning and achieves better performance. Many effective regularization techniques have been exploited in transfer-learning, for example, manifold mixup [Mangla et al., 2020], self-distillation [Tian et al., 2020], and self-supervised learning [Zhang et al., 2020b], which leads to significant improvement on the generalization of image representations and the FSC performance.

Ensemble learning combines multiple learners to solve the same problem and exhibits better generalization performance than any individual learners. When combining ensemble learning with deep Convolutional Neural Networks (CNN), the new paradigm usually requires large-scale training data for classification tasks [Horv´ath et al., 2021; Agarwal et al., 2021], making it challenging to be adopted for FSC. Recently, two notable studies [Dvornik et al., 2019; Bendou et al., 2022] employed an ensemble of deep neural networks for FSC tasks under either a meta-learning or a transfer-learning setting. They demonstrated that ensemble learning is also applicable to FSC. Yet, these works are still preliminary and lack a theoretical analysis to explain the underlying reason behind the promising performance. To address this challenge, we provide an FSC ensemble learning theorem for the transfer-learning regime. Its core idea is a tighter expected error bound on the novel classes, in which the expected error on the novel classes can be reduced by implementing ensemble learning on the base classes, given the base classes-novel classes domain divergence.

The generalization ability of ensemble learning is strongly dependent on generating diverse individuals [Yang et al., 2013]. As shown in Figure 1 (a), traditional methods often use different backbone networks as individuals, which significantly increases the computation and storage costs. Our work finds that different-order statistics of the CNN features are complementary to each other, and integrating them can better model the whole feature distribution. Based on this observation, we develop a parameter-free ensemble method, which takes the same backbone and equips different branches with multi-order statistics as learning individuals. We name this method Ensemble Learning with Multi-Order Statistics (ELMOS), as shown in Figure 1 (b). The main contributions of this paper are summarized as follows:

• To our knowledge, this is the first theoretical analysis to guide ensemble learning in FSC. The derived theorem proves a tighter expected error bound is available on novel classes.
• We propose an ensemble learning method by adding multiple branches at the end of the backbone networks, which can significantly reduce the computation time of the training stage for FSC.
• This is the first time that multi-order statistics is introduced to generate different individuals in ensemble learning.
• We conduct extensive experiments to validate the effectiveness of our method on multiple FSC benchmarks.

2 RelatedWork

2.1 Few-shot Classification

According to how the base set is used, FSC methods can be roughly categorized into two groups, meta-learningbased [Bertinetto et al., 2019; Zhang et al., 2020a] and transfer-learning-based [Chen et al., 2019; Liu et al., 2020]. Meta-learning creates a set of episodes to simulate the real FSC test scenarios and simultaneously accumulate metaknowledge for fast adaptation. Typical meta-knowledge includes optimization factors such as initialization parameters [Finn et al., 2017] and task-agnostic comparing ingredients of feature embedding and metric [Snell et al., 2017; Wertheimer et al., 2021]. Recent literature on transfer learning [Tian et al., 2020; Chen et al., 2019] questioned the efficiency of the episodic training in meta-learning, and alternatively used all base samples to learn an off-the-shelf feature extractor and rebuilt a classifier for novel classes. Feature representations play an important role in this regime [Tian et al., 2020]. To this end, regularization techniques such as negative-margin softmax loss and manifold mixup [Liu et al., 2020; Mangla et al., 2020] have been adopted to enhance the generalization ability of cross-entropy loss. Moreover, self-supervised [Zhang et al., 2020b; Rajasegaran et al., 2020] and self-distillation [Ma et al., 2019; Zhou et al., 2021] methods have also shown promising performance in transferlearning. To this end, supervised learning tasks can be assisted by several self-supervised proxy tasks such as rotation prediction and instance discrimination [Zhang et al., 2020b], or by adding an auxiliary task of generating features during the pre-training [Xu et al., 2021b]. When knowledge distillation is adopted, a high-quality backbone network can be evolved through multiple generations by a born-again strategy [Rajasegaran et al., 2020]. All these methods suggest the importance of obtaining generalization representations, and we will leverage ensemble learning to achieve this goal.

2.2 Ensemble Learning

Ensemble learning builds several different individual learners based on the same training data and then combines them to improve the generalization ability of the learning system over any single learner. This learning scheme has shown promising performance on traditional classification tasks with deep learning on large-scale labeled datasets [Horv´ath et al., 2021; Agarwal et al., 2021]. Recently, ensemble learning for FSC methods has been presented. For example, [Dvornik et al., 2019] combined an ensemble of prototypical networks through deep mutual learning under a meta-learning setting. [Bendou et al., 2022] reduced the capacity of each backbone in the ensemble and pre-trained them one by one with the same routine. However, the size of the ensemble learner increased for inference in the former work, while the latter required extra time to pre-train many learning individuals. Therefore, it still lacks efficient designs for learning individuals in FSC ensemble learning. Moreover, these works did not involve any theoretical analysis of the underlying mechanism of ensemble learning in FSC. In this paper, we investigate why ensemble learning works well in FSC under the transfer-learning setting. Based on the analysis, we propose an efficient learning method using a shared backbone network with multiple branches to generate learning individuals.

2.3 Pooling

Convolutional neural network models progressively learn high-level features through multiple convolution layers. A pooling layer is often added at the end of the network to output the final feature representation. To this end, Global Average Pooling (GAP) is the most popular option, however, it cannot fully exploit the merits of convolutional features because it only calculates the 1st -order feature statistics. Global Covariance Pooling (GCP) such as DeepO2P [Ionescu et al., 2015] explores the 2nd -order statistic by normalizing the covariance matrix of the convolutional features, which has achieved impressive performance gains over the classical GAP in various computer vision tasks. Further research shows that using richer statistics may lead to further possible improvement. For example, Kernel Pooling [Cui et al., 2017] generates high-order feature representations in a compact form. However, a certain order statistic can only describe partial characteristics of the feature vector from the view of the characteristic function of random variables. For example, the first- and second-order statistics can completely represent their statistical characteristic only for the Gaussian distribution. Therefore, higher-order statistics are still needed for the non-Gaussian distributions, which are more ubiquitous in many real-world applications. This motivates us to calculate multi-order statistics to retain more information on features.

3 The ProposedMethod

Here we present the proposed method. We start with a formal definition of FSC, and then present a theorem on FSC ensemble learning. This theorem leads to the development of an ensemble learning approach with multi-order statistics.

3.1 Theory Foundation

Under the standard setting of few-shot classification, three sets of data with disjoint labels are available, i.e., the base set Sb, the validation set Sval and the novel set Sn. In the context of transfer-learning, Sb is used for pre-training a model to well classify the novel classes in Sn, with the hyperparameters tuned on Sval. Let Sb = {(xi, yi)}Nb i=1 denotes the source domain with Nb labelled samples and Sn denotes the target domain labelled with K samples in each episode, where Nb >> K. Let the label function of Sb and Sn be fb and fn, respectively. During the pre-training, a learner h is obtained to approximate the optimal mapping function h∗ based on all Nb training samples in Sb from all possible hypotheses H. When ensemble learning is introduced into the pre-training, several learners denoted as {ho}Oo=1 can be obtained. With the ensemble technique of weighted averaging, the final learner h is produced as:

where αo is the weight parameter. There is a domain shift between the base and novel classes [Tseng et al., 2020], and we use the L1 distance [Kifer et al., 2004] to measure the domain divergence between Sb and Sn:

where ηb(x) and ηn(x) is the density functions of Sb and Sn respectively.

Theorem 1 (FSC Ensemble Learning) Let H be a hypothesis space, for any h ∈ {ho}Oo=1 ∈ H is learned from Sb, and h = ∑Oo=1 αoho ∈ H, the expected error on Sn respectively with h and h holds the following relationship:

where λ = EX∈Sb |fn(x) - fb(x)| is a constant, en(h) is the expected error on Sn with h, eb(h) is the expected error on Sb with h, eb(h) is the expected error on Sb with h.

The proof is provided in the Supplementary Material.

Remark 1 The core idea of Theorem 1 is to define a tighter expected error bound on the novel classes with the learned mapping function in the form of ensemble learning during the pre-training. Theorem 1 tells that the true error on the novel classes can be reduced by implementing ensemble learning on the base classes, given the domain divergence between the novel class and base class. This can well explain the effectiveness of ensemble learning in few-shot classification, in which multiple learners are assembled to enhance the generalization on the base set, resulting in better performance in novel classes.

3.2 FSC via Ensemble Learning withMulti-order Statistics

Overview

Our method employs the transfer-learning paradigm in a twophase manner. In the first phase, a good feature extractor is pre-trained on the base set. In the second phase, FSC evaluation is done on the novel set with the pre-trained feature extractor. Following Theorem 1, we introduce ensemble learning in the first phase to improve the FSC performance. The key to this phase is to effectively train multiple diverse individuals. Different from the previous works [Dvornik et al., 2019; Bendou et al., 2022] that use many different networks as individuals, we add multiple branches after the backbone network to create individuals for reducing training costs. Each branch calculates different-order statistics for pooling to highlight the discrepancy between the individuals. This step is optimized by supervised losses. After pretraining, features from different branches are concatenated for FSC evaluation. We name this method as Ensemble Learning with multi-Order Statistics (ELMOS) for FSC. An overview of ELMOS is shown in Figure 2, and a flow description of ELMOS is given in Algorithm 1.

Pre-training via Multi-order Statistics

The proposed model architecture mainly consists of the following four components: an image processing module, the backbone network, a multi-order statistics module, and a supervised classifier module. The image processing module is denoted as M (·), which performs transformation of multiscale rotation to augment the original base set and their label space. The backbone network is denoted as Bθ (·) and parameterized by θ, which converts each image into a tensor of size H ×W × d. The multi-order statistics module module is denoted as S (·), which maps the tensor from the backbone into multiple feature representations to generate individual learners for ensemble learning. The supervised classifier module is composed of softmax classifiers LW (·) and the projectors LU (·) with parameter matrices W and U , respectively, which are used to build the supervised losses for pre-training.

Given L samples be randomly sampled from Sb with Cb classes, in which an image and its corresponding label are denoted as (xi, yi), yi ∈ {1, 2, ...Cb}. M (·) scales the images with the aspect-ratio of 2:3 and rotates the images with {0◦ , 90◦ , 180◦ , 270◦ } under both the new and the original scales, resulting in eight times expansion of training samples. Feed xi into Bθ to produce a tensor feature of Ti = Bθ(xi) ∈ RH×W×d . Next, we reshape the tensor Ti into the matrix Ti ∈ RHW×d , and view each row vector in the matrix tj ∈ Rd as an observation of the random variable of t ∈ Rd . When d = 1, the first characteristic function of variable t in the Laplace operator is given by:

where f(t) and F(t) are the density function and distribution function of t, respectively. Let ψ(s) = lnϕ(s) be the second characteristic function of the random variable t.

Theorem 2 (The Inversion Formula for Distributions)

Let t be a random variable with distribution function F(t) and characteristic function ϕ(s). For a, b ∈ C(F) and a < b,

Corollary 1 (Uniqueness) If two distributions of F1(t) and F2(t) are identical, then the corresponding characteristic functions ψ1(s) and ψ2(s) are identical.

See proof of Theorem 2 and Corollary 1 in [Shiryaev, 2016]. From Theorem 2 and Corollary 1, we can see that there is a one-to-one correspondence between the characteristic function and the probability density function such that the characteristic function can completely describe a random variable.

The oth -order cumulant of the random variable t is defined as the oth derivative of function ψ(s) at the origin, which is:

Then the Taylor series expansion of function ψ(s) at the origin with respect to s yields:

where Rs(so ) is the remainder term. It can be seen from Equation (5) that the oth -order cumulant of t is the coefficient of the term so in Equation (5).

Proposition 1 Consider a Gaussion distribution f(t) with mean µ and variance Σ2 for the random variable t, its second characteristic function is:

Consequently, the cumulant of the random variable t are:

Remark 2 Proposition 1 implies that for Gaussian signals only, the cumulants are identically zero when the order is greater than 2. Please note this conclusion can be naturally extended to the scenario of multivariate variables when d > 1. For the random variables with Gaussian distribution, the first and second-order statistics can completely represent their statistical characteristics. However, the non-Gaussian signals are more common in real-world applications. In this case, higher-order statistics also contain a lot of useful information. Therefore, we propose a multi-order statistics module consisting of multiple branches, each equipped with different order statistics of the tensor feature Ti.

In particular, we employ three branches in the multi-order statistics module, which respectively calculate three orders cumulants of the variable t with the observations in Ti. The specific formulation of the 1st -order, 2nd -order and 3rd -order cumulants of t are expressed as:

As ci2 and ci3 are d × d matrices, we flatten them into d2 - dimensional vectors and finally get the feature representations of zi1, zi2 and zi3. We use these three features as individuals in ensemble learning, which respectively pass through their corresponding softmax classifier LW (·) and projectors LU (·). So the o-th (o = 1, 2, 3) outputs are:

where LWo(·) is the o-th softmax classifier with the parameter matrix of Wo, woj is the j-th component of Wo. LUo(·) is the o-th projector with the parameter matrix Uo. P o ij is the j-th component of the output probability from the o-th softmax classifier. uio is the output vector from the o-th projector. We simultaneously employ Classification-Based (CB) loss of cross-entropy and Similarity-Based (SB) loss of supervised contrastive in supervised learning for each individual [Scott et al., 2021]. These two losses are formulated as:

where yij is the j-th component of label yi, τ is a scalar temperature parameter. Q(uio) is the positive sample set, in which each sample has the same label as uio. uqo is the q-th sample in Q(uio). Then the learning objective function for the o-th individual is:

The overall loss function with ensemble learning is:

where αo is a weight controlling the contribution of each individual in the ensemble learning. The pre-training adopts the gradient descent method to optimize the above loss function.

Few-shot Evaluation

The phase of few-shot evaluation still needs to construct a set of N-way K-shot FSC tasks, with a support set and a query set in each task. The support set randomly selects K samples from each of the N classes that are sampled from Sn, which is denoted as Sp = {xs, ys}NK s=1 , where (xs, ys) is the s-th images and its corresponding label. The query set consists of the remaining images in these N classes, which is denoted as Sq = {xq}Qq=1 with any image of xq. After pre-training, we get rid of the softmax classifier LW(·) and projectors LU(·) and fix the backbone network Bθ(·) and the multi-order statistics module module S(·). The support set Sp is input into Bθ(·) and S(·) to produce the output features:

Algorithm 1: Ensemble Learning with multi-Order Statistics (ELMOS) for FSC

Input: Base set Sb, support set Sp, query set Sq; augmentation module M (·), backbone S(·), softmax classifier LWo, projector LUo and logistic regression gξ (·); temperature

network Bθ (·), multi-order statistics module parameter τ , weight αo (o = 1, 2, 3). Output: Final prediction of the query samples Stage 1: Pre-training with ensemble learning for numbers of training epochs do Sample a mini-batch with any image of {xi, yi}; Feed xi into T (·) and Bθ (·) to obtain feature map Ti ∈ RH×W×d ; Pass Ti through S(·) to output features zio, (o = 1, 2, 3); Pass zio through LWo and LUo to get the output probability and projection feature ; Calculate optimization loss for each individual via Equation (9); Calculate overall loss for pre-training via Equation (10); Update the parameters of θ, Wo, Uo using SGD; end Stage 2: Few-shot evaluation for all iteration = 1, 2, ..., MaxIteration do Feed xs ∈ Sp into Bθ(·) and S(·) to output feature zso, (o = 1, 2, 3); Concatenate zso into the feature zs to train the classifier of gξ (·); end

Classify the query samples according to Equation (13).

where ◦ is the stack operator. The features zs1, zs2, zs3 are concatenated into a final expression of xs:

where con(·) is the concatenated operator. A logistic regression classifier gξ (·) parameterized by ξ is then trained with zs and its corresponding label ys. The query image xq is finally classified as:

where ˆ yq is the inference label value of xq.

4 Experiments

4.1 Datasets

miniImageNet contains 600 images over 100 classes, which are divided into 64, 16 and 20 respectively for base, validation and novel sets. tiredImageNet consists of 779, 165 images belonging to 608 classes, which are further grouped into 34 higher-level categories with 10 to 30 classes per category. These categories are partitioned into 20 categories (351 classes), 6 categories (97 classes) and 8 categories (160 classes) respectively for base, validation and novel sets. CIFAR-FS is derived from CIFAR100 and consists of 100 classes with 600 images per class. The total classes are split into 64, 16 and 20 for base, validation and novel sets. Caltech-UCSD Bird-200-2011(CUB) has a total number of 11,788 images over 200 bird species. These species are divided into 100, 50, and 50 for the base, validation and novel sets, respectively.

4.2 Implementation Details

In the experiments, we primarily used ResNet12 architecture with 4 residual blocks. Each block had 3 convolutional layers with 3×3 kernels. The number of kernels for the 4 blocks was 64, 160, 320, and 640, respectively. Amax-pooling layer was added at the end of the first three blocks. The last block was branched with three pooling layers, which respectively modeled different statistical representations of the images. We opted for the SGD optimizer with a momentum of 0.9 and a weight decay of 5e-4. The learning rate was initialized to be 0.025. We trained the network for 130 epochs with a batch size of 32 in all the experiments. For miniImageNet, tiredImageNet and CIFAR-FS, the learning rate was reduced by a factor of 0.2 at the 70-th and 100-th epoch. For CUB, the learning rate was reduced by a factor of 0.2 for every 15 epochs after the 75-th epoch. We randomly sampled 2,000 episodes from Sn with 15 query samples per class for both 5way 1-shot and 5-shot evaluations, to produce the mean classification accuracy as well as the 95% confidence interval.

4.3 Ablation Studies

The effectiveness of our method is attributed to the ensemble of different branches equipped with multi-order statistics. In this section, we conducted ablation studies to analyze the effect of the 1st -order, 2nd -order and, 3rd -order statistical pooling and their combination on the miniImageNet, CIFAR-FS and CUB datasets. Above methods are respectively denoted as B 1, B 2, B 3,and ELMOS. Their accuracies under 5-way

1-shot and 5-shot tasks on three datasets are shown in Table 1. From the results, we can see that: (1) On all three datasets, the test accuracy of B1 and B3 is higher than B2 under the 1-shot task, but the test accuracy of B2 is higher than B1 and B3 under the 5-shot task. The above phenomenon shows that different order statistics provide different information about the images. (2) The test accuracy of ELMOS is higher than B1, B2 and B3 under both 1-shot and 5-shot tasks, which illustrates that different order statistics complement each other. Combing them can bring more useful information for classification, resulting in higher classification performance.

For each individual in the ensemble learning, the optimization is cooperatively accomplished by the ClassificationBased (CB) loss and Similarity-Based (SB) loss [Scott et al., 2021]. Hence, we conducted ablation experiments to analyze the contribution of each loss on three benchmark datasets: miniImageNet, CIFAR-FS and CUB. Subsequently, we pretrained the model respectively with CB and SB loss alone and their combination, resulting in three methods denoted as CB, SB and CB&SB. The test accuracies under different methods are shown in Figure 3. The test results show that the accuracy of CB&SB is higher than CB and SB, which implies that both CB and SB losses play important roles in our method.

4.4 Comparison with the Most RelatedMethod

Our method is most related to EASY [Bendou et al., 2022], which is also a FSC ensemble learning method in context of transfer learning. The comparison of results between them on CIFAR-FS and CUB datasets is shown in Table 4. From the results, we can see that our method beats EASY by a very large margin under both 1-shot and 5-shot tasks. Please note that our method is more efficient that EASY, because EASY needs to pre-train multiple individual networks, which spends much more pre-training time than our method.

4.5 Comparison with State-of-the-Art Methods

We compare the performance of our method with several state-of-the-art methods. As shown in Table 2 and Table 3, we can see the performance of our method ranks at the top under both 1-shot and 5-shot tasks on CUB. Specifically, our method exceeds the second-best model DeepBDC by 1.2% and 1.0% respectively in 1-shot and 5-shot settings. From Table 3, we can see that our method beats state-of-the-art methods under both 5-way 1-shot and 5-way 5-shot tasks on the dataset of miniImageNet, tiredImagegNet, and CIFARFS. Specifically, on miniImageNet, PAL and IE behave the second best respectively in 1-shot and 5-shot settings. Our method beats them by 0.93% and 1.39%. On tiredImageNet, our method outperforms the second-best MFS by 0.21% and 0.39% respectively in 1-shot and 5-shot settings. On CIFAR- FS, our method achieves 0.31% and 0.13% improvement over IE for 1-shot and 5-shot respectively. In brief, our method consistently outperforms the state-of-the-art FSC methods under both 5-way 1-shot and 5-way 5-shot tasks on multiple datasets.

5 Conclusion

This paper analyzes the underlying work mechanism of ensemble learning in FSC. A theorem is provided to illustrate that the true error on the novel classes can be reduced with ensemble learning on the base set, given the domain divergence between the base and the novel classes. Multi-order statistics on image features are further introduced to produce learning individuals to get an effective ensemble learning design. Comprehensive experiments on multiple benchmarks have illustrated that different-order statistics can generate diverse learning individuals due to their complementarity.

Acknowledgments

This work was partially supported by Joint Fund of Ministry of Education for Equipment Pre-research(8091B022123), Research Fund from Science and Technology on Underwater Vehicle Technology Laboratory (2021JCJQ-SYSJJLB06905), Key Laboratory of Information System Requirements, No: LHZZ 2021-M04, Water Science and Technology Project of Jiangsu Province under grant No.2021063, Qinglan Project of Jiangsu Province.