Boosting Few-Shot Open-Set Recognition with Multi-Relation Margin Loss

1 Introduction

Few-shot learning (FSL) has emerged as a promising direction for tackling the challenge of recognizing new classes with few labeled samples [Chen et al., 2019; An et al., 2023]. Conventional few-shot learning methods mostly hold a closed-set assumption that testing samples belong to the predefined classes of training samples [Liu et al., 2020b]. However, in real-world applications, there always exist unforeseen testing samples out of classes of training samples, which results in severe distribution shifts [Hsu et al., 2020].

∗Corresponding author

Hence, the learned model should not only classify known classes but also reject unknowns.

Open-set recognition (OSR) [Bendale and Boult, 2016; Oza and Patel, 2019; Perera et al., 2020] considers the scenario where testing samples could come from unknown classes. The existing OSR methods mainly learn an unknown class detector by utilizing the characteristics of known classes from substantial datasets. But having a sufficient number of training samples is inconsistent with few-shot learning. Therefore, directly applying OSR methods in the few-shot setting could degrade the performance.

Recently, few-shot open-set recognition (FSOSR) is proposed to tackle the new problems that classify the known classes and reject the unknown ones with a few samples. The main challenge of FSOSR is how to detect unknown class samples while maintaining the classification capability of known classes with limited labeled training samples. Existing methods for FSOSR mostly construct an ambiguous distribution of known classes while ignoring the latent distribution information of unknowns [Jeong et al., 2021; Pal et al., 2022]. These methods easily lead to confusion between unknown and known samples and further degrade the open-set performance. Limited methods attempt to learn the unknown information by directly utilizing or generating pseudo-unknown class samples [Huang et al., 2022; Pal et al., 2023]. However, these methods heavily rely on the quality of pseudo-unknown samples and the latent true distribution of unknown classes may not be adequately represented by these pseudo-unknown samples. Hence, how to properly delineate the distribution of unknowns is an essential problem for FSOSR.

In this paper, we propose multi-relation margin (MRM) loss, which seeks to dynamically refine the distinct decision boundary for known classes and implicitly delineate the distribution of unknown classes without relying on pseudounknown samples. Specifically, MRM enlarges the margin between different classes and adjusts a hypersphere for each known class with a learnable radius to form a finer boundary between the closed and open spaces. In order to capture the entire data distribution information and further refine the margin and radius, we excavate multiple relationships among paired samples, including self-relation and relative relation, where self-relation is the similarity relation inside a pair and relative relation is the correlation among different pairs. They both make a significant impact on exploiting the potentiality of each pair and precisely measuring the distribution of paired samples. Therefore, the areas away from the boundaries of known classes implicitly delineate the distribution of unknowns, and open-set detection can be easily conducted by rejecting samples located in these areas. The effectiveness of MRM is demonstrated by experiments on public benchmark datasets.

In summary, our contributions are as follows:

• We propose a novel multi-relation margin (MRM) loss to boost the performance of FSOSR. Compared with previous methods, MRM is able to implicitly delineate the distribution of unknown classes without relying on additional pseudo-unknown class samples.
• We explore a new formulation of margin-based deep metric learning for FSOSR. Our method enforces the margin between different classes by extracting the multirelationship of paired samples to learn a finer boundary between closed and open spaces.
• We perform adequate experiments to verify the effectiveness of MRM. Results reveal that few-shot methods with MRM loss can improve the unknown detection of AUROC by a significant margin while correctly classifying the closed-set.

2 RelatedWork

Few-shot learning. Few-shot learning focuses on training models with limited labeled samples to develop the capacity of recognizing novel classes [An et al., 2021; Ma et al., 2022a]. Representative few-shot learning literature can be typically organized into two branches: optimizationbased methods and metric-based methods. Optimizationbased methods can achieve generalization ability within a small number of update steps. Specifically, MAML [Finn et al., 2017] adapts model parameters to novel tasks based on the loss computed by secondary gradients. Metric-based methods try to model an appropriate feature metric space and predict query samples based on the distance function. ProtoNet [Snell et al., 2017] suggests measuring the Euclidean distance between the mean of the support samples, indicated as class prototypes, and the query samples for classification. On that basis, FEAT [Ye et al., 2020] transforms class prototypes to task-adaptive prototypes with a set-to-set function. In our work, we mainly consider the classification combined with ProtoNet and FEAT. Although prior FSL studies have demonstrated promising results under the closed-set assumption that the query set and support set belong to the same classes, the performance in the open-set setting is not guaranteed.

Open-set recognition. Open-set recognition describes a scenario where new classes unseen in training occur in testing, thus classifiers must be able to properly identify seen samples while rejecting unseen ones [Geng et al., 2020]. OpenMax [Bendale and Boult, 2016] was the first solution towards open-set deep network combined with Extreme Value Theory to detect unknowns by thresholding. Later researchers mostly use discriminative or generative approaches to tackle this issue. Specifically, C2AE [Oza and Patel, 2019] and CGDL [Sun et al., 2020] utilize class conditioned autoencoders that have been trained on all of the train samples to define a threshold. Since it requires a large amount of training samples to train an unseen sample detector or a generative model, directly applying these OSR methods to the few-shot setting could cause over-fitting and lead to poor performance.

Few-shot open-set recognition. FSOSR has gained popularity recently, which addresses OSR in the FSL setting. Existing research on FSOSR mostly trains a model without considering the distribution of unseen classes, which could degrade the performance on unknown class detection. SnaTCHer [Jeong et al., 2021] thresholds the distance between original prototypes and query replaced transformed prototypes to detect unseens. [Pal et al., 2022] reinforce the feature extractor through a residual attention network. Limited methods focus on the distribution of unknowns by directly utilizing or generating additional pseudo-unknown samples. [Song et al., 2022] utilize background features from seen classes as the pseudo-unseen classes for classifier training. ATT [Huang et al., 2022] augments the classifier with additional negative prototypes via a negative generator. However, the pseudo-unknown based approach heavily depends on the quality of pseudo samples, and these pseudounknowns may not be adequate to represent the latent true distribution of unknown classes. In our work, we aim to implicitly delineate the distribution of unknowns through deep metric learning.

Margin-based deep metric learning. Deep metric learning is an important series of meta-learning methods for learning better features. In the metric learning area, the standard softmax loss is regarded as insufficient for discrimination on different training classes [Liu et al., 2020a]. It can only construct an ambiguous distribution. According to previous studies [Sohn, 2016; Song et al., 2016], integrating the large margin to the standard loss could help learn highlydiscriminative features and efficiently improve the performance on visual recognition tasks. As the representative of margin-based methods, Triplet loss [Schroff et al., 2015] enforces each input sample as an anchor and works on the triplet of samples {x+i , xai , x-i }, where y+ i = ya i and yi ̸= ya i . xai is usually called an anchor in the i-th triplet, x+i is the positive sample and x-i is the negative sample. It assumes the anchor is closer to the similar point than the dissimilar one by the margin. Center loss [Wen et al., 2016] encourages samples to concentrate on their corresponding class center. However, these methods do not utilize all the samples contained in a batch, which could result in losing valuable information during training. On the contrary, Multi-similarity loss [Wang et al., 2019b] considers each pair of samples in a batch that construct an informative structure among anchor-sample pairs and provide a new weighting scheme by calculating the derivative of the loss function.

Although previous margin-based losses have achieved promising results, they normally assume that sufficient samples can be provided for training. Recently, some researchers are devoted to applying margin-based deep metric learning in the few-shot setting [Li et al., 2020]. For instance, [Liu et al.,

2020a] suggest using a negative margin to improve the novel class performance. However, the negative margin will simultaneously degrade the classification capacity for base classes. In our work, we focus on detecting unseen samples while maintaining the classification capability of known classes.

3 Method

3.1 Preliminary

Few-shot open-set recognition (FSOSR) aims to classify known classes and detect unknown ones with limited training samples. FSOSR follows the typical FSL episode training and testing strategy. Similarly to closed-set few-shot classification, for given N classes with K labeled supports per class, an episode is called as an N-way K-shot problem. Formally, a few-shot task can be represented as T = (DS ,DQ). Specifically, for the support set DS = {xSi , yS i }NK i=1 , where xi ∈ X S and yi ∈ Y S indicate sample in the support set X S is from known classes with the known label space Y S . Different from closed-set few-shot learning, the query set includes unknown class queries which can be denoted as DQ = DK ∪ DU , where DK stands for the known set and DU stands for the unknown set. The known query set is represented as DK = {xKi ∈ X K , yK i ∈ Y S }NQ i=1 , which indicates the known query is sampled from classes with the same label space Y S as support set, and Q is the number of known queries in each class. The unknown query set is represented as DU = {xUi ∈ X U , yU i ∈ Y U }N U i=1 , where N U is the number of unknown queries and Y S ∩ Y U = ∅. The goal of FSOSR is to train a model with small support set DS that can classify known queries DK and detect unknown queries DU .

3.2 Multi-RelationMargin Loss

Instead of generating pseudo-unknown class samples to approximate the distribution of unknowns, MRM aims to obtain a finer boundary between the closed and open spaces and further implicitly delineate the distribution of unknown classes. A question is then raised: how to learn discriminative spaces for closed and open sets from limited samples?

Inspired by margin-based methods, MRM tackles this question by optimizing margin spaces. Compared with existing methods, MRM divides the distribution of known classes into clearer regions and takes full advantage of the distribution information. The illustration and comparison of different margin-based losses and MRM loss are presented in Figure 1. Specifically, MRM mainly implements three strategies: (1) Anchor generating: utilize prototype as the anchor for each pair to represent global distribution in the class; (2) Margin and radius refining: force the margin between different classes while refining a hypersphere for each known class to preserve the distribution information; (3) Pair-wise relation modeling: extract multiple relationships among paired samples to cover the entire distribution information contained in a pair.

Based on these strategies, MRM loss can be formulated as:

Specifically, MRM works on pairs of samples that consist of positive pair {pc, fθ(x+i )} and negative pair {pc, fθ(x-i )}, Where xi ∈ X S indicates the sample from the support set. For class c, the support set can be divided into positive and negative sets, including X + c = {x+i }Ki=1 and X - c = {x-i }(N-1)K i=1 , where y+ i = c, yi ̸= c. pc represents the class prototype and Rc denotes the radius of hypersphere. fθ(·) is the feature extractor with learnable parameters θ. d(·) is the Euclidean distance function, m stands for the margin. λ is a constant trading off the regulation of radius and margin loss item, α and β are the scaling factors for positive and negative sets respectively. The details of each strategy are as follows.

Anchor Generating

In this paper, we apply two types of anchor generation techniques. Initially, we utilize the class prototypes as the classspecific anchor. Moreover, we incorporate the anchor generation process with a transformation function to adapt and modify the task-specific anchor.

Class-specific anchor. The class prototype [Snell et al., 2017] is proposed based on the assumption that there exists an embedding space where points cluster around a single prototype representation for each class. Therefore, MRM utilizes the class prototype as the anchor for each pair instead of an individual sample to achieve the representative features of known classes. The class prototype is formulated as the mean vector of support embedding vectors:

where X S c is the support set labeled to class c. The central idea is to learn more centralized features by forcing the samples closer to their corresponding prototypes and further to samples from different classes. With the anchor pc, MRM could obtain a finer boundary between the positive and negative set by a margin m. Specifically, MRM enforces the margin between the positive pair {pc, fθ(x+i )} and negative pair {pc, fθ(x-i )}:

By utilizing the class-specific anchor p, we can derive the class-specific multi-relation margin loss, denoted as LMRMC , based on Eq.(1).

Task-specific anchor. According to previous studies, transformation functions are demonstrated to provide effective FSOSR performance [Jeong et al., 2021; Ma et al., 2022b]. Inspired by these methods, we apply a standard set-to-set function to modify more distinguishable prototypes as anchors to consider the entire distribution of the task [Ye et al., 2020]. The set-to-set function transforms a set of original prototypes P = {pc}Nc=1 to a set of adapted prototypes P ′ = {p′c }Nc=1 , which projects the input feature into query, key, and value space with learnable transformation matrices. The training process is as follows:

where A is the attention weights matrix. Wq,Wk, and Wv ∈ RM×M are learnable projection kernels for query, key, and value space respectively. σ indicates the layer normalization, and s(·) is the softmax function. By incorporating the modified prototype p′ , we can obtain the task-specific multirelation margin loss denoted as LMRMT , which adapts the original prototype p as described in Eq.(1).

Margin and Radius Refining

Previous algorithms mostly attempted to compress sample points of the same class as much as possible. Consequently, these samples will be shrunk into one point in the embedding space and lose their similarity structure inside the class, which degrades the recognition performance. Inspired by [Wang et al., 2019a], instead of pulling intra-class samples as compact as possible, MRM learns a hypersphere for each known class with a learnable radius to preserve the distribution structure inside it. Specifically, MRM forces the distance between the class prototype and positive sample to be smaller than a threshold, which is the radius of each class's hypersphere. Mathematically,

where x+i ∈ X + c and Rc is the hypersphere radius of class c. Details about updating the hypersphere by optimizing the radius are explained in Section 4.2. Hence, we can obtain a distinct decision boundary for each class while taking the prototype as a center point. Based on the positive boundary, the margin between positive and negative sets in Eq.(3) can be refined as:

Since the number of negative pairs is much larger than that of positive pairs in a few-shot task, we adjust the radius based on the negative pairs which are more informative. According to Eq.(6), R is limited to the distance between the anchor and negative samples, we can approximate it by calculating the negative distance. Besides, we add a trade-off hyperparameter that allows some points to be mapped outside the sphere. For class c, the hypersphere could be updated by adjusting the

Where h(·) is the quantile function and ν is the hyperparameter that controls violations of the boundary.

Pair-Wise Relation Modeling

In order to utilize the entire distribution information contained in a pair, MRM extracts self-relations and relative relations among paired samples. Self-relation is denoted as the similarity relation between the anchor and positive or negative sample in a pair. Relative relation is computed by considering correlations of other pairs. According to previous research [Wang et al., 2019b], they both have a significant impact on exploiting the potentiality of each pair and building a discriminative distribution structure, whereas most existing methods only partially explore this factor.

As Multi-similarity loss suggests, pair-wise relation could be modeled through sample weighting, and the partial derivative of the loss function with respect to each pair is its weight. Paired samples with higher weights are considered to be more informative in the optimizing process. For instance, negative pairs that are in close proximity to the anchor and closer to the anchor than other negative samples are assigned higher weights, which means such negative samples are more difficult to classify, and thus carry more valuable distribution information.

Since the main goal of MRM is to encourage positive pairs to be closer while pushing negative pairs away from each other, it is reasonable to assume ∂LMRM ∂d(pc,fθ(x+i )) ≥ 0 for the positive pair {pc, fθ(x+i )}, thus its weight wc+ i can be computed as:

where the weight is computed jointly from its selfrelation by measuring the anchor and positive sample in eα(d(pc,fθ(x+i ))-Rc) , and relative relations by comparing to other positive pairs {pc, fθ(x+k )} with the same anchor pc in the item d(pc, fθ(xk+)) -Rc. Similarly, for the negative pair {pc, fθ(x-i )}, we assume ∂LMRM ∂d(pc,fθ(x-i )) ≤ 0, and the weight wci is formulated as:

Remark. In comparison to existing margin-based losses, MRM offers several key advantages: (1) MRM divides the distribution of known classes into distinct regions and employs a learnable radius centered on the prototype to refine the distribution of each class. This allows MRM to detect unknown samples that are located away from known regions, while existing methods only construct ambiguous decision boundaries. (2) MRM utilities both intra-class and inter-class distribution of samples to learn an appropriate radius for each known class, efficiently preventing overfitting and enhancing generalization in the few-shot setting, which is not addressed by existing methods.

4 Experiments

4.1 Datasets and Evaluation Protocols

Following previous studies, we conduct experiments on three public benchmark datasets CUB-200 [Wah et al., 2011], tieredImageNet [Ren et al., 2018], and miniImageNet [Vinyals et al., 2016] to verify the effectiveness of MRM.

We evaluate the model with closed-set classification accuracy (Acc) and the area under ROC curve (AUROC) for unknown class detection. The Acc measures the classification capacity via known samples, and the AUROC evaluates the unknown detection capacity via both known and unknown samples. Since prior FSL studies have already demonstrated promising results under the closed-set assumption, the main challenge of FSOSR is how to obtain better AUROC while maintaining the Acc. Following [Liu et al., 2020b], we set N = 5 and K = 1, 5 during meta-training and meta-testing. For each episode, we sample 5 known classes and 5 unknown classes, with each containing 15 queries. Code is available at https://github.com/Casie-che/MRM.

4.2 Implementation Details

Backbone training. Following previous FSOSR methods [Jeong et al., 2021], we use ResNet12 [He et al., 2016] based architecture as the feature extractor which creates 640dimensional feature vectors before the classifier. As [Ye et al., 2020] suggest, we first pre-train the backbone with a simple classifier to classify all known classes with the cross-entropy loss. The classification performance of sampled 1-shot tasks over the penultimate layer embedding is evaluated to identify the best pretrained model, whose weights are subsequently utilized to initialize the embedding function. Then we use a SGD optimizer to train the feature extractor and transformers over 200 epochs. The initial learning rate is set to 0.0002 for the feature extractor and 0.002 for transformers with a multistep learning rate schedule. MRM is finetuning the feature extractor over 30 epochs with 0.0005 weight decay. During training, We use the validation set to select the best model. The comparison results are calculated over 600 evaluation episodes on the test set.

Hypersphere updating. To construct appropriate decision boundaries for each known class, we utilize dynamic radii instead of a fixed threshold during training. Since the network parameters θ and radius of hypersphere R are generally on different scales, using a common optimal learning rate may be inefficient. Inspired by [Ruff et al., 2018], we optimize θ and R in an alternating minimization coordinate descent approach. We train the network parameters θ firstly while the radius R is fixed, then after one epoch, we calculate the radius for each class based on the embeddings extracted from the network of the latest update.

Classifier setting. Inspired by [Jeong et al., 2021], we utilize two classifiers to detect unknown samples. The first uses the distance between a query and its predicted class prototype, which is denoted as Distance. The other is SnaTCHerF, which measures the distance between the transformed prototypes and original class prototypes. To evaluate the efficiency of MRM, we first plug LMRMC in different few-shot methods. We finetune the ProtoNet [Snell et al., 2017] as the representative of class-specific methods with MRM loss and Distance classifier to get MRMC-ProtoNet. Then plug LMRMT in FEAT [Ye et al., 2020] to get MRMT -FEAT and MRMT -SnaTCHerF, which respectively utilize Distance and SnaTCHer as the classifier.

Experimental setting. For simplicity, we maintain the same hyper-parameters of our methods for all the experiments which are obtained from the validation set. We λ = 0.1, α = 1, β = 3,m = 1 for the loss function in Eq.(1), and ν = 0.1 for hypersphere updating in Eq.(7).

4.3 Comparison with PriorWork

In order to validate the effectiveness of MRM, we conduct thorough comparisons of few-shot methods plugged in the MRM loss with state-of-the-art results in FSL. Additionally, we compare the performance of MRM with classical marginbased losses which are widely used in deep metric learning.

Comparison with Few-Shot Methods

We first compare with the standard FSL methods, including baseline [Chen et al., 2019], ProtoNet [Snell et al., 2017] and FEAT [Ye et al., 2020]. Then we select conventional OSR methods, including NN [Mendes J´unior et al., 2017], OpenMax [Bendale and Boult, 2016], and CounterFactual [Neal et al., 2018]. Next, we choose PEELER [Liu et al., 2020b], SnaTCHer [Jeong et al., 2021], ATT [Huang et al., 2022], and ProCAM [Song et al., 2022] as representatives of FSOSR methods.

We illustrate the average closed-set classification accuracies and unknown detection AUROCs over 600 test episodes with 95% confidence intervals under 5-way 5-shot and 1-shot settings respectively. We cite most of the baseline results based on [Huang et al., 2022] and [Song et al., 2022]. Table 1 demonstrates the results for CUB-200, Table 2 and Table 3 show comparisons on tieredImageNet and miniImageNet respectively. For each setting, the best results are highlighted in bold.

According to the results, we can find that OSR methods show unsatisfactory classification accuracies in the few-shot setting, whereas FSL methods perform poorly in unknown class detection. On the contrary, methods plugged in MRM loss can improve the unknown detection of AUROC by a significant margin and maintain the closed-set accuracy. For instance, on CUB-200, tieredImageNet, and miniImageNet, MRMC-ProtoNet has around 13%, 13%, and 16% performance improvements compared with the vanilla ProtoNet under 1-shot setting, and 5%, 13%, 17% improvements under 5-shot setting, while correctly classifying the closed-set. Results further verify the effectiveness of the combination of MRM and transformation function. Compared with stateof-the-art FSOSR methods, methods with LMRMT achieve better AUROC in most settings without relying on additional pseudo-unknown class samples.

Comparison with Margin-Based Methods

To provide a more comprehensive explanation of the advantages of MRM loss, we conduct a comparative analysis with existing margin-based losses, including Triplet loss [Schroff et al., 2015], Center loss [Wen et al., 2016], and Multisimilarity loss [Wang et al., 2019b]. We illustrate the average unknown detection AUROCs over 600 test episodes on 5-way 5-shot miniImageNet tasks in Figure 2. Remarkably, results clearly demonstrate that MRM surpasses the performance of the compared margin-based losses. By dividing the distribution of known classes into clearer regions and refining the distribution of each class, MRM enables effective identification of unknown samples located away from known regions. Through the comparison of unknown detection AUROCs, MRM demonstrates its superiority in building a highly discriminative distribution structure.

4.4 Ablation Study

Visualization of the effect of MRM loss. We plot t-SNE [van der Maaten and Hinton, 2008] figure to visualize the distributions of images sampled from miniImageNet in the feature space learned by standard softmax loss and MRM loss in Figure 3, respectively. In Figure 3a, the feature extractor learned by softmax loss can only construct ambiguous decision boundaries while ignoring the distribution of unknowns. It might easily lead to confusion between unknown and known samples as all features just follow one distribution. Figure 3b shows that the feature extractor trained by MRM loss can obtain a finer boundary between classes and a distinct decision boundary for each class, which enables the model to detect unknown samples by rejecting samples away from closed boundaries while maintaining the classification capability.

Effect of different strategies for MRM loss. We perform our ablation study on the 5-way 1-shot and 5-shot tasks on miniImageNet to analyze the effectiveness of different strategies in Table 4. Since the main challenge of FSOSR is how to improve detection capability for unknown classes, we mainly focus on the performance of AUROC. We compare our full model with a number of stripped down versions by evaluating the AUROC of ProtoNet plugged in different margin losses and MRMC-ProtoNet. Specifically, 'MRM #1' utilizes the individual sample as an anchor in a pair and forces the margin between positive and negative samples. 'MRM #2' utilizes class prototypes as anchors. On that basis, 'MRM #3' refines a hypersphere for each known class with a learnable radius. The results show an increasing trend in unknown class AUROC, which indicates that integrating a large margin to the loss function could improve the performance efficiently, and MRM could further build a more discriminative distribution structure to boost FSOSR performance.

Performances on different numbers of unknown classes. During the conducted experiments, we maintain a fixed number of 5 unknown classes. However, in real-world applications, the presence of unknown samples can vary. To account for this, we investigate the performance of AUROC for MRMC-ProtoNet in different unknown class settings, specifically focusing on 5-way 1-shot and 5-shot tasks on miniImageNet. We vary the number of unknown classes from 1 to 10, while keeping the loss function and classifier unchanged. As shown in Figure 4, MRM demonstrates robust

(a) Softmax loss

(b) Ours: MRM loss

Figure 4: Performances on different numbers of unknown classes on 5-way miniImageNet tasks. Average unknown detection AUROCs (%) over 600 test episodes.

performance across different unknown class configurations.

5 Conclusion

In this paper, we propose a novel loss function named multirelation margin (MRM) that can plug in existing few-shot methods to boost the performance of few-shot open-set recognition. We first analyze the limitations of pseudo-unknown based approaches, then propose to enlarge the margin between different classes and refine the decision boundaries for known classes to implicitly delineate the distribution of unknowns without relying on pseudo-unknown samples. To better capture the distribution information, we further construct multiple relationships among paired samples. Detailed experiments reveal that MRM indeed helps to delineate the distribution of unknowns and leads to better unknown detection of AUROC while correctly classifying the known classes.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 62076062) and the Social Development Science and Technology Project of Jiangsu Province ( No.BE2022811). Furthermore, the work was also supported by the Collaborative Innovation Center of Wireless Communications Technology and the Big Data Computing Center of Southeast University.