Boosting Few-Shot Open-Set Object Detection via Prompt Learning and Robust Decision Boundary

1 Introduction

Object detection [Ren et al., 2015; Redmon et al., 2016; Lin et al., 2017] has achieved remarkable success, enabling downstream tasks by leveraging large-scale labeled training data under the close-set setting, where training and testing sets share the same classes. However, in real-world scenarios such as safe autonomous driving, training data typically follows a long-tail distribution and many unknown objects may lack clear definitions, which could potentially lead to serious safety risks with close-set detector. To address these limitations, Few-shot Open-set Object Detection (FOOD) [Su et al., 2024] enables the detection of known classes and rejection of unknowns with only few-shot close-set training data, which breaks the conventional assumption of identical class labels in training and testing. Despite its potential, this task is highly challenging due to limited training data and the absence of labels for unknown objects, resulting in weak generalization for unknown rejection and low recall rates.

Existing FOOD methods have utilized weight sparsification [Su et al., 2024] or moving weight average [Su et al., 2023] to facilitate the generalization for unknown rejection. However, the decision boundary between known and unknown classes is still ambiguous due to limited visual information. This ambiguity often leads to the misclassification of unknown classes as known ones with high confidence, resulting in a relatively low recall rate. Recently, the visionlanguage models [Radford et al., 2021] introduce textual modality as complementary for downstream tasks through prompt learning [Zhou et al., 2022b; Zhou et al., 2022a], shedding light on few-shot out-of-distribution image classification [Miyai et al., 2023; Li et al., 2024] and open vocabulary detection [Zhong et al., 2022; Minderer et al., 2022]. However, there are still countless objects that remain outside the scope of the predefined vocabulary in the real world, as depicted in Fig. 1(a). For instance, assuming the current label consists of the vocabulary ['car', 'bird', 'horse'], applying it to the two images in Fig. 2 may lead to false or missed detection. Ideally, the model should distinguish between "what is known" and "what is unknown", as shown in Fig. 2 (right). Therefore, combining prompt learning with the construction of a robust decision boundary could enhance the FOOD detector for unknown rejection.

In this paper, we develop the first prompt-based FOOD framework for unknown rejection. As no training data is available for unknown classes, we select pseudo-unknown samples from region proposals for unknown class optimization. Drawing inspiration from the gradient-based attribution method [Chen et al., 2023] for uncertainty estimation, we propose an Attribution-Gradient based Pseudo-unknown Mining (AGPM) method to select pseudo-unknown samples. It benefits from the diverse interpretative capabilities of different texts for the same image, which is reflected in the attribution gradient differences within the large model. However, these pseudo-unknown samples often contain a mixture of known and unknown properties, which cannot fit the real unknown distribution. As shown in Fig. 1(b), the region proposal could contain features of both the car within the vocabulary and the unknown tire, causing an ambiguous decision boundary between knowns and unknowns. To alleviate this problem, the proposed Conditional Evidence Decoupling (CED) method decouples known and unknown properties from pseudo-unknown samples. This approach is derived from the uncertainty mining property of Evidential Deep Learning [Sensoy et al., 2018] while removing the evidence influence of the ground truth class, as shown in Fig. 1(c). Furthermore, to reduce the impact of local abnormal features on the final decision, the proposed Abnormal Distribution Calibration (ADC) method adjusts the output probability distribution of local features to regularize the model and strengthen the unknown decision boundary. Experimental results demonstrate the superiority of our method on both known and unknown classes. We summarize our main contributions as follows:

• To the best of our knowledge, this is the first work to employ prompt learning for the FOOD task. We propose an Attribution-Gradient based Pseudo-unknown Mining method by innovatively quantifying the text interpretative capabilities in the image-text joint space to mine high uncertainty region proposals.
• We propose Conditional Evidence Decoupling to extract distinct information from pseudo-unknown samples. To enhance robustness, we introduce Abnormality Distribution Calibration to regularize the model and shape a discriminative unknown decision boundary.
• Extensive experiments demonstrate the effectiveness of our FOOD method, which outperforms previous visiononly frameworks and achieves superior unknown rejection performance.

2 RelatedWork

Vision Language Models (VLMs) [Radford et al., 2021; Zhong et al., 2022] leverage extensive image-text pre-training data, enabling zero-shot detection of desired classes based on given vocabulary, and they could quickly adapt to downstream tasks with prompt learning [Zhou et al., 2022b; Zhou et al., 2022a]. VLMs use predefined vocabulary to extend their understanding of the real world. However, real-world data often follows a long-tail distribution, where many rare or non-typical objects are difficult to define and cannot be effectively addressed by Open-Vocabulary Detection (OVD) [Zareian et al., 2021; Zhong et al., 2022; Minderer et al., 2022; Liu et al., 2023]. We tackle this challenge by FOOD, which ensures accurate recognition of known classes in vocabulary while preventing false detections. It is valuable for advancing the robustness and versatility of vision-language models in handling complex and diverse real-world scenarios.

Pseudo-unknown Sample Mining. There are no training samples for unknown classes, thus, the goal of pseudounknown sample mining is to select highly uncertain samples from foreground and background proposals for subsequent optimization of unknown classes. [Han et al., 2022] used a maximum entropy for pseudo-unknown sample mining in open-set object detection. [Su et al., 2024] employed maximum conditional energy in few-shot open-set object detection, and in FOODv2 [Su et al., 2023], they selected proposals with high evidence uncertainty as pseudo-unknown samples. While these methods all operated within the visual feature space, we explore pseudo-unknown sample mining with additional textual information.

Few-Shot Open-Set Object Detection. Compared with Few-Shot Open-Set Recognition (FSOSR) [Liu et al., 2020; Jeong et al., 2021; Wang et al., 2023a; Boudiaf et al., 2023; Nag et al., 2023], the task of few-shot open-set object detection (FOOD) required fine-grained, region-level representations and cannot overlook the impact of background region proposals for the unknown rejection. [Su et al., 2024] initially established a benchmark for the FOOD task. To enhance generalization for unknown classes, [Su et al., 2023] proposed a Hilbert-Schmidt Independence Criterion (HSIC) based moving weight averaging technique to regularize the updating of model parameters. In this paper, we are dedicated to decoupling known and unknown information in pseudo-unknown samples with evidential deep learning to establish a robust decision boundary between known and unknown classes.

3 Method

Our method is a prompt-based few-shot open-set object detection framework including Attribution-Gradient based Pseudo-unknown Mining for uncertainty quantification, Conditional Evidence Decoupling for unknown optimization, and Abnormal Distribution Calibration for robust unknown decision boundary. An overview of our method is shown in Fig. 3. RegionCLIP is adopted [Zhong et al., 2022] as the base framework, composed of two image encoders, a separately trained region proposal network (RPN), and a text encoder. To alleviate the issue of traditional RPNs falsely being classagnostic [Wang et al., 2023b; Sarkar et al., 2024], we train the RPN with a parallel branch to compute the centerness score [Tian et al., 2020], which provides robust localization ability from object position and shape.

3.1 Preliminary

We formalize the FOOD task based on previous research [Su et al., 2024] as training a detector with a class-imbalanced training dataset to accurately classify the K known classes CKnown = CBase ∪ CNovel, reject all unknown classes CUkn, and distinguish between foreground and background according to CBG.

Conventionally, the prompt learning method CoOp [Zhou et al., 2022b] aligns the region feature R with class-specific prompts T, where context words of prompt templates (e.g., "a photo of a") are replaced with continuously learnable parameters, denoted as tc = {v1, v2, . . . , vL,wc}. The semantic alignment loss is defined as:

where S (·, ·) represents the cosine similarity and τ denotes the temperature parameter, yij is an indicator (0 or 1) of sample i belonging to category j in the ground truth label.

Following [Han et al., 2022], we implement enqueue/dequeue operations based on the memory bank and regularize the model with the following visual alignment loss to align the region feature with the same class:

where z is 128-dimensional latent embeddings mapped from R, ci is the class label for the i-the proposal, ε is a hyperparameter, and Q(ci) represents the embedding queue for class ci.

3.2 Attribution-Gradient based Pseudo-unknown Mining

Due to the lack of training data for unknown classes, it is difficult to establish a clear unknown decision boundary. To tackle the above issue, we select a subset of known proposals as pseudo-unknown samples, which may exhibit features of unknown classes. Inspired by the gradient-based attribution method, which was first introduced in the sensitivity analysis (SA) [Simonyan et al., 2013], it evaluates the sensitivity of a particular input feature on the final prediction output for visual interpretability [Selvaraju et al., 2017]. We propose a novel Attribution-Gradient based Pseudo-unknown Mining (AGPM) method to mine high-uncertainty pseudo-unknown samples, which are then employed to construct an unknown decision boundary. Specifically, we take the intermediate feature layer Z (in Fig. 3) as the target layer. For a given proposal feature Rx, we obtain the attribution gradient at Zkij corresponding to the maximum text-image matching score:

where i, j, and k represent the indices of height, width, and channel, respectively. We can obtain the attribution gradient map G corresponding to different proposals. Consequently, we perform global aggregation of attribution gradients as follows:

where γ k ij is an indicator function such that γ k ij = 1 if Gkij ̸= 0 and γ k ij = 0 if Gkij = 0, and | · | denotes the absolute function, resulting in a scalar aggregated outcome. This result could serve as a metric for quantifying uncertainty and assessing the differences between known and unknown classes.

We then analyze the distributions of Aglobal for known, background, and unknown classes with all labels available, identifying distinct distribution patterns, as shown in Fig. 4. Under the premise of having only known class labels, a higher Aglobal aligns more closely with the distribution characteristic of unknown classes. Therefore, we select the proposals corresponding to the top-k highest Aglobal from the foreground and background proposals as pseudo-unknown samples with sampling ratio Sfg:bg.

3.3 Conditional Evidence Decoupling For Unknown Optimization

For FOOD, the unknown objects are easily misclassified into known ones with a high confidence score, which could be attributed to its coupling of known and unknown information. To decouple and learn distinct information from the pseudounknown samples, we reserve a placeholder beyond the vocabulary for unknown classes and model the relationship between known and unknown classes based on conditional evidence. Specifically, we employ Evidential Deep Learning (EDL) [Sensoy et al., 2018] based on the evidence framework of Dempster-Shafer Theory (DST) [Sentz and Ferson, 2002] and the subjective logic (SL) [Jøsang, 2016] to estimate uncertainty. By assuming that output probabilities P follow a Dirichlet distribution, denoted as P ∼ Dir (P | α), EDL constructs distribution of distributions for uncertainty modeling. Drawing on the DST and SL theory, for a classifier with K + 2 classes, we denote exp(lj i ) as the evidence output for the j-th class from the i-th proposal, where lj i = S (Ri,Tj) /τ. Consequently, this allows deriving the parameters for the Dirichlet distribution:

To extract distinct knowledge from identical features, we optimize evidence for known and unknown classes separately. Furthermore, we eliminate the evidence of the ground-truth class while optimizing for the unknown class, and conversely for known classes. In this case, we can alleviate the performance degradation caused by the contradictory evidence of decoupled classes. We formalize this as a conditional EDL loss in the following form:

where ψ(·) represents the digamma function, Lukn and Lgt optimize the evidence for the known and unknown classes, respectively. Subsequently, we use the object perception score as a weight factor to balance the optimization between known and unknown classes, which is derived from the geometric mean of the original objectness score Sobj and the centerness score Scenter fromRPN:

Intuitively, a higher Spercept of foreground proposals indicates more known information, thereby increasing the weight for optimizing known classes, conversely for background proposals. Consequently, we derive the following foreground and background conditional evidence decoupling losses:

(11)

To this end, the final loss expression is as follows:

3.4 Abnormal Distribution Calibration For Robust Decision Boundary

By employing conditional evidence decoupling, the detector can distinguish known and unknown classes using global features. However, certain local anomalous features Zxy still pose a disruption to the decision-making process. Therefore, we delve into the reasons for the differences in global attribution gradient distributions by aggregating local attribution gradients. We observe that, compared to known classes, unknown classes exhibit a greater number of outliers in locally aggregated attribution gradients, as shown in Fig. 5. For the attribution gradient map G, we perform aggregation along the channel dimension, resulting in local aggregation as follows:

for each local position (x, y), Alocal is a scalar, C is the total number of channels. We posit that outlier gradients reflect uncertain local features that hinder global discrimination between known and unknown classes. To address this, we recalibrate the probability distribution by reducing nonground-truth logits to mitigate overconfidence. Specifically, we project the pseudo-unknown local features Zxy into the image-text joint space. The match scores between local and textual features are then computed to obtain the local output logits l′ . These logits are then adjusted using the following abnormality distribution calibration loss to recalibrate the local output distribution:

where Hnorm(p) = -∑c p′ c log p′ c/ log(K) represents the normalized entropy, serving as a weighting factor to constrain the learning of the unknown class. For each pseudo-unknown sample, we select the top-m highest Alocal local features to recalibrate the output probability distribution,

3.5 Overall Optimization

We adopt a two-stage fine-tuning strategy [Wang et al., 2020] to train the few-shot open-set detector, for the base training stage:

Lbase = Lreg + LSalign + γtLValign , (15) and for the few-shot fine-tuning stage:

where Lreg is smooth L1 loss for box regression, γt is a stepwise decreasing weight strategy similar to [Han et al., 2022], β is a hyperparameter and λt = exp(log(λ) · (1 - t/T)) ∈ [λ, 1] denotes the weight that changes exponentially with the current iteration (t) and the total iteration (T ), of which intention is to first learn well-defined semantic clusters and then gradually establish decision boundary between known and unknown classes.

4 Experiments

4.1 Experimental Details

1) Datasets: Following [Su et al., 2024], the data splits VOC10-5-5, VOC-COCO, and COCO-RoadAnomaly [Lis et al., 2019] are used for performance evaluation. VOC10-5-5 includes 10 base, 5 novel, and 5 unknown classes from PASCAL VOC [Everingham et al., 2010]. VOC-COCO has 20 base classes from PASCAL VOC, 20 novel classes from nonoverlapping MS COCO [Lin et al., 2014], and 40 unknown classes. COCO-RoadAnomaly evaluates model generalization in open-set road scenes.

2) Evaluation Metrics: For the FOOD task, the mean Average Precision (mAP ) of known classes (mAPK) and novel classes (mAPN) are adopted as known class metrics. For unknown class metrics, the recall (RU) and average recall (ARU ) of unknown classes are reported as in [Su et al., 2023]. Furthermore, we report Wilderness Impact (WI) under a recall level of 0.8 to measure the degree of unknown objects misclassified to known ones: WI = PK PK∪U -1, and Absolute Open-Set Error (AOSE) to count the number of misclassified unknown objects as in [Han et al., 2022].

3) Implementation Details: We employ RegionCLIP [Zhong et al., 2022] as the image encoder, and ResNet-50 [He et al., 2016] pre-trained on ImageNet as the RPN image encoder. Class-specific prompt training follows CoOp [Zhou et al., 2022b] with a context length of 16, using a two-stage training strategy [Wang et al., 2020], where the base training phase trains a base detector using Cbase, and the few-shot fine-tuning phase refines the model with a small, balanced set comprising both Cbase and Cnovel. We employ SGD with 0.9 momentum, 5e-5 weight decay, and a batch size of 1 on a GTX 1080 Ti GPU. The learning rate is 0.0002 for base training and 0.0001 for fine-tuning. Visual alignment loss settings follow the setting in [Han et al., 2022]. Other hyperparameters τ , ε, λ, and β are 0.01, 0.1, 1e-4, and 1.0, respectively.

4) Baselines: We compare vision only open-set framework including: ORE [Joseph et al., 2021],PROSER [Zhou et al., 2021],OPENDET [Han et al., 2022],FOOD [Su et al., 2024],and FOODv2 [Su et al., 2023]. Furthermore, we employ OPENDET and FOODv2 within our open-set detection framework, denoted by OPENDET(+Ours) and FOODv2(+Ours) to conduct a fair comparison. We employ max entropy and max evidential uncertainty as pseudounknown sampling methods respectively, with unknown probability loss and IoU loss as optimization strategies for unknown classes respectively.

4.2 Main results

1) Experiments on VOC10-5-5: Table 1 presents the FOOD results on VOC10-5-5, where we report the results of finetuning on 1, 3, 5, and 10 shots, averaging ten runs per setting for a fairer comparison. Based on our framework, both OPENDET(+Ours) and FOODv2(+Ours) show significant improvements compared to their original versions, demonstrating the advantages of our open-set framework. Compared to previous state-of-the-art methods with traditional open-set framework, our approach (with k = 3, Sfg:bg = 1 : 3,m = 1) achieves significant improvement on unknown class metrics, surpassing the second best (mean results) by 18.95%RU , 7.24%ARU , 3.58 WI and 324.13 AOSE. Our method consistently achieves the state-of-the-art performance on most metrics related to unknown classes, despite a slight decrease in accuracy for known classes.

2) Experiments on VOC-COCO: Table 2 displays the FOOD results on VOC-COCO, which is a more challenging dataset. We report results of fine-tuning on 1, 5, 10, and 30 shots, averaging ten runs per shot setting to ensure a fair comparison. Both OPENDET(+Ours) and FOODv2(+Ours) also show significant improvements compared to the original frameworks. Compared to prior state-of-the-art methods with the traditional open-set framework, our approach (with k = 3, Sfg:bg = 1 : 1,m = 1) shows a marked improvement, outperforming the second best (mean results) by 6.31% RU, 1.38% ARU, 4.42 WI and 70.00 AOSE. The 1-shot AOSE performance does not surpass previous benchmarks, likely due to the strong learning capability of prompt-based methods with limited samples, which incurs overfitting of known classes. Our method generally achieves higher ARU and lower WI and AOSE, effectively rejecting unknown objects rather than misclassifying them as known.

4.3 Ablation Studies

We ablate the proposed losses with the 1-shot VOC 10-5-5 setting in Tab. 3. By employing attribution gradients to filter pseudo-unknown samples, the proposed LCED establishes a discriminative decision boundary between known and unknown classes through decoupled evidential learning. The regularization with LADC yields improved WI and AOSE without adversely affecting mAPK, indicating its facilitation for the decision boundary construction.

We conduct a more detailed ablation analysis in Tab. 4, '(+Ours)' denotes our open-set detection framework, while '(P)' and '(U)' represent the Pseudo-unknown sample mining method and the Unknown class optimization method from the corresponding paper, respectively. The top section demonstrates the advantages of our framework. The middle section reports the experimental results of different pseudo-unknown sample mining methods. Our AGPM shows better overall performance. The bottom section presents the results of various optimization methods for unknown classes. When we apply only Eq. 7 to optimize unknown classes (NonDecoupled), it results in a degradation of accuracy for known classes. Subsequently, we remove the conditions j ̸= gt in Eq. 7 and j ̸= ukn in Eq. 8 (Non-Conditional), which significantly reduces mAPK. By retaining the conditions, we achieve a better trade-off between known and unknown metrics. For details on the parameter ablation study, please refer to the appendix.

4.4 Visualized results

We conduct visual comparisons between FOODv2 [Su et al., 2023] and our proposed method in Fig. 6 under 10-shot VOC10-5-5 experimental setup. It reveals that our method successfully recalls more unknown objects across three openset datasets and makes more accurate distinctions between known and unknown objects. For the VOC dataset, the two images on the left illustrate that FOODv2 mistakenly classifies an airplane as an unknown object, while our method correctly identifies it. The right demonstrates that our approach successfully detects the car within the vocabulary and rejects the unknown classes (tree and billboard). This suggests that our approach enables enhanced perception of object presence and facilitates the unknown rejection.

5 Conclusion

In this paper, we propose a novel approach for addressing the challenging few-shot open-set detection problem. By introducing prompt learning to the FOOD task and incorporating an unknown class placeholder, our framework captures information beyond the known vocabulary. To address the unavailable training data of unknown classes, an attribution-gradient based method is proposed to mine high-uncertainty samples as pseudo-unknowns. Additionally, we propose a conditional evidence decoupling loss and a local abnormal distribution calibration loss to improve unknown class optimization and establish a robust decision boundary for unknown rejection. Extensive experiments demonstrate that our method achieves state-of-the-art performance, surpassing existing approaches.

Acknowledgments

This paper is supported by the National Key Research and Development Program of China (Grant No. 2024YFB3310904), the Science and Technology Project of Hainan Provincial Department of Transportation (Grant No. HNJTT-KXC-2024-3-22-02), the National Natural Science Foundation of China (Grant No. 62206184, 62272018, 62372448), the Basal Research Fund of Central Public Research Institute of China (Grant No.246Z4306G) and Zhongguancun Laboratory.