KTCN: Enhancing Open-World Object Detection with Knowledge Transfer and Class-Awareness Neutralization

1 Introduction

Object detection, a fundamental task in computer vision, is applied across diverse real-world scenarios, including autonomous driving[Zou et al., 2019; Solovyev et al., 2021], medical diagnostics[Ozturk et al., 2020; Loey et al., 2021], and industrial safety[Zhou et al., 2021; Guo et al., 2022]. Traditional detection tasks[Zhang et al., 2020; Tian et al., 2019] commonly rely on training with closed datasets, anticipating the model to recognize objects already labeled within the dataset. However, it is impractical to annotate all objects in natural environments. Consequently, traditional detection models cannot identify unlabeled objects, limiting their application in real-world scenarios.

*Corresponding Author

To tackle this, ORE[Joseph et al., 2021] introduced the Open-World Object Detection (OWOD). The OWOD task consists of two main components: identifying potential objects and incremental learning. The former demands the model to recall as many latent objects as possible, while the latter expects the model to fine-tune existing knowledge at minimal cost to detect newly added objects. Unfortunately, the original implementation of ORE suffered from data leakage; the problem was also identified in OW-DETR[Gupta et al., 2022], CAT[Ma et al., 2023], and PROB[Zohar et al., 2023].

ORE introduces an energy-based model to differentiate between known and unknown samples. Nevertheless, constructing the energy model necessitates access to fully annotated data, resulting in potential data leakage (Figure 1). Subsequent works recognized the issue and employed deformable DETR[Zhu et al., 2020] as a baseline model, utilizing attention mechanisms[Gupta et al., 2022], adaptive pseudo-labeling mechanisms[Ma et al., 2023], and probabilistic objectness[Zohar et al., 2023] to mine potential object samples. However, Faster R-CNN[Ren et al., 2015] holds an advantage[Dhamija et al., 2020] in detecting unknown categories and has been adopted as a baseline network for open-vocabulary[Gu et al., 2021; Zhong et al., 2022], openworld[Wu et al., 2022b; Zhao et al., 2023] and novel category discovery[Zheng et al., 2022; Fomenko et al., 2022] object detection tasks. Therefore, we adopt Faster R-CNN as our baseline model in this paper.

Our exploration in OWOD centers on three critical challenges: (1) how to identify potential unknown objects, (2) how to introduce supervision for these objects without reducing the model's performance on known categories, and (3) how to address the model's bias towards known objects.

Addressing the initial concern, we utilize the zero-shot capability of the SAM (Segment Anything Model[Kirillov et al., 2023]) to generate masks for all objects in the image, providing these masks as pseudo-labels to guide the model in identifying potential unknown objects. As shown in the bottom-left corner of Figure 1, we segment all objects in the image, obtaining class-agnostic masks for each object. Computing the maximum and minimum coordinates allows us to establish bounding boxes for the object. Following this, through a filtering process, we generate pseudo-labels for potential targets.

The second issue arises due to competition between known and unknown categories. To introduce supervision for potential objects, we utilize SAM to generate pseudo-labels. However, competition inevitably occurs during matching as the number of labels increases. Many predictions that could be matched to known instances are instead assigned to pseudo-labels, which affects the model's performance. To address this, we propose the Dual Matching Label Assignment (DMLA) method. Specifically, we consider that ground truth has fewer errors, and the model's performance on known categories entirely depends on the supervision of the available labels. Therefore, the matching of pseudo-labels must not affect the matching of ground truth labels. For the model's prediction, we prioritize matching the annotations of known categories. The unmatched portions are then assigned to pseudo-labels. Finally, we merge the results of both matchings. With this design, DMLA can resolve the competition between known and unknown categories and provide sufficient supervisory signals for training.

The issue of model bias is a significant concern, which is discussed in several works[Zhao et al., 2023; Wang et al., 2023b]. The OWOD task, the prediction is classspecific, leading to a bias towards known categories. To address this, we propose the Class-Awareness Neutralizer (CAN), which utilizes the class-agnostic centerness scores from RPN to mitigate the model's bias towards known categories (bottom-right corner of Figure.1). Objects with higher centerness scores are likelier to be objects requiring detection, attributable to the RPN's class-agnostic predictions. Thus, we utilize the centerness score to update the confidence of predictions. A noteworthy point to consider is that updating confidence during training does not effectively address the issue of bias due to the model's reliance on class labels for optimization, which compromises its ability to handle unknown classes.

With the above methods, we achieve nearly twice the performance over the previous state-of-the-art (SOTA) (Figure 2). Our model constructs with the pure convolutional neural network without employing dense attention mechanisms. Consequently, it demonstrates a significant advantage in terms of inference speed. We summarize our contributions as follows:

• We strengthen the model's supervision for potential objects by transferring knowledge from larger model.
• We propose Dual Matching Label Assignment (DMLA) to address the competition between known and unknown categories.
• In order to address the bias issue within the model, we propose a method called Class-Awareness Neutralizer (CAN), which is designed to mitigate bias without the need for training participation.
• Evaluation on the OWOD task using Pascal VOC and MS COCO datasets shows that our method achieves 60.2 mAP and 41.5 Unknown Recall at 36.36 FPS, establishing a new SOTA.

2 RelatedWorks

2.1 Open-Set Object Detection

The aim of Open-Set Object Detection (OSOD) is to identify unlabeled objects within the training dataset, which was initially introduced by [Dhamija et al., 2020]. [Miller et al., 2019] and [Miller et al., 2018] utilize Dropout Sample (DS) to assess the uncertainty of the detector. [Han et al., 2022] and [Miller et al., 2021] propose the secondary decision boundary for potential targets. Additional research[Kim et al., 2022; Wang et al., 2022; Qi et al., 2022] focuses on generating high-quality unknown candidates. The OWOD task is more challenging compared to the OSOD task, as it not only requires the model's ability to detect unlabeled objects within the training set but also evaluates the model's capacity for fine-tuning newly annotated instances.

2.2 Open-World Object Detection

The open-world object detection task was first introduced by ORE[Joseph et al., 2021]. Current research primarily focuses on generating pseudo-labels for potential objects. ORE[Joseph et al., 2021] designates predictions with high centrality scores and no overlap with annotated objects as potential candidate samples. UC-OWOD[Wu et al., 2022b] identifies candidate samples that are not successfully matched in the RPN as potential candidates. OW-DETR[Gupta et al., 2022] calculates objectness scores based on the attention mechanism and selects the Topk samples as pseudo labels. Both RE-OWOD[Zhao et al., 2023] and CAT[Ma et al., 2023] employ selective search[Uijlings et al., 2013] to generate pseudo labels. Additionally, some methods attempt to estimate robust objectness. 2B-OCD[Wu et al., 2022a] uses a second IOU-based objectness branch to estimate objectness and uses it to confirm detected known and unknown instances during inference. PROB[Zohar et al., 2023] uses probabilistic objectness to adjust the model's confidence output. In this study, we propose the transfer of knowledge from large visual models to guide the model's learning of potential unknown objects.

3 ProblemFormulation

Let's consider a model with inputs = {X, Y}. Here, X is derived from the collection of images DX, within the dataset D, while Y denotes the corresponding set of labels. The label set Y comprises n distinct labels: Y = {y1, y2, . . . , yn}. Each label yi includes a category li and its corresponding Bounding Box annotation bi = [xi, yi,wi, hi], thus yi = [li, bi]. Each category li belongs to the category set K, li ∈ K.

In a traditional object detection task, the training set and test set share the same category set K. However, in the setting of OWOD, they are different. OWOD consists of four tasks T = {T1, T2, T3, T4}. The input image set DTi X for each task comes from D, DTi X ∈ D. They have common parts but do not contain each other, i.e., DTi X ∩ DTj X ̸= ∅, DTi X ∪ DTj X ̸= DTi X

or DTj X , i ̸= j and i, j ∈ [1, 2, 3, 4]. The category set KTi for each task label satisfies that the previous task is a proper subset of the subsequent task, i.e. KTi-1 ⊂ KTi . All tasks share the same test dataset. The category set of the test set includes the category sets of all training sets, Ktest = K = (KT1 ∪KT2 ∪KT3 ∪KT4 ). When the model is trained on task Ti, categories satisfying l ∈ Ktest and l /∈ Ki are considered unknown classes.

T1 evaluates the recall ability of potential objects, i.e., the identifying potential objects. T2-T4 simulate incremental learning: when the annotator labels new categories, the ability to fine-tune existing knowledge.

4 Methodology

4.1 Overall Architecture

The overall structure is shown in Figure 3, which is built upon the standard Faster R-CNN with Feature Pyramid Network (FPN[Lin et al., 2017a]). Initially, the input image is processed through the backbone network and FPN to generate multi-scale feature maps. The class-agnostic RPN then generates candidate bounding boxes. The topk candidate boxes, determined by the objectness scores produced by RPN, are selected and non-maximum suppression (NMS) is applied to eliminate redundant proposals. The remaining candidate boxes are sent to the RoI Head for fine-tuning and classspecific classification.

To provide supervision to potential objects, SAM is utilized to process the input image for generating raw boxes. These boxes are subsequently filtered using both IOU and the shortest side criteria (Section 4.2). To address the competition between ground truths and pseudo labels, the proposed Dual Matching Label Assignment (DMLA) is utilized in both RPN

and RoI Head for label assignment (Section 4.3). In addition, during the inference stage, we use the Class-Awareness Neutralizer (CAN) proposed in this paper to adjust the prediction confidence. (Section 4.4).

4.2 Transference of Knowledge in Large Model

The primary challenge in OWOD lies in identifying potential targets and subsequently providing effective supervision. Given the significant advantages that SAM demonstrates in zero-shot tasks[Kirillov et al., 2023], we employ it to generate pseudo labels for objects that lack annotations. Initially, SAM is utilized for the segmentation of the input image X to obtain class-agnostic masks:

where nsam and nms are the number of masks and the number of pixels in the current mask, respectively. Subsequently, the maximum coordinates are calculated for each mask to generate the collection of bounding boxes:

As show in Figure 4, BSAM may contain ground truths or noises. To avoid this, we utilize the IOU and shortest side of boxes to filter.

For each pseudo-label, we calculate its IOU with all known annotations and determine the maximum value. If the IOU is greater than the threshold TrIOU, we consider it as duplicate with gt instances:

where gti is the instance of known categories. Despite the zero-shot capabilities of SAM, it still generates considerable noise. Upon inspection, we observed that SAM often predicts new masks for certain internal pixels of already segmented objects. these noisy masks are smaller relative to their encompassing objects; thus, employing an appropriate threshold, TrSZ, effectively filters out most noise. Thus, we compute the minimum of their lengths and widths. If its value is less than the threshold Trsz, we treat it as noise:

where xi1 , xi2 , yi1 , yi2 are bounding box coordinates generated in equation (2).

4.3 Dual Matching Label Assignment (DMLA)

Label assignment is a crucial step in object detection, which is used to determine the relationship between the model's predictions P and labels Y . To provide supervision for potential objects, SAM is utilized to generate pseudo labels. Many samples could be matched to known annotations, but due to those labels, are assigned to pseudo-labels. This competition impairs the model's performance.

To address this, we propose DMLA as shown in the bottom left of Figure 3. To avoid the ambiguous[Ge et al., 2021], we adopt the Max IOU[Lin et al., 2017b] for known categories. We first calculate the IOU between ground truth labels Ygt and all predictions P :

where np is the number of P. Then, we compute the maximum value for each prediction associated with known labels:

During RPNmatching, if the IOU is greater than the threshold Trh, the prediction is treated as a positive sample. If it is less than the threshold Trl, it is treated as a negative sample. Predictions between Trl and Trh are ignored.

where Rπ+ 1 , Rπ∼ 1 and Rπ1 denote the positive, ignored and negative sample set generated in first matching Rπ1 , respectively. Then, we match pseudo labels B2 with Rπ1 to produce the corresponding positive, ignored, and negative sample sets Rπ+ 2 , Rπ2 and Rπ∼ 2 . Ultimately, the matching results generated by DMLA is Rrpn = Rπ+ 1 ∪ Rπ∼ 1 ∪Rπ+ 2 ∪ Rπ∼ 2 ∪Rπ2 . In the RoI Head, DMLA executes a similar matching process. The difference is that both the first and second matches do not generate an ignored set. Therefore, the set produced by DMLA in the RoI Head is Rroi = Rπ+ 1 ∪ Rπ+ 2 ∪ Rπ2 .

4.4 Class-Awareness Neutralizer (CAN)

The bias problem in the model manifests as the higher classification probability for certain categories. For the OWOD, model does not encounter annotations of the unknown category, thus the bias problem still exists. Although we introduced SAM to generate pseudo labels for potential categories, these labels have lower accuracy compared to the manual annotations of known categories.

To address this, we propose the CAN, which uses the classagnostic ability of RPN to adjust the confidence (Figure 3 bottom right). Specifically, since the prediction of RPN is in a class-agnostic form, we believe that those with higher objectness in RPN are more likely to object needed detection:

where S1, ..., Skn, Sun and Sbg denote known, unknown and background categories, respectively. So ∈ [0, 1] represents corresponding centerness. The product will reduce confidence, but at the same time, it will improve the absolute ranking of predictions with high centerness, making it more likely to be retained in non-maximum suppression. The principal square root can increase the scores of these predictions, preventing them from being filtered out due to lower confidence.

5 Experiments

5.1 Datasets

Consistent with previous works[Ma et al., 2023; Zohar et al., 2023], we use the VOC[Everingham et al., 2010] and COCO[Lin et al., 2014] to construct a hybrid dataset. Then, the dataset is divided into four tasks, with the respective numbers of training images and instances for each task being: T1(16551, 47223), T2(45520, 113741), T3(39402, 114452), T4(40260, 138996). T1 includes the categories from VOC[Everingham et al., 2010], and each subsequent task adds 20 new categories based on the previous tasks for the incremental experiment. The test set includes all categories with 10246 images and 61707 test instances. During evaluation, all previously seen categories are considered known categories (including newly added ones), while the remaining ones are considered unknown categories.

5.2 Metric

We adhere to the standard evaluation metrics for the OWOD task[Joseph et al., 2021; Gupta et al., 2022]. The performance of the model on known categories is assessed by the widely accepted object detection metric, Mean Average Precision (mAP). For unknown categories, model's performance on objects without annotations is evaluated using the recall rate (U-Recall). In incremental learning tasks (T2-T4), the performance on known categories is divided into two aspects: performance on categories previously encountered, and performance on categories introduced in the current task. The former is used to assess the impact of the introduction of new categories on the model's existing knowledge during incremental learning tasks, while the latter evaluates the model's adaptability and performance on newly introduced categories.

5.3 Details

Our experiments are based on Detectron2[Wu et al., 2019] with 2 × NVIDIA GeForce RTX 4090 (total 48G). We set the batch size to 8 and the initial learning rate to 0.005. The maximum number of iterations was set at 120k. When the number of iterations reached 60k and 100k, we reduced the model's current learning rate to one-tenth of its original value. In multi-scale training, the shortest side of the image was scaled between 220 and 1088, with the longest side not exceeding 1333. For incremental learning, we set the learning rate to 0.001 (T2-T3) and 0.0005 (T4), the number of warm-up iterations to 500, and the maximum number of iterations to 80k, without reducing the learning rate. Apart from T1, all tasks employed replay to review knowledge[Joseph et al., 2021]. For SAM, we chose SAM H to generate pseudo-labels.

5.4 ComparisonWith State-of-the-art Models

We conducted comparative experiments with other SOTA methods, and the results are shown in Table 1. ORE established an energy model that updates detected known categories to unknown categories, which may lead to data leakage as it uses fully annotated data. Following [Zohar et al., 2023; Ma et al., 2023], we removed the energy model in ORE and evaluated its performance (ORE-EBUI). Although CAT achieved higher results by introducing the cascaded structure, it brought a large amount of computational consumption. Therefore, we added a version without the cascaded structure (CAT† ).

Identifying Potential Objects (T1). In T1, our method achieved comprehensive leadership, whether in unknown recall rate or mAP on categories already seen. In terms of performance on known categories, we exceed PROB by 0.7 mAP and surpass CAT† by 0.7 mAP. For performance on unknown objects, we achieved the highest unknown recall rate of 41.5.

• (b) DMLA for RoI Head and Larger SAM Model. Full: onetime matching of all labels. πotm: the one-to-many label assignment. πoto: the one-to-one assignment.
• (c) IOU Threshold for Initial Filtering. The threshold, TrIOU, is used to control the filtering of pseudo-labels by comparing them with ground truth labels.
• (d) Match Sampling. ✔ indicates that we randomly select ordered pairs from positive/negative set and convert them into ignored set.
• (e) Second Filtering of Pseudo Labels. Number refers to the length of shortest side filtered during the second filtering process, while Inner denotes the use of calculated overlap between masks for filtering purposes.
• (f) Class-Awareness Neutralizer. The table compares CAN usage: non-utilization (Wo), application to both known and unknown (Full), application to unknown (Unknown) and application to train (Training).
• (g) Incremental Learning. In the notation 5 0, the first numeral signifies the value of TrSZ during T1, while the subsequent numeral reflects its value in T2. Mv Prev indicates the addition of categories previously encountered to the known categories set in the initial IOU filtering, thereby filtering out potential annotations of known categories.

Table 2: Ablation Study. We conducted a comparison of all configurations within the experiment and explored additional potential methodologies. In the table, mAPPre, mAPCK and mAPBo represent the mAP for previously encountered categories, newly introduced categories in the current task, and for all categories encountered, respectively.

The closest result to this is CAT, but we still lead significantly with a recall rate advantage of 75.1% (41.5 vs 23.7). For CAT† , the advantage reaches 90.4% (41.5 vs 21.8). For PROB, we achieved more than double performance 213.9%.

Incremental learning (T2-T4). InU-Recall, wemaintain the advantage (T2: 101.1%CAT), (T3: 62.7%CAT). With the increase in known categories, our model gradually falls behind PROB in known categories. Such a result is acceptable since KTCN recalls more unknown targets, which reduces the proportion of seen categories in the top 100 predictions.

5.5 Ablation Study

In this section, we demonstrate the step-by-step construction of our KTCN. We start with Faster R-CNN.

DMLA for RPN (table 2a). Initially, we investigate which category of sets is most conducive for matching pseudo labels in RPN. The result indicates that performing the secondary match within the negative sample set yields the most favorable results (56.3 mAP and 5.1 U-Recall).

DMLA for RoI Head and Larger SAM Model (table 2b). At this stage, we introduce a contrast with one-to-one label assignment πoto as an alternative solution to boundary ambiguity[Ge et al., 2021]. However, compared to its oneto-many label assignment counterpart πotm, it still lags by 4.5 in the performance of the unseen category (27.8 vs 32.3). Therefore, we employ πotm and match in the negative sample for unknown categories. Building upon this, we introduce the SAM H model with higher precision, which yields a 2.4 improvement in U-Recall, achieving 34.7.

IOU Threshold for Initial Filtering (table 2c). TrIOU is utilized to filter out samples in pseudo-labels that overlap with ground truth labels. Setting TrIOU to 0.9 allows the model to reach its peak performance in both known and unknown positions, achieving 55.6 mAP and 35.1 U-Recall.

Match Sampling (table 2d). We incorporate FPN and train the model exclusively with known category labels. We posit that randomly converting positive or negative samples into ignored samples[Lin et al., 2017b] could detrimentally impact the model's performance. The findings reveal that sampling set during the RPN and refraining from sampling during the RoI Head leads to the best outcome (59.7 mAP).

Second Filtering of Pseudo-Labels (table 2e). The minimum side is used to filter small noises. In addition, we provide an alternative solution, mask filtering (Inner), which calculates quantifies the overlap by computing the ratio of overlapping pixels to the total pixels in the smaller mask. If the overlap rate is more than 0.9, we treat it as noise. With TrSZ = 5, the model achieves the highest U-Recall of 41.2. Although the mask filtering lags behind by 2.1 points in U-Recall, it demonstrates a marginal enhancement in mAP by 0.2. We maintain the use of minimum side filtering to attain an elevated recall rate.

Class-Awareness Neutralizer (table 2f). In this section, we investigate the impact of CAN on predictions for known and unknown categories separately. The results indicate that employing CAN within all categories yields the highest performance in both mAP and U-Recall. When CAN is utilized in training, the performance declines by 0.4 mAP and 0.5 URecall since it hurts class-agnostic ability of RPN.

Incremental Learning (table 2g). A larger threshold significantly reduces the number of pseudo-labels, which is beneficial for fine-tuning the model on known categories (5 25:

46.1 vs 5 0: 45.3). Conversely, it demonstrates a disadvantage in U-Recall (5 25: 34.6 vs 5 0: 39.2). Thus, we chose TrSZ = 5 as the threshold in incremental learning for balance in known and unknown. Building upon this, we explored whether applying IOU filtering to all previously encountered categories would yield better performance, given that pseudolabels may include categories previously seen. However, as indicated in table 2g, it underperformed others in both known and unknown categories. Therefore, during T2 - T4, we report our results using a consistent TrSZ = 5 and applying IOU filtering solely to the newly introduced categories.

5.6 Visualization

In Figure 5, we present the qualitative analysis of our model and a comparison of inference speeds with the latest works. Our approach demonstrates superior unknown object recall capabilities, as evidenced by its ability to identify paintings, light fixtures (see the first row), throw pillows, windows (see the second row), table lamps, and red pillows (see the third row), while other methods classify them to the background. In inference speed, CAT's cascading structure results in the lowest speed. Compared to the original Faster RCNN, our method does not introduce any additional parameters, thus offering a significant inference advantage. Specific, Our method leads CAT 12.27 (36.36 vs 24.09) and PROB 10.5 (36.36 vs 25.86), showcasing not only the efficiency of KTCN but also its potential for applications.

6 Conclusions

In this study, we introduced the large visual model, the Segment Anything Model (SAM), to generate pseudo labels for potential objects. To leverage the knowledge of SAM, we proposed the Dual Matching Label Assignment (DMLA) to address the competition in one-to-many label assignment. Furthermore, we developed the Class-Awareness Neutralizer (CAN) to eliminate the model's bias toward seen categories. With these methods, we achieved new state-of-the-art (SOTA) in recall rates of the unknown category and inference speed. We expect the methods proposed in this paper to advance the application of open-world object detection tasks in the real-world environment.

Acknowledgements

This work was also partially supported by Guangdong Artificial Intelligence and Digital Economy Laboratory (Guangzhou)