Self-Classification Enhancement and Correction for Weakly Supervised Object Detection

1 Introduction

Object detection aims to localize objects of interest and classify them, which is a fundamental task in the field of computer vision. The recent decade has witnessed rapid progress [Girshick, 2015; Ren et al., 2015; Liu et al., 2016; Redmon et al., 2016] in various object detection scenarios [Nie et al., 2023; Wang et al., 2023; Jiao et al., 2024; Wang et al., 2024b], benefiting from the development of convolutional neural networks (CNN). In spite of the remarkable advances, current fine-grained instance-level annotations are labor-intensive and time-consuming to obtain. This paper focuses on the domain of weakly supervised object detection

∗Corresponding authors: Lechao Cheng and Zhou Yu

(WSOD) [Su et al., 2022], which requires only image-level annotations, i.e., existing object categories in a given image, to achieve the object detection task.

Recent WSOD approaches [Tang et al., 2017; Wei et al., 2018; Ren et al., 2020; Huang et al., 2020] generally convert WSOD into a two-stage multi-class classification (MCC) pipeline. In the first stage, a multiple instance detection network (MIDN) [Bilen and Vedaldi, 2016] is constructed, leveraging multiple instance learning to introduce competition both among object classes and proposals (denoted as 1st MCC). This process effectively identifies candidate regions that contain significant class-specific patterns. However, MIDN often suffers from partial-located issues, wherein high scores are assigned to the detections localizing only the most discriminative parts. To overcome this challenge, in the second stage, cascaded online multi-class classifiers [Tang et al., 2017; Chen et al., 2020] are integrated to refine the classification scores of MIDN, and assorted strategies [Zeng et al., 2019; Ren et al., 2020] are designed to generate pseudo instance-level labels for training these classifiers (denoted as 2nd MCC). Despite the promising results achieved by these methods, as shown in Figure 1, they overlook the potential classification ambiguities and the unique strengths of the two distinct multi-class classification tasks across the two stages:

• (i) During the 1st MCC task, some mis-located proposals, especially those only containing discriminative parts of an object, are classified as the corresponding object class and leveraged to generate its class-specific features. However, after multiple refinements in the 2nd MCC task, these proposals will be classified as the background class, whose features are instead pushed toward those of mis-located proposals from other classes. These ambiguities compromise the quality of the classspecific features produced by the detector.
• (ii) These two MCC tasks are guided by image-level and pseudo-instance-level labels, respectively. As a result, the first MCC task excels at identifying the classes present in the image, while the second one focuses on more accurate instance-level location. However, previous methods rely solely on the second MCC task during inference, overlooking the classification benefits provided by the first task.

In this paper, We present a novel self-classification enhancement and correction (SCEC) framework to overcome these two limitations. To alleviate the classification ambiguities, we introduce a self-classification enhancement module during the second stage, which integrates an extra intraclass binary classification (ICBC) task to bridge the gap between the two distinct MCC tasks. ICBC task aims to enhance the network's discrimination between positive and mislocated samples in a class-specific manner, rather than directly grouping them together with background samples into a single 'background' class, as done in the 2nd MCC. To sufficiently optimize the ICBC classifiers, we generate various types of mis-located samples based on the 2nd MCC results. Furthermore, the ICBC results are utilized in reverse to refine the pseudo labels for the 2nd MCC task, allowing the two tasks to complement each other. To harness the unique strengths of the two distinct MCC tasks, we introduce a selfclassification correction algorithm, which leverages the 1st MCC results to rectify mis-classifications in the detections produced by the second one.

Our primary contributions are summarized as follows:

• We propose a self-classification enhancement module that incorporates both the base multi-class classification and an intra-class binary classification to alleviate the classification ambiguities. These two tasks are carried out in a mutually reinforcing manner.
• We introduce a self-classification correction algorithm to alleviate the mis-classification problem of detections, leveraging the image-level classification strength of the first multi-class classification results during inference.
• Extensive experiments on the prevalent PASCAL VOC 2007 and 2012 datasets demonstrate the superior performance of our framework.

2 RelatedWork

Weakly Supervised Object Detection. Weakly Supervised Object Detection (WSOD) has been widely studied in recent years. The pioneering work WSDDN [Bilen and Vedaldi,

2016] first integrates multiple instance learning into the CNN architecture by designing a two-stream network, i.e. classification branch and detection branch. By combining the results from these two branches, WSDDN converts the WSOD task into a multi-class classification problem for proposals. However, such a solution often struggles to generate accurate detections. To alleviate this problem, OICR [Tang et al., 2017] proposes a two-stage pipeline where WSDDN is utilized as a basic detector, and its results are utilized to generate pseudo seed boxes for training several subsequent online instance classifiers for further refinement. Most recent WSOD approaches are developed based on this pipeline. Some methods improve the detection capability of the basic detector, e.g., adding extra supplement modules for more complete detections [Yan et al., 2019; Yin et al., 2021; Yin et al., 2022], and enhancing the generated image or proposal feature [Ren et al., 2020; Huang et al., 2020]. Some other methods design various strategies to improve the quality of pseudo seed boxes, e.g., constructing spatial graphs [Tang et al., 2018] or appearance graphs [Lin et al., 2020] for topscoring proposals, bringing top-down objectness [Zeng et al., 2019], and applying spatial likelihood voting [Chen et al., 2020]. Otherwise, [Yang et al., 2019] add online regression branches to refine the initial proposals.

Different from them, our method extends the widely used online classifier into a self-classification enhancement module, which brings intra-class binary classification to enhance the network's discrimination between class-specific positive and mis-located samples, thus improving the detection capability of the network.

3 Method

3.1 Overview

The overview of the proposed model is illustrated in Figure 2. First, an image and the generated region proposals are fed to the RoI feature extractor, i.e., CNN backbone and an RoI pooling layer followed by two FC layers, to obtain proposal feature vectors. Next, the feature factors are fed into MIDN module to produce instance-level scores, which are summed for the training of MIDN with image-level labels. Meanwhile, the feature factors are fed into several subsequent self-classification enhancement (SCE) module to obtain MCC and ICBC scores. For each SCE module, the ICBC branch is trained using MCC scores, while the pseudo labels of MCC branch are generated from ICBC Guided Seed Mining (IGSM) algorithm. After that, an R-CNN head is constructed to produce classification scores and regression coordinates. During inference, self-classification correction (SCC) algorithm is applied to generate prediction results.

3.2 BasicWSOD Framework

In the weakly supervised setting, distinguishing between positive and negative proposals directly during training becomes challenging, given the absence of instance-level labels. To overcome this problem, the pioneering work WSDDN [Bilen and Vedaldi, 2016] adopts multiple instance learning into the CNN architecture to convert the WSOD task into the multi-class classification task for proposals. Following recent works [Tang et al., 2017; Ren et al., 2020], we apply WSDDN as our basic detector, referred as Multiple Instance Detector Network (MIDN).

Given an image I, its image-level labels Y = [y1, y2, · · · , yC] ∈ RC×1 is available according to theWSOD setting, where yc ∈ {0, 1} indicates the presence or absence of class c. Its proposals R = {R1,R2, · · · ,RN} are pregenerated from Selective Search [Uijlings et al., 2013] before training. First, the proposal feature vectors are generated through a CNN backbone, an RoI pooling layer [Girshick, 2015], and two FC layers. Next, these vectors are fed into two parallel branches, i.e., classification and detection branches, to obtain proposal scores. For each branch, a scoring matrix xcls (xdet ) ∈ RC×|R| is first obtained by an FC layer, where |R| and C represents the number of proposals and categories, respectively. The two scoring matrices are then normalized by the softmax operation through orthogonal directions, i.e. σ(xcls ) for category direction and σ(xdet ) for proposal direction. After that, the proposal scores are generated by the element-wise product of these two matrices: xbox = σ(xcls )⊙σ(xdet ). Finally, the image-level scores are obtained by aggregating over the proposal direction of xbox : ximg c = ∑|R| i=1 xbox c,i . In this way, the image-level labels can be utilized for supervision through binary cross-entropy loss: LMIDN = - ∑Cc=1 [yclogximg c + (1 - yc) log (1-ximg c )] .

Following OICR [Tang et al., 2017], the basic WSOD framework adds several online instance classification (OIC) branches after the basic detector to generate more accurate detections. Each branch contains an FC layer and a softmax operation and outputs a scoring matrix xoic ∈ R(C+1)×|R| . The top-scoring proposals from the C-th branch are utilized to generate pseudo labels yoic to train the C + 1-th branch, through the cross-entropy loss: LOIC = - 1 |R| ∑|R| i=1 ∑C+1 c=1 wiyoic c,i logxoic c,i . The loss weight wi, which acts as a confidence score, is obtained from the score of the seed box which has the highest overlaps with Ri. Additionally, following [Yang et al., 2019; Yin et al., 2021], we construct an R-CNN branch subsequently, which contains a classification sub-branch and a regression sub-branch. The weighted cross-entropy loss and smooth-L1 loss are applied to train these two sub-branches, respectively.

3.3 Self-Classification Enhancement

The basic WSOD framework refines the initial detection results by applying OIC branches with multi-class classification. However, as illustrated in Sec. 1, this approach lead to classification ambiguities, where the features of mis-located samples from all classes are pushed together, despite that they are utilized to generate class-specific features during MIDN's training process. Furthermore, such a solution will weaken the model's ability to distinguish the mis-located samples from their closed intra-class positive ones. To this end, we introduce Self-Classification Enhancement (SEC) module to tackle this problem.

The SCE module comprises two parallel branches: one for the base multi-class classification and the other for an enhanced intra-class binary classification. These branches work harmoniously during online training, supplementing each other's strengths.

Multi-Class Classification. The multi-class classification (MCC) layer shares the same structure with the original OIC layer, containing an FC layer followed by a softmax.

Intra-Class Binary Classification. We incorporate the intraclass binary classification (ICBC) task to enhance the network's discrimination between intra-class positive and mislocated samples. To maintain consistency with the MCC layer, we adopt a simple yet effective design to achieve the ICBC task. Specifically, given the proposal feature vector f, the ICBC branch consists of an FC layer to generate score matrices and a sigmoid function for normalization:

where a higher xicbc c,i indicates that the proposal Ri is more likely to be a positive sample for class c, while a lower value suggests the opposite.

Sampling strategy for ICBC task. We adopt MCC results to select training samples U for the ICBC task. Specifically, we first choose a set of top-scoring proposals as the pseudo seed boxes S = {S1, S2, · · · , SN} according toMCC results. After that, we propose different strategies to select positive and mis-located samples based on these seed boxes.

A straightforward way is to apply the IoU sampling strategy. Specifically, for each proposal, we calculate its Intersection over Union (IoU) with all seed boxes, and take the maximum value IoUi. Next, we denote the positive samples as Rpos = {Ri|IoUi ≥ τh} and the mis-located samples as Rneg = {Ri|τl ≤ IoUi < τh}, where IoUi is the overlaps between Ri and its closest seed box Sj. Correspondingly, Ri shares the same category Ci with Sj. Then, we generate the pseudo label Y icbc i = [yicbc 1,i , yicbc 2,i , · · · , yicbc C,i ] of proposal i according to the divisions:

However, the samples selected from IoU sampling strategy are insufficient for achieving the ICBC task. Given the limited number of training samples, we further apply a gridding sampling strategy to generate additional mis-located samples for augmentation. Specifically, for each seed box with class c, we first apply box scaling by a scaling factor θ = 0.5. In this way, the width and height of the seed box are randomly sampled in [(1-θ)w, (1+θ)w] and [(1-θ)h, (1+θ)h], respectively. The center of the box remains unchanged. Afterward, we generate an n × n grid on the scaled seed box. We treat each grid as a potential mis-located sample that may only contain a part of an object. It is worth noting that although some grids containing only background may inevitably be selected, their impact is minimal due to their limited quantity. These selected grids G = {G1,G2, · · · ,GN} are then fed to the RoI pooling layer to generate region features, which are subsequently passed to the ICBC layer. Their pseudo labels are assigned as follows: yicbc c,Gi = 0, Gi ∈ G.

We denote all the selected samples as U = Rpos ∪Rneg ∪G. Considering the division of positive and mis-located samples is class-specific, we utilize sample weight to ensure these samples only participate in the losses of their corresponding categories:

where pi is the confidence of Ri, obtained by the MCC score of its closest seed box Sj, and qi is set to 1.5. Finally, we adopt weighted binary cross-entropy loss for training ICBC layer:

ICBC Guided Seed Mining. Seed box mining plays a significant role in training the MCC branch. Some methods pursue the accuracy of seed boxes by using top-scoring strategies [Tang et al., 2017], while some others focus on the recall of seed boxes by relaxing the top criteria and adopting non-maximum suppression (NMS) algorithm to remove redundant ones [Ren et al., 2020]. Different from them, we adopt a soft scoring threshold to mine accurate seed boxes and utilize ICBC results for further fine-tuning.

Specifically, for each existing class c (yc = 1), we first find the proposal Rk with the top score xmcc c,k . Here, we adopt the scores from the previous MCC branch following [Tang et al., 2017]. Next, instead of selecting top-K proposals, we set a soft scoring threshold according to the top score in this class: τscore = αxmcc c,k . We select the proposals whose MCC scores are higher than the threshold τscore. After that, we apply NMS algorithm on the selected proposals to remove redundant boxes, obtaining the base seed boxes Sbase.

Compared to the MCC, ICBC demonstrates superior proficiency in discerning the positive samples from their closed but mis-located ones in a class-specific manner. To this end, we apply ICBC results to fine-tune the obtained base seed boxes Sbase. Briefly, for each seed box Si in Sbase, we first obtain its surrounding proposals by setting an overlap threshold (τsur = 0.5). Next, among all the surrounding proposals, we select the one Sˆi that has the maximum ICBC score in the class of Si, and add it to the seed boxes. According to the previous comparison between ICBC and MCC, Sˆi can be regarded as a potential refinement of Si in location, which is shown in Figure 3.

Finally, the fine-tuned seed boxes Sft, along with the base seed boxes Sbase, are utilized to train the MCC layer. We generate pseudo labels of all the proposals according to their max overlaps with seed boxes. The overlap thresholds are set the same with those in ICBC training, while the mis-located samples are labeled as C +1. After that, we use these pseudo labels to train MCC layer with weighted cross-entropy loss:

where ymcc c,i and xmcc c,i represent the pseudo labels and MCC score of proposal Ri in class c, respectively. The loss weight wmcc i is the score of the seed box which has the highest overlaps with Ri.

Finally, we replace the original OIC branches with our SEC modules, and we train the network end-to-end by combining all the losses mentioned before:

where LR-CNN is the loss for R-CNN branch, T is the number of online enhanced instance classification modules, and γ keeps the balance between LMCC and LICBC.

3.4 Self-Classification Correction

A common pipeline for WSOD inference involves two main steps: 1) aggregating the online MCC scores (e.g., OICs & classification scores in R-CNN branch) as the final results; 2) refining the boxes through the regression outputs. In contrast, the MIDN result is empirically discarded due to its inadequate detection performance. We often observe a common flaw in such a solution: Some non-exist categories are incorrectly assigned high scores in the final results, leading to many mis-classification samples. We conduct a toy experiment to empirically show the suboptimal classification of final results. We perform statistical analysis on the occurrences of all detected boxes' predicted categories within the image. In other words, if the category appears in the image, the box will be considered a positive sample; otherwise, it is deemed a negative sample. This methodology allows us to calculate the image-level classification (ILC) accuracy of the bounding boxes. We evaluate multiple models at different training iterations, as shown in Figure 4, and compare the detection results from the conventional pipeline (green line) with those fromMIDN (blue line).

Intrigued, two observations stand out: 1) the ILC accuracy of pipeline result falls short of the 50%mark, and 2) the MIDN result exhibits notably higher ILC accuracy compared to the pipeline one. We ascribe this disparity to two main factors. On one hand, only parts of confident proposals participate in the training of the OICs, thus the penalty for nonexistent categories has been attenuated. On the other hand, MIDN employs binary cross-entropy loss for the summation of the instance-level scores, which exerts stronger constraints on the image-level classification.

Based on this observation, we propose a simple yet effective strategy to refine the pipeline results with MIDN, termed as Self-Classification Correction (SCC). Briefly, we first predict the existing image-level categories according to MIDN results through confident proposal selection and an argmax operation, and then obtain the absent categories CN:

where M is MIDN results, C is the set of classes, and τmidn is the scoring threshold for MIDN. Subsequently, we reduce the pipeline scores of these absent categories by multiplying a scoring factor λ.

As shown in Figure 4, SCC algorithm (red line) brings about substantial enhancements in ILC accuracy compared to the pipeline (green line), showcasing improvements ranging from 20% to 35%. Remarkably, pipeline utilizing SCC even achieves superior ILC accuracy when compared to MIDN. Consequently, this refinement strategy contributes to a notable reduction in instances of mis-classification, thus enhancing detector performance without introducing additional training parameters.

Additionally, to make the SCC algorithm more effective, we adopt the misclassification tolerance (MCT) strategy [Wu et al., 2024] to further improve the classification ability of MIDN. The motivation behind the MCT strategy is to introduce tolerance for unrepresentative samples with high semantic similarity to an incorrect class, thereby preventing these samples from dominating the training process and forcing the model to memorize them. Suppose Np classes are present in the image, we identify the misclassified classes containing unrepresentative samples when their score rankings fall within the range of [Np,Np + Tn], and assign their class weights to a when calculating LMIDN. Similarly, the corresponding incorrect classes, whose score rankings fall within the range of [0, Np], are also assigned the same class weight.

4 Experiments and Analysis

4.1 Datasets

Following previous works, we evaluate our proposed method on two popular object detection datasets Pascal VOC 2007 and Pascal VOC 2012 [Everingham et al., 2010], which contain 20 categories. For both two datasets, we train on trainval splits (5,011 images in VOC 2007 and 11,540 images in VOC 2012) and applies two kinds of metrics for evaluation: (1) The mean of average precision (mAP) on the test split (4,951 images in VOC 2007 and 10,991 images in VOC 2012); 2) Correct localization (CorLoc) on the trainval split. Only imagelevel labels are utilized during training.

4.2 Implementation Details

Following a widely-used setting, we adopt VGG16 [Simonyan and Zisserman, 2014] pre-trained on ImageNet [Deng et al., 2009] as the backbone and Selective Search [Uijlings et al., 2013] for proposal generation. The whole framework is end-to-end optimized using stochastic gradient descent (SGD), and the momentum, weight decay, and batch size are set as 0.9, 5 × 10-4 , and 4, respectively. The initial learning rate is set as 1 × 10-3 for the first 70k, 170k iterations, and it is dropped by a factor of 10 for the following 20k, 40k iterations for VOC 2007 and VOC 2012, respectively. We set α to 0.9 and the NMS threshold τnms to 0.1 in the SEC module. λ and τmidn in SCC algorithm are set to 0.01 and 0.001, respectively. The hyperparameters of MCT strategy are set as the same with [Wu et al., 2024], i.e., Tn = 1, a = 0.4. The loss weight γ is set to 0.1 for the training balance. Following the previous WSOD works, τl, τh, and T are set to 0.1, 0.5, and 3, and the images are multiscaled to {480, 576, 688, 864, 1000, 1200} for both training and inference.

4.3 Comparison with State-of-the-art Methods

In Table 1, we compare the performance of the state-of-art methods with single model on Pascal VOC 2007 and VOC 2012 datasets in terms of mAP. Our method achieves stateof-the-art performance with 58.2% mAP on VOC 2007 and 55.5% mAP on VOC 2012, surpassing other methods by at least 0.8% and 0.9%, respectively. Our method also achieves outstanding performance in terms of CorLoc, setting new state-of-the-art benchmarks with 71.9% on VOC 2007 and 73.4% on VOC 2012. Our method outperforms recent works [Ren et al., 2020] and [Chen et al., 2020] that improve the seed-box mining but remain reliant on the multi-class classification scores. On one hand, our method considers the scoring variance among different categories and images, thus making a reasonable balance between the accuracy and recall of seed boxes. On the other hand, we use the ICBC scores for further refinement, thus improving the localization accuracy of these seed boxes. Some other methods [Lin et al., 2020; Zeng et al., 2019] also apply other information for assistance to improve the seed-box mining, however, their improvement limits on the online training procedure. Instead, our method brings intra-class binary classification (ICBC) task to directly enhances the network's discrimination between intra-class positive and negative samples, thus benefiting the feature representation of the whole network.

4.4 Ablation Study

Effect of Each Component. We present experimental results on VOC 2007 to validate the effectiveness of each component, as summarized in Table 3. Starting with the basic WSOD framework, referred to as the "baseline", we achieve an initial mAP of 54.6%. Incorporating the ICBC task into the WSOD framework leads to a notable improvement of 0.6%, underscoring the effectiveness of the ICBC branches in enhancing the network's ability to distinguish between positive and mislocated samples in a class-wise manner. Subsequently, integrating the IGSM algorithm further improves seed box quality, boosting performance to an mAP of 56.7%. Overall, the self-classification enhancement module delivers a significant mAP gain of 2.1%.

To evaluate the self-classification correction module, we first apply the self-classification correction algorithm during inference, resulting in a clear 0.8%mAP improvement. This gain can be attributed to the algorithm's capability to effectively reduce high-scoring misclassified samples. By introducing the misclassification tolerance strategy, we achieve the highest performance of 58.2% mAP. These results demonstrate that enhancing the classification performance of MIDN can further expand the potential upper limit of the selfclassification correction algorithm.

Figure 5 shows the detection results of different models on VOC 2007 test set. Compared with the baseline model (1st row), the integration of the OEIC module (2nd row) largely alleviates the mis-location problem in different cases, including localizing only discriminative parts (a) and containing redundant areas (including background and other objects) (b). Furthermore, more missing objects are detected (c) due to the design of the IGSM algorithm in SEC module. Lastly, as shown in (d), the utilization of the MSCR algorithm can further alleviate the mis-classification problem.

Effect of ICBC sampling strategy. We conduct experiments by employing different sampling strategies for the training of ICBC task, as shown in the upper part of the Table 4. Among all the settings, the combination of the IoU sampling strategy and the gridding strategy achieves the best performance and is insensitive to variations in grid size.

Effect of IGSM algorithm. We conduct experiments to evaluate each component of IGSM algorithm, as shown in the lower part of the Table 4. On one hand, we first replace our scoring strategy with original top-1 strategy, which only selects the proposals with highest scores as seed boxes, resulting in a 1.0% mAP drop, as our scoring strategy more effectively locates various objects using class-wise soft thresholds. Additionally, removing ICBC-guided fine-tuning causes a 1.2% mAP drop, highlighting the superior capability of ICBC in distinguishing positive samples from closely related but mislocated ones.

5 Conclusion

We propose a novel framework for weakly supervised object detection to address the limitations of the two-stage multi-class classification pipeline. On one hand, we introduce a self-classification enhancement module that enhances the network's discrimination between intra-class positive and mis-located samples, and leverage it to enhance the quality of seed boxes. On the other hand, we introduce a self-classification correction algorithm to fine-tune the online classification scores, significantly reducing mis-classification detections. Extensive experiments on the widely used VOC datasets demonstrate the effectiveness of our framework.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants (62422204, 62472139) and in part by the Zhejiang Provincial Natural Science Foundation of China under Grant LQN25F030014, LDT23F02025F02. This work is also supported by the Open Project Program of the State Key Laboratory of CAD&CG (Grant No. A2403), Zhejiang University.