Augmenting Anchors by the Detector Itself

1 Introduction

Object detection, which needs to localize and recognize each object in images or videos, is a fundamental and critical problem in computer vision. With the development of deep learning, current object detectors are usually based on convolution neural networks, like FCOS [Tian et al., 2019], RepPoints [Yang et al., 2019; Chen et al., 2020], TOOD [Feng et al., 2021], Faster R-CNN[Ren et al., 2016], DETR [Carion et al., 2020], DeFCN [Wang et al., 2021], and Sparse RCNN [Sun et al., 2021]. They can be roughly classified into anchor-free methods and anchor-based methods. Anchorbased methods firstly place a lot of anchors with different scales and aspect ratios, and then refine these anchors to generate proposals or detection results, while anchor-free methods directly generate detection results from feature maps.

* These authors have contributed equally to this work.

† Corresponding author.

The anchor-free methods were once widely regarded as the standard paradigm for future object detectors. However, ATSS [Zhang et al., 2020] has demonstrated that anchor-free methods are not more effective than anchor-based ones. For example, with same tricks and appropriate strategies for judging positive and negative samples of hand-designed anchors, the anchor-based ATSS can perform better than anchor-free FCOS. Based on ATSS, PAA [Kim and Lee, 2020] proposes a method of judging positive and negative samples by Gaussian Mixture Model and achieves large improvements, which further proves the conclusion mentioned above. Both of these two models use a method to estimate the critical point of the distribution of positive and negative anchors. However, even if the best critical point is estimated, there is no guarantee that the positive and negative samples will be completely separated, and placing more hand-designed anchors can not solve this problem. Therefore, how to improve the quality of anchors has become an important problem. 1

In current anchor-based object detectors' paradigm, highquality anchors usually lead to better performance, because they make it easier to separate positive and negative anchors, which is important to improve the recall rate of ground truth boxes and reduce false positives. In standard RPN, anchors with an IoU greater than 0.7 are marked as positives, those with an IoU below 0.3 are marked as negatives, and others are ignored in training. Thus, placing more anchors with different aspect ratios and scales will improve the performance of RPN. However, since the aspect ratio and scale can be any positive number, it makes the anchor designation a nontrivial problem. Thus, some methods try to learn high-quality anchors by a neural network module, like AttratioNet [Gidaris and Komodakis, 2016], GA-RPN, and Cascade RPN. All these methods can be viewed as gradient-based anchor augmentation methods, and their overall training and inference overheads almost double due to their strategy of dense prediction and large memory accesses in feature alignment.

1 More details can be found in https://arxiv.org/abs/2105.14086

In this paper, we propose a gradient-free anchor augmentation method named AADI, which means Augmenting Anchors by the Detector Itself. AADI can be applied to both two-stage and single-stage methods, and Figure 1 shows an example of AADI for RPN [Ren et al., 2016]. AADI comprises two processes, including augmentation and refinement process. During training process, AADI first uses RPN to augment hand-designed anchors to improve their quality in augmentation process, and then uses the augmented anchors and ground truth to train the parameters of RPN in refinement process. Intuitively, AADI can be viewed as a variational EM algorithm which is typically used to solve the problems with latent variables, and the augmentation and refinement process are corresponding to its E-step and M-step, respectively. Here, we iterate it only once to improve the computational efficiency. Moreover, in the augmentation process, AADI uses dilated convolution [Yu and Koltun, 2015] to augment each hand-designed anchor, and in the refinement process, it employs RoI Align [He et al., 2017] to extract features for augmented anchors and uses an equivalent fully connected layer to refine these anchors. Note that the dilated convolution and fully connected operations share parameters.

In order to make RPN's parameters be reused in these two processes, we need to carefully design the dilation and kernel size of convolution layer to ensure that the convolution layer and fully connected layer could obtain same input feature for a specific anchor. As Figure 2 shows, supposing that the dilation of a convolution layer is d, then the process of a 3×3 convolution layer is same as that of an RoI Align for a (3×d)×(3×d) anchor. Furthermore, the operation of a 3×3 convolution layer for each anchor is equal to a fully connected layer whose output dimension is 9. Thus, unlike the ineffective masked deformable convolution layer in GA-RPN or adaptive convolution layer in Cascade RPN which is used for tackling the feature misalignment problem of hand-designed and learned anchors, AADI avoids this problem by determining the scale and aspect ratio of anchors according to the dilation and kernel size of convolution operation. Consequently, the hyper-parameters of AADI are kernel size and dilation of convolution layer, that is, AADI converts the scale and aspect ratio of anchors from a continuous space to a discrete space, which greatly alleviates the problem of anchor designation.

During training, the augmentation process does not need to calculate gradient, which makes the training speed of AADI fast. During inference, AADI only takes top 2000 augmented anchors for further refinement, which makes the inference speed fast. Extensive experiments on MS COCO dataset show that AADI can efficiently improve the detection performance of anchor-based object detectors, specifically, AADI achieves at least 2.4 box AP improvements on Faster R-CNN, 2.2 box AP improvements on Mask R-CNN, 1.8 box AP improvements on RetinaNet, and 0.9 box AP improvements on Cascade Mask R-CNN. And it runs only slightly slower than standard RPN. Our main contributions are as follows:

• We propose a novel and effective method, AADI, which augments anchors by the detector itself and can significantly improve anchors' quality. It can be applied on all anchor-based object detectors without adding any parameters or hyper-parameters.
• AADI determines the scale and aspect ratio of anchors according to the kernel and dilation of a convolution layer, and it determines the aspect ratio according to the shape of the kernel, which greatly eases the anchor design difficulty.
• We carry out experiments on MS COCO dataset to verify the effectiveness of AADI. With only little extra computation costs, AADI achieves significant performance improvements on many representative object detectors with different backbones.

2 RelatedWorks

Current object detection methods can be broadly categorized into anchor-based methods and anchor-free methods. Anchor-based methods firstly place many hand-designed anchors on the feature maps, and then perform classification and box regression on these anchors. Therefore, better anchors can greatly alleviate the difficulty of these two tasks. AttratioNet [Gidaris and Komodakis, 2016] uses ARN to iteratively refine the hand-designed anchors several times. However, ARN needs to employ RoI Pooling to extract features for each anchor, and it is not a lightweight network, which makes its training and inference speed slow. GA-RPN [Wang et al., 2019] first predicts a suitable anchor for each position, and then uses the predicted anchors to train RPN. Cascade RPN [Cai and Vasconcelos, 2018] stacks two RPNs together to improve the quality of proposals. Its first RPN is used to refine the hand-designed anchors without classification, and the second one plays the same role as a standard RPN. GARPN and Cascade RPN have effectively improved the quality of proposals by improving the quality of anchors. However, they both need to use DCN[Dai et al., 2017] or its variants to align the features and anchors, and there is a lot of memory access during this operation, resulting in slower training

Algorithm 1 Pseudocode of AADI

Input:

• A is a set of hand-designed anchors;
• Aa is a set of augmented anchors;

G is a set of ground truth boxes on the image;

F is feature maps obtained by backbone;

f is RPN;

θ is a set of paramters of RPN;

• ϵ is learning rate;

Output:

P is a set of proposals;

• 1: while not converge do
• 2: Aa ← f(A,F, θ)
• 3: P ← f(Aa,F, θ)
• 4: L← loss(P, G)
• 5: △θ ← ∂L ∂f(Aa,F,θ) × f(Aa,F,θ) ∂θ
• 6: θ ← θ - ϵ ×△θ
• 7: end while

and inference speed. In this paper, we first filter a small set of augmented anchors, and then use RoI Align to extract the features for augmented anchors, which can significantly reduce the memory access.

Compared with anchor-free methods, anchor-based methods are more intuitive, and they do not need to determine which ground truth should be placed on which level of pyramid features. However, FCOS [Tian et al., 2019] finds that determining the scale and aspect ratio of anchors is not a trivial issue, and these two parameters have a great influence on the performance. Thus, FCOS proposes an anchorfree paradigm to generate the final results. For each position which is located in the center region of the ground truth, it directly predicts the distance from the position to the four corners of the ground truth. RepPoints [Yang et al., 2019] is also an anchor-free method. It no longer directly predicts bounding boxes, but predicts a set of representative points for each ground truth, and then groups them to bounding boxes. DETR and Deformable DETR [Zhu et al., 2020] use Transformer to directly predict the detection results to avoid the anchor designation problem.

Although the hyper-parameter adjustment of anchor-free methods is simpler, ATSS and PAA demonstrate that anchorbased methods can significantly outperform anchor-free methods with suitable anchor assignment strategies. However, the anchors of both ATSS and PAA are predefined, although both ATSS and PAA are very robust to the scale and aspect ratio of anchors, they are hard to further improve the performance by placing more anchors with different scales and aspect ratios. However, AADI can adaptively augment the hand-designed anchors to improve their quality.

3 Method

3.1 AADI

High-quality anchors are critical for all anchor-based object detectors, so, how to adaptively generate better anchors is an important factor. Unlike previous gradient-based methods, AADI just uses the detector itself to augment the handdesigned anchors without gradient, and it can be used in both two-stage and single-stage object detectors. The overall pipeline of AADI for a common two-stage object detector is shown in Figure 3(b), and its Pseudocode is shown in Algorithm 1. Compared with AttratioNet, AADI does not need to calculate gradient during anchor augmentation. And compared with the IBBR(Iterative Bounding Box Regression) of IoU-Net [Jiang et al., 2018], which has a similar inference pipeline. The key difference is that AADI works on both training and inference period, while IBBR only works on inference. Taking AADI for two-stage object detectors as an example, AADI comprises two processes, including an augmentation process and a refinement process, and the details are illustrated as follows.

Augmentation process. As shown in Figure 3(b), AADI first gets the input of anchors and feature maps from the backbone, and then uses RPNc to augment the anchors. RPNc is same as the standard RPN, and it does not need to calculate the gradient during training. As most of augmented anchors are either redundant samples or negative samples, so we remove them by Non-Maximum Suppression (NMS) and keep only 2000 anchors with the highest score to improve the model's efficiency. As shown in Figure 1, the augmented anchors have a higher Intersection over Union (IoU) than the original anchors.

Note that there can be multiple RPNc with different dilations, and these RPNcs can share parameters.

Refinement process. The refinement process gets input of augmented anchors from the augmentation process and feature maps from the backbone, and then uses RPNf to predict proposals. RPNf and RPNc have the same parameters, but they are implemented in different ways. RPNc is same as the standard RPN, and it uses convolution to get augmented anchors, while RPNf is implemented in a fully connected way. In addition, RPNf employs RoI Align to extract features for each augmented anchor. Note that different RPNcs are corresponding to different RPNfs if their parameters are not sharing. Moreover, RPNf needs to calculate gradient during training, so RPNc will also be trained indirectly.

The key problem of ensuring that this paradigm can work is the features obtained by the two RPNs should be same. For RPNc, the features for each anchor are determined by the kernel size and dilation of a convolution layer, while for RPNf , the features are determined by RoI Align. So, there are two conditions to meet. First, the output size of RoI Align needs to be same as the kernel size of the convolution layer, which is easy to meet. Second, the dilation and the scale of each anchor must be carefully designed to align the RoI, which is hard to meet.

To address this issue, we propose to set scale and aspect ratio of hand-designed anchors by Eq. 1 and Eq. 2, respectively. Where m and n are height and width of kernel size of a convolution layer, and d is the dilation. In Figure 2, we visualize the anchors and their features sampled by a 3×3 convolution layer under different dilations, and it is easy to find that the features sampled by a 3×3 convolution layer are same as the features extracted by a 3×3 RoI Align. In this way, the aspect ratio of the anchors is also determined by the shape of the ker-

• (c) Box subnet of RetinaNet.
• (d) Box subnet of AADI-RetinaNet.

Figure 3: The pipeline of AADI for RPN and RetinaNet.

nel. For example, for a 3×3 convolution layer, if its dilation is set to 2, then the scale of the anchors is 3 × 2 = 6, and the aspect ratio is 33 = 1. Since m, n and d all are must be integers, so the designation space of anchors is converted from a continuous space to a discrete space, which greatly alleviates the problem of anchor designation.

balanced by λ. In the implementation, binary cross entropy loss and Smooth L1 Loss [Girshick, 2015] are used as classification loss and box regression loss, respectively, and λ is empirically set to 5.

3.3 Inference

During inference, the augmentation process is same as that of obtaining proposals by standard RPN, and the refinement process is just a two-layer fully connected layer which is used to refine the augmented anchors. Here, AADI only takes the top 2000 augmented anchors for refinement, while Cascade RPN needs to augment all regressed anchors, which leads to higher computation overheads.

3.4 AADI-RetinaNet

AADI can be naturally deployed in anchor-based single-stage object detectors. We only apply AADI on RetinaNet (AADIRetinaNet) due to the limited time and resources. Since RetinaNet uses 4 multi-layer convolution networks to extract features for classification and regression tasks respectively, and the kernel of all convolution layers is 3×3, so we can use the last layer of the box subnet to extract features for the augmented anchors. The pipeline of AADI-ReitinaNet are shown in Figure 3(d), and the training process of AADI-RetinaNet is similar to AADI-RPN, However, during inference, since NMS is relatively time-consuming, we only select the top 2000 augmented anchors without NMS to do further refinement. Here, the IoU threshold of NMS of post-processing is set to 0.6. Furthermore, Group Normalization [Wu and He, 2018] is used to make the training process smoother.

3.2 Training

If AADI is only trained on the augmented anchors, AADI's robustness to hand-designed anchors is relatively poor, which will reduce the performance of anchor augmentation. Therefore, during training, we first select an anchor with the largest IoU for each ground truth, then append them to the augmented anchors, which can guide the AADI to augment the hand-designed anchors, we call this strategy Anchor Guided.

The number of positive augmented anchors is very large, if all of them are used directly, the memory will be exhausted. In addition, we found that the overlap of some augmented anchors is large. Therefore, we employ NMS to filter out the redundant anchors according to their IoUs with the ground truth, and the training strategy of the refinement process is same as standard RPN. Similar to RPN, AADI can be trained in an end-to-end manner using multi-task loss as follows:

Where Lreg and Lcls are the box regression loss and the classification loss, respectively. And these two loss terms are

4 Experiments

4.1 Dataset and Evaluate Metrics

Our experiments are implemented on the challenging Microsoft COCO 2017 [Lin et al., 2014] dataset. It consists of 118k images for training, train-2017 and 5k images for validation, val-2017. There are also 20k images without annotations for test in test-dev.

We train all our models on train-2017, and report ablation studies on val-2017 and final results on test-dev. In this paper, Average Recall(AR) is used to measure the quality of proposals, which is the average of recalls at different IoU thresholds, and ARx means recall at IoU threshold of x%, ARs, ARm, and ARl denote the AR for small, medium, and large objects, respectively. The detection results are subjected to standard COCO-style Average Precision (AP) metrics which include AP (mean of AP over all IoU thresholds), AP50, AP75, APs, APm and APl, where AP50 and AP75 are APs for IoU threshold of 50% and 75%, respectively, and APs, APm and APl are APs for small, medium, and large objects, respectively. Moreover, AP box and AP mask are denoted as AP for object detection and instance segmentation, respectively.

4.2 Implementation Details

All experiments are based on detectron2 [Wu et al., 2019]. And our codes are deployed on a machine with 8 Tesla V100SXM2-16GB GPUs and Intel Xeon Platinum 8163 CPU. Our software environment mainly includes Ubuntu 18.04 LTS OS, CUDA 10.1, and PyTorch [Paszke et al., 2017] 1.6.0.

The hyper-parameters of our model are following detectron2, and all models are based on FPN [Lin et al., 2017]. For AADI-RPN, the dilations of RPNc is set to 2 and 4, and these two RPNc are not sharing parameters, moreover. For AADI-RetinaNet, the dilation is set to 2. Moreover, the strategy of Anchor Guided is used in all models.

As a common practice [Girshick et al., 2014], all network backbones are pretrained on ImageNet1k classification set [Deng et al., 2009], and their pre-trained models are publicly available. With 1x [Wu et al., 2019] training schedule and ResNet-50 (R-50) as backbone, it takes about 4.2 hours for our model to converge, and about 2.9 hours with Mixed Precision training [Micikevicius et al., 2017].

4.3 Results

Region Proposal Performance. We first apply AADI on RPN (AADI-RPN) of Faster R-CNN to verify its performance on region proposal, and the model is trained on COCO train-2017 with 1x training schedule. Since it is difficult to reproduce the performance of AttratioNet, ZIP[Li et al., 2019] and C-RPN [Fan and Ling, 2019], we only compare recall rate with published results. For other models, their performance is tested on the same machine, and the training time indicates the average time cost per batch, while the test time indicates the average inference speed on COCO val2017. All region proposal results are shown in Table 1, and the results show that AADI-RPN, by a large margin, outperforms standard RPN in terms of AR under all settings, and it achieves the highest AP box over other models. Furthermore, although AADI-RPN has a lower recall rate than the current state-of-the-art Cascade RPN, its inference speed is about 70% faster. For training time, AADI-RPN achieves the fastest training speed, even faster than RPN. This is mainly because only a small set of the augmented anchors are participated in the training process, and the way to compute the gradient is changed from convolution to fully connected operation, which significantly reduces the training overheads.

Object Detection Performance. We conducted extensive experiments on four typical object detection frameworks: RetinaNet, Faster R-CNN, Mask R-CNN, and Cascade Mask R-CNN, as well as two different backbone networks: R-50 and tiny Swin Transformer (Swin-T) [Liu et al., 2021] to verify the anchor augmentation capabilities of AADI. All detection results with 3x training schedule are shown in Table 2, and we can see that AADI achieves significant performance boosts in all detectors with different backbones, which demonstrates its strong robustness. In particular, when using Swin-T as the backbone, we observe +2.1 AP box , +2.7 AP box , +2.4 AP box , and +0.9 AP box gains over RetinaNet, Faster R-CNN, Mask R-CNN, and Cascade Mask R-CNN, respectively. Furthermore, their AP mask are also improved. It is worth noting that the Swin Transformer based object detectors are currently state-of-the-art, while AADI still achieves significant performance boosts with only little extra computation costs, which reflects the efficiency of AADI. Meanwhile, we can find that the performance gains on AP box 50 are small, while large gains are obtained on AP box 75 , similar finding could be found for GA-RPN and Cascade RPN. Therefore, the main advantage of AADI is to improve the quality of anchors, thereby improving the detection performance.

5 Ablation Study

We conduct our ablation study on COCO train-2017, and report the results on COCO val-2017 dataset. All models' backbone is R-50 unless specially noted.

It can be seen from Eq. 1 and Eq. 2 that the scale and aspect ratio of anchors is determined by the kernel and dilation of a convolution layer. For RPN, since the kernel is usually fixed to 3 × 3, so we only change the scale of anchors by adjusting the dilation. Here, we test the region proposal performance of RPN with dilation of 2, 3 and 4, respectively. Furthermore, the effects of the proposed training strategy of Anchor Guided in augmentation process are also tested. All results are shown in Table 3.

From Table 3, we can conclude that larger dilation, worse performance of small objects, and better performance of large objects, and vice versa. This is because the scale of anchors will become larger as the dilation becomes larger, and large anchors are beneficial for large objects, but harmful for small objects during training. Furthermore, the strategy of Anchor Guided can consistently improve the performance of AADIRPN with different dilations, especially for large objects. Compared with standard RPN, AADI-RPN consistently gets better performance by a large margin under different settings, which indicates that AADI is an effective method.

We find that there is not much difference in the performance of AADI-RPN with different dilations. Therefore, we test detection performance to determine which dilation or its combination is the optimal choice. We further test whether the parameters are shared or not when using different dilation combinations. All detection results are shown in Table 4, and the strategy of Anchor Guided is used for all tests.

From Table 4, we empirically find that setting dilation to 3 ( the scale of anchors is 9 ) achieves the best detection performance when only one dilation is used, which may correspond to the dominate scale of objects in the dataset. We also observe a small performance gap among different dilations, which indicates the robustness of AADI. Moreover, since different dilations correspond to different scales of anchors, we expect that using two or more RPNs with different dilations will bring further performance gains. Therefore, we deploy two or more RPNs with different dilations in AADI and test the performance of these different dilation combinations. The results show that when two RPNs share the same parameters, there is almost no performance gain compared to using only one RPN. When the parameters are not shared, the performance can be substantially improved, and we can obtain the highest AP of 41.3 when the dilations of two RPNs are set to 2 and 4, respectively. On top of this, we add another RPN with dilation of 3 to test whether the performance can be further improved. However, a lower AP of 41.0 is obtained, probably due to newly added RPN parameters raising the difficulty of optimization. Moreover, compared with Faster R-CNN, our best practice of AADI achieves 3.4 AP gains.

6 Conclusion

In this paper, we propose a gradient-free anchor augmentation method named AADI. It does not add any parameters or hyper-parameters, and it does not significantly affect training and inference speed. We conduct experiments on RetinaNet and different R-CNNs to verify its performance. Specifically, AADI brings at least 2.2 box AP improvements on Faster/- Mask R-CNN, 1.8 box AP improvements on RetinaNet, and 0.9 box AP gains on Cascade Mask R-CNN. In our future work, we will try to combine AADI and other anchor-based methods together to further improve detection performance.

Acknowledgements

This work was partially supported by The Key Program of National Natural Science Foundation of China (No. U1903213), the Natural Science Foundation of China (No. 61772296) and Shenzhen Fundamental Research Fund (No. JCYJ20170412170438636).