Corner Affinity: A Robust Grouping Algorithm to Make Corner-guided Detector Great Again

1 Introduction

We roughly classify existing object detectors into centerguided [Ren et al., 2015; Cai and Vasconcelos, 2018; Lin et al., 2017b; Tian et al., 2019; Zhou et al., 2019; Carion et al., 2020; Liu et al., 2021] and corner-guided [Law and Deng, 2018; Duan et al., 2019; Dong et al., 2020], among which we argue the corner-guided manner is more bionic in the production of an object bounding box.

Center-guided methods usually model object upon one center along with width and height yet corner-guided ones utilize an instance via paired of corner keypoints. Intuitively, when we want to obtain high-quality bounding boxes manually, we usually label each box from the top-left corner to the bottom-right one and we rarely label the center point directly. This is because the degree of freedom to determine a corner is lower than a center (a corner needs two boundaries vs. a center requires four boundaries) and thus a corner can more easily confirm the objects' boundaries. Accordingly, cornerguided detector enjoys potential ability to yield higher-quality detection boxes. Therefore, we believe this type of detector existing a higher potential deserves to be explored further. However, the challenging corner grouping process hinders the development of this method.

∗This work was done when the first author was interning at Intel Labs China.

†Equal contribution

‡Corresponding author

Although previous methods [Law and Deng, 2018; Duan et al., 2019; Dong et al., 2020] have made great progress in corner grouping, they also face their distinctive shortcomings. As shown in Figure 1, CornerNet, whose corner grouping method relies on local response of object feature, fails to distinguish corners of different objects with similar appearance. Suppose we copy-and-paste an instance to form a new input, CornerNet cannot work because the exact same instance appears twice. Upon CornerNet, CenterNet filters out falsepositives (FP) via center points yet cannot solves the scenario that a center point of the third object lies exactly within the center of an FP. However, this scenario often occurs in some datasets, such as aircraft parked side by side in aerial images. CentripetalNet cannot address the object occlusion problem due to the center overlapping, e.g., pedestrian detection scenario. In summary, existing heuristic corner grouping methods are dataset (COCO [Lin et al., 2014]) driven with poor generalization, but much less effort has been paid to solve it. To tackle this problem, we propose the Corner Affinity, whose motivation is to cover all scenarios aforementioned and bring robust grouping capability for corner-guided detector.

Specifically, Corner Affinity embeds three attributes for each target corner, i.e., locations, shapes, and semantic. Locations and shapes are embedded in the structure affinity (SA) that is a key part of the Corner Affinity, in which we encode the shape (e.g., width and height) knowledge of each instance in its corner location via a strong supervision manner. SA only embedded low-level construction similarity of corners is not enough to accomplish grouping in some extreme scenarios, such as two objects with similar shapes coincide. To this end, we devise the contexts affinity (CA) part for Corner Affinity, in which we utilize the pull-push [Newell et al., 2017; Law and Deng, 2018] algorithm to mine high-level semantic similarity of the corresponding corner pairs to make coincident objects distinguishable. In the CA part, the affinity value is encoded upon the contexts response of instance in a self-supervision manner with no need for a real ground-truth value. Besides, to make the two affinities better interact, we devise the Corner Affinity function, which runs as coupling CA and SA. In this way, SA and CA interplay so that even if the SA value of two corners belong to different instances is large, CA will decay it to make sure that the value of overall Corner Affinity is low, vice versa.

We select three datasets covering different detection scenarios, i.e., COCO [Lin et al., 2014], Citypersons [Cordts et al., 2016], and UCAS-AOD [Zhu et al., 2015], to verify the effectiveness of our design. Experimental results show that the proposed Corner Affinity boosts AP of 5.8%, 35.8%, and 17.2%, on the above three benchmarks upon CornerNet baseline, proving the solid effectiveness of our design. We hope the efficient Corner Affinity can attract more attention to promote the development of bionic corner-guided detector.

2 RelatedWork

2.1 Center-guided Detector

Center-guided detectors generally use potential center points/areas, acting as reference positives, to regress the object bounding box (e.g., height and width). Classical anchorbased detector belongs in this field, which regards center as an attribute of the preset anchors. Faster R-CNN [Ren et al., 2015] popularizes the center-guided anchor mechanism in its Region Proposal Network (RPN). The aim of RPN is generating a few of proposals from a set of candidate boxes that are encoded via their centers along with heights and widths. In addition, the design of the center-guided RPN makes the detector can be end-to-end trainable. Afterwards, the centerdriven anchor boxes are widely used in RPN-based two-stage detectors. To further explore the efficiency of models, some center-guided one-stage detectors also appeared. They remove the RPN and directly do regression and classification at the centers of anchor boxes. For example, SSD [Liu et al., 2016] improves the performance via densely placing center anchors in multiple layers.

Unlike the center-guided anchor-based detectors [Redmon and Farhadi, 2017; Liu et al., 2016; Lin et al., 2017b], FCOS [Tian et al., 2019] proposes a center-guided anchor-free manner. It treats lots of pixels in center areas of bounding boxes as positive samples and directly regresses four vectors (the distances from each pixel to the corresponding box's borders) without anchor guidance. Instead of four vectors, Reppoints [Yang et al., 2019] predicts a set of representative points in the center area (positive pixels).

Despite the center-guided mechanism's great success, it is actually difficult to pinpoint the center of a box. This is because a center requires to be determined via all four boundaries of the object, needing four degrees of freedom.

2.2 Corner-guided Detector

Corner-guided detectors usually predict corners via outputting heatmaps, which we argue is more bionic in the generation of the object bounding box. Intuitively, when we want to obtain high-quality bounding boxes manually, each box is labeled from the top-left corner to the bottom-right one and we rarely label the object box by a center point along with height and width. A corner only needs two boundaries (degrees of freedom) to be determined.

CornerNet [Law and Deng, 2018] detects objects by predicting and grouping pairs of corner points. The grouping method in CornerNet is that if a top-left corner and a bottomright corner belong to the same object, the distance between their embedding vectors will be small. To further optimize this grouping strategy, CenterNet [Duan et al., 2019] adds a prediction branch of center points based on corners pairs estimation, making the corners matching become triplets matching. CentripetalNet [Dong et al., 2020] proposes a new centripetal grouping approach to match paired corner points and achieves state-of-the-art performance.

To sum up, corner grouping is meaningful yet challenge for this kind of detector. Our Corner Affinity aims to produce more robust corner pairs to further advance the development of such human-like corner-guided detector.

3 Method

As shown in Figure 2, the execution of Corner Affinity requires an output of the corresponding corner affinity map. Each (top-left or bottom-right) map is composed of three channels, i.e., two for encoding the structure affinity (SA, blue arrow-lines) and one for the contexts affinity (CA, green arrow-lines). The SA and CA interact via a devised function to be the overall Corner Affinity. In the following subsections, we will detail the SA, CA, and Corner Affinity function.

3.1 The Structure Affinity

The structure affinity (SA) is a key part of Corner Affinity, aiming to mine preliminary construction similarity of cor- Corner Affinity ner pairs through the corresponding object's shallow structure knowledge. We define the shape and location of an instance as the structure knowledge. To obtain that, we regress the shape (e.g., width and height) information of each instance at the ground-truth corner location, as shown in Figure 2 (the blue arrow-line).

Figure 2: Diagram of the proposed Corner Affinity. Equipped with Corner Affinity, the model will output two corner affinity maps for top-left and bottom-right corner keypoints, respectively. Each corner affinity map is composed of three embedding dimensions: two for the structure affinity (SA) and one for the contexts affinity (CA). The SA and CA are coupled via designed functions to constitute the overall Corner Affinity. By the distance metric of Corner Affinity, estimated corner points can be accurately matched to be detection boxes.

Specifically, the network needs to generate two (top-left and bottom-right) 2-dimension (width and height) regression maps. Note that we utilize regression to encode SA values, which is very different from popular detectors that use regression to obtain detection boxes. We adopt the smoothL1 [Ren et al., 2015] to mine the shape knowledge of each instance:

where ˆwk and ˆhk are ground-truth width and height vectors, and ⃗ wk and ⃗ hk are predicted ones. N is the number of instances. The log is the logarithmic function used to constrain the length of height and width vectors to allow the learning process easier.

After obtaining the SA regression map, we can decode the predicted width (⃗w) and height (⃗h) vectors at each estimated corner location. As shown in Figure 3, the ⃗w and ⃗h can not only form a rough object box but also decode a new vector that point to the opposite corner. Accordingly, we design the structure affinity (SA) as follows:

where the boxtl and boxbr represent boxes formed via the corresponding regressed ⃗w and ⃗h. d means the value of corner drifting from the end of decoded vectors to target corners.

And D represents the distance value of two predicted corners. More details are shown in Figure 3.

As described in Eq. 2, the SA is composed of the IoU (Intersection-over-Union) of two formed boxes and a bias named corner drifting. It is intuitive and reasonable that if different identity corners (top-left and bottom-left) belong to the same instance, their formed boxes will overlap significantly. Thus, we utilize the IoU as the basic distance metric of SA. However, vanilla IoU cannot measure offsets of the decoded vectors directly. Therefore, we propose the corner drifting as a bias of SA. The corner drifting is the mean of top-left and bottom-right drfitings, as shown in Figure 3. Based on the above design, the value range of SA is -1 to 1 and the closer the value is to 1, the higher the possibility that two corners belong to the same instance.

3.2 The Contexts Affinity

As mentioned in Section 3.1, the structure affinity only embeds low-level construction information, which is not enough to perform grouping under extreme scenarios, e.g., two objects with similar shapes coincide. To this end, we introduce the contexts affinity (CA) part to mine high-level distinguishable semantic knowledge for Corner Affinity.

Inspired by CornerNet, we utilize the Associative Embedding [Newell et al., 2017] method to predict an embedding value for each corner. The value is predicted based on the feature local response via a self-supervision manner, which is no need for a real ground-truth value. More specifically, we use the "pull" loss to close the embedding distance of paired corners and the "push" loss to separate the embedding distance of irrelevant corners as shown in Eq. 3. We don't care what the value of each corner is and just need minimize embedding distances of corners that belong to the same object and maximize those of different ones. Thus, each embedding without a real ground-truth mine the high-level semantic knowledge of an instance.

where ek is the average of etlk and ebrk . ∆ is set to be 1 following CornerNet. N is the number of objects. Similar to the SA loss, we only apply the CA loss at the ground-truth corner location.

We smooth the distance of embedding to be the contexts affinity. Suppose a top-left corner with an embedding etl and a bottom-right one with an ebr, we define the corresponding CA as follows:

where the tanh function is employed to normalize the distance of embedding values. The value range of CA is 0 to 1, yet unlike SA, that the closer the CA value is to 1, the lower the possibility that two corners belong to the same object.

3.3 The Corner Affinity Function

The overall Corner Affinity is composed of the structure affinity and contexts affinity. To allow these two parts better interact, we devise corner affinity function, which runs as coupling CA and SA, as follows;

where SA and CA are represented in Eq. 2 and Eq. 4, respectively. σ is a manually set Gaussian variance and its value is set to 0.5 empirically.

Upon the above designs, Corner Affinity encodes not only low-level structural knowledge but also high-level semantic knowledge. Even in extreme situation that two objects with similar shape overlap (the SA value of two corners belong to different instances may be large), CA will decay SA to make sure that the value of overall Corner Affinity is low, v.v.. In brief, only when the SA value is large and the CA value is small, the Corner Affinity value is large so that the two corners are grouped together. The hyper-surface of corner affinity function can be seen in Figure 2.

For a better explanation of the Corner Affinity, we conduct qualitative analysis of it. As shown in Figure 4, we can think of the construction of Corner Affinity as a pseudo-3D box in which the SA is the "underside" and CA represents the "depth". In this way, inseparable corners with similar instance shapes or coincident centers in the 2-dimensional plane become separable in the 3-dimensional plane through the promotion of dimension (CA). Hence, the interaction of the SA and CA plays an important role in the grouping process to cover varied kinds of scenes.

4 Experiments

4.1 Datasets and EvaluationMetrics

We use three benchmark datasets to verify the effectiveness and generalization of the proposed Corner Affinity. Their instructions are as follows:

MS-COCO COCO [Lin et al., 2014] is a large-scale and challenging benchmark in object detection, which contains 80 categories and more than 1.5 million object instances. We train on the train2017 split, which contains 118k images and 860k instances. We compare the proposed Corner Affinity with state-of-the-art methods on both the val and test-dev splits. All ablation studies are performed on the val2017 set that contains 5k images and 36k objects.

Citypersons We select this dataset [Cordts et al., 2016] to test the effectiveness of the proposed Corner Affinity under dense occlusion scenario. Citypersons contains six different labels, i.e., ignore regions, pedestrians, riders, sitting persons, other persons with unusual postures, and group of people. We keep and merge the labels of pedestrians and riders that accounts a large proportion in vanilla data. There are 18204 persons in 2471 images on our processed training set. We show the performance of the proposed Corner Affinity on the validation set that contains 439 images and 3666 persons.

UCAS-AOD There are 7482 aircraft in 1000 images in UCAS-AOD [Zhu et al., 2015]. This dataset exits an extreme scenario that numerous similar objects arranged evenly and regularly. Thus, we use it to further verify the robust of our Corner Affinity. The annotations we use is the cleaned and refined version offered by [Wei et al., 2020].

Evaluation Metrics We use the AP (average precision) metric to measure performance of object detection. The AP used in COCO is computed over ten different IoU thresholds (i.e., 0.5:0.05:0.95) and all 80 categories. For Citypersons and UCAS-AOD, since the annotation (bounding box) is not as precise as COCO, the AP under a high IoU is meaningless, so we only test the AP with 0.5 IoU.

4.2 Implementations Details

Training details During training, we set the input resolution to 511 × 511, which yields an output resolution of 128 × 128 for all models. For COCO, we train our model on eight NVIDIA V100 GPUs with 8 × 32 GB RAM. The model is trained from scratch to 600k iterations with a batch size of 64. The learning rate is set to 2.5e-4 and dropped 10× at the 500k iteration. For Citypersons and UCAS-AOD, we use two NVIDIA P100 GPUs with 2×16 GB RAM. All models are trained from scratch to 50k iterations with a learning rate of 1e-4 enjoying a batch size of 12. Standard random color jittering, cropping, and mirror flipping are performed as data augmentation. We use Adam as the optimizer for the full training objective. Other settings are the same as CornerNet baseline.

Testing details We followCornerNet to extract each corner coordinate. Firstly, we use softmax and 3×3 max-pooling on heatmaps and remain the top100 scored top-left and bottomright corners. Then we use the predicted offsets to refine the locations of corners. Next, we decode the structure affinity and contexts affinity in the corresponding corner location and combine them to be the overall Corner Affinity. Finally, Corner Affinity is performed to group corners that belong to the same object instance and Soft-NMS is run to filter the detection results. In this stage, each image is is fed into the network with its original resolution.

4.3 Main Result

COCO We utilize the challenging COCO dataset to verify the effectiveness of our Corner Affinity. As shown in Table 1, the Corner Affinity brings 5.8% boosting, from 40.5% to 46.3%, on AP for CornerNet baseline on COCO test-dev set without bells and whistles. And only updated with the proposed new grouping algorithm, CornerNet surpasses popular detection baseline [Cai and Vasconcelos, 2018; Bochkovskiy et al., 2020; Tian et al., 2019; Yang et al., 2019], with a large margin, proving the corner-guided detector exists high ceiling and our Corner Affinity remarkably promotes the development of this detector via optimizing corner grouping. It is worth noting that our pure grouping optimization produces more improvements for vanilla CornerNet than the other two single-stage variants, i.e., CenterNet [Duan et al., 2019] and CentripetalNet [Dong et al., 2020] and is competitive to the two-stage variant CPN [Duan et al., 2020]. The singlestage two not only optimize corner grouping upon CornerNet, but also use stronger corner enhanced features yet still gain weaker accuracy than our pure grouping optimization. The two-stage CPN, without grouping process, introduces subnetworks with numerous parameters to classify proposals is not as neat as our design. The experimental results show that the proposed Corner Affinity is the optimal corner grouping strategy at present.

We also produce comparisons with the newest favorite Transformer-based approaches. As shown in Table 1, employing Corner Affinity, CornerNet (Hourglass-104) yields an AP of 45.1% and an AP75 of 48.3% on COCO val set, surmounting recent advanced self-attention-based single-stage detectors (e.g., DETR [Carion et al., 2020] and Sparse RCNN [Sun et al., 2021]), even without any feature enhanced module, e.g., FPN [Lin et al., 2017a]. Higher AP75 means higher-quality detection boxes. All the above experimental results demonstrate the proposed new grouping algorithm, Corner Affinity, is splendid, which allows the antiquated corner-guided detector, CornerNet, to be a superb baseline again even in the Transformer-based detection era.

Citypersons & UCAS-AOD We select these two datasets to test the generalization performance of Corner Affinity under two extreme scenarios, i.e., occlusion scene (Citypersons) and symmetrical arrangement of similar objects scene (UCAS-AOD). As shown in Table 2, Corner Affinity can still produce excellent accuracy upon higher-quality corner pairs under all aforementioned challenging situations. Specifically, compared with vanilla CornerNet, Corner Affinity boosts AP of amazing 35.8% and 17.2% on Citypersons and UCASAOD, respectively, firmly proving our design brings more robust grouping capability for corner-guided detectors.

4.4 Ablation Study

In this section, we conduct ablation analyses mainly on COCO val2017 set and partially on Citypersons.

Effectiveness of each part of Corner Affinity The proposed Corner Affinity is coupled of the structure affinity (SA) and contexts affinity (CA). SA is utilized to generate primary corner pairs via construction information of objects and CA runs as amending corner pairs via deeper semantic features. Each part can work individually, and thus we mask one to test the usefulness of the other. As shown in Table 3, for COCO datasets, only with the SA, a model can yield a decent AP of 44.3%. When only employing CA, the model only obtain an AP of 40.2%. Then when we couple the SA and CA, the AP reaches highest 45.1%. One may argue the AP of overall Corner Affinity is only 0.8% higher than that of pure SA. It is because there are too few cases of occlusion in the COCO dataset. To better show the superiority of Corner Affinity, we also do experiments on Citypersons dataset. For this data, Corner Affnity surpass 8.1% AP than that only with pure SA, showing the proposed Corner Affinity can cover more scenarios and is more reasonable.

Effectiveness of corner drifting The structure affinity (SA) is composed of an IoU and a bias, i.e., corner drifting. We devise the corner drifting to directly measure the distance of regressed vectors and true predicted corners. As shown in Table 4, the proposed corner drifting lifts 0.4% and 1.3% AP on COCO and Citypersons, respectively. The experimental results prove the designed corner drifting is resultful.

Optimal threshold for Corner Affinity Corner Affinity needs a threshold to be applied to group corners. Accordingly, we conduct ablation experiments to search the best threshold value. According to the Table 5, 0.1 is a suitable choice. Therefore, we set the value of Corner Affinity threshold to 0.1 for all experiments.

5 Conclusion

In this paper, we propose a robust grouping algorithm, termed as Corner Affinity, to produce more reliable and precise corner pairs for the bionic corner-guided detector. Equipped with the proposed Corner Affinity, the antiquated CornerNet baseline becomes great again. Along with Corner Affinity, we provide a new grouping idea, i.e., lifting the dimension of output attributes for each corner to cope extreme scenarios. We believe the human-like corner-guided detector still enjoys potentials and we hope our Corner Affinity can attract more attention to the corner-guided detection manner in the era dominated by the center-guided methods.