MLP-DINO: Category Modeling and Query Graphing with Deep MLP for Object Detection

1 Introduction

Object detection (OD) plays a crucial role in computer vision tasks by identifying the bounding boxes and categories of objects within images [Girshick, 2015; He et al., 2017]. Over the years, convolution-based detectors [Girshick, 2015; Liu et al., 2016; Lin et al., 2020; Wang et al., 2023] have made remarkable advancements in this field. However, those methods usually rely on hand-designed components and complex pipelines [Carion et al., 2020], which imposes certain limitations on their performance and scalability.

∗Corresponding author.

Recently, transformer-based detectors simplify the pipeline of OD and get more attention in computer vision community. DEtection TRansformer (DETR) [Carion et al., 2020] is a novel end-to-end detection framework that casts OD as a set prediction problem to match the predicted object and ground truth by using a bipartite matching method. DETR achieves comparable results with the classical state-of-the-art detectors while eliminating the need of hand-designed components.

However, we argue that the DETR-like framework suffers from the box-sensitive category predictions and imbalancedquery distribution issues. On one hand, the bipartite matching process in DETR assigns each decoder query to a single object. The bounding box and category of this object is predicted simultaneously by the assigned query. This paradigm results in the category prediction are mutually influenced by the box prediction. As illustrated in Fig. 2 (a), the prediction scores of the categories decrease as the accuracy of the box predictions decreases. Here, we mark this issue as the Box-Sensitivity in Category Prediction (BS-ICP). On the other hand, the existing model [Carion et al., 2020; Yao et al., 2021; Li et al., 2022; Zhang et al., 2022] select the positional queries from all tokens solely based on the predicted confidence scores or static learnable embeddings.

• (a) Category Score-Iou of Box (b) Performance-ODMetrics

Figure 2: The models with (-w-) QICS achieves both higher scores of categories prediction and boosted performance than that of without (-wo-) utilizing QICS on val2017 of COCO dataset. The DINO and MLP-DINO model use ResNet50 and Strip-MLP-TSWMP as the backbone, respectively.

These approaches explicitly neglect the significant spatial information of each query, leading to an imbalanced spatial distribution of selected queries. Fig. 3 (a) illustrates that a considerable proportion of query points tends to cluster around a single object, resulting in lower query hit-rate for other objects. The hit-rate represents the proportion of the number of objects that are successfully hit by the query points, out of the total number of objects within the image.

Query-Independent Global Categories Modeling. For the issue of BS-ICP, our observations indicate that modeling the categories information globally would significantly boost the detection performance. Therefore, we propose the QICS approach, which contains an image-level classification layer and classification loss, to reduce the dependency of category recognition on the box prediction process. In particular, this approach uses the feature from backbone and encoder to predict all categories of objects to be detected. Thus, this category prediction is query independent and unrelated to each box predictions, enabling the model to identify all potential object categories within the image. This query independence method draws inspiration from the partitioned intelligence observed in brain organoid.

Long-range and Short-range Information Modeling. In the original DETR architecture, conventional CNNs, such as ResNet50 and ResNet101 [He et al., 2016], are employed as the backbone to generate lower-resolution feature maps. Then, these feature maps are directly utilized by the subsequent encoder and decoder modules. Here, we make a hypothesis that incorporating a backbone model capable of aggregating both long and short range information can significantly enhance the detection performance across objects of varying scales. To verify our hypothesis and further improve the category predictions, we conduct a comprehensive research on CNN-based, Transformer-based, and MLP-based backbone architectures in DETR-like frameworks. Recently, MLP-based models [Liu et al., 2022a] have shown effectiveness in capturing both long-range and short-range information, yielding promising performance in image classification tasks. We introduce a new MLP-based backbone model into the DINO framework, and the experiment results demonstrate their effectiveness. As presented in Fig. 2, the category predictions and detection performance are remarkably improved with MLP-based backbone model than CNN-based model.

Query Graphing for Distribution Balanced. Query selection is a significant process in DETR-like models [Yao et al., 2021; Zhang et al., 2022]. In original DETR, the query has no clear meaning and is simply initialized as learnable parameters. The follower-up works [Yao et al., 2021; Liu et al., 2022b] make efforts mainly on optimizing the query formulations, such as specifying the composition of the query [Meng et al., 2021], initializing the query in static [Li et al., 2022], dynamic [Zhu et al., 2020] or mixed [Zhang et al., 2022] ways, etc. However, these works neglect the spatial distribution of queries, leading to an imbalance distribution in selected queries. As illustrated in Fig. 3 (a), most selected queries cluster around a large object, leading to many redundant queries and a low query hit-rate on other objects.

We argue that incorporating the spatial information into the query selection process can effectively mitigate the query redundancy and improve the hit-rate of queries on objects. In this study, inspired by the functional connectivity networks observed among biological neurons, we propose a novel GQS method that utilizes the relative distance information between paired queries to select decoder queries with higher discreteness. This approach ensures that the selected queries are more representative and informative. Fig. 3 (b) illustrates a sparser distribution with higher hit-rate of queries to objects, balancing the selected queries distribution.

Our new proposed MLP-DINO detector is illustrated Fig. 4, which is the first attempt to integrate the deep MLP model into transformer-based detection framework. The performance comparison between MLP-DINO and other popular models is presented in Fig 1. Our contributions are as follows:

• We design a new MLP-DINO model by introducing the deep MLP model with our Sharing Weight on MiniPatch (SWMP) approach into DINO framework, modeling the long and short range information simultaneously, and propose a QICS method to decouple the box regression process and boost the category predictions.
• We propose a novel GQS method to balance the spa-

• tial distribution of queries, reducing the redundancy of queries and improving the hit-rate of query points to objects, enhancing the model's performance.
• Extensive experiments indicate that MLP-DINO remarkably improves the performances of the DINO model. MLP-DINO achieves 54.6% AP with only 44M parameters under 36-epoch setting, higher than original DINO by +3.7% AP with fewer parameters and FLOPs.

2 RelatedWork

Classical Detectors. Over the past decade, convolutionbased detectors have made tremendous advancements, primarily including anchor-based and anchor-free methods. Anchor-based methods, like Faster R-CNN [Girshick, 2015] and Mask R-CNN [He et al., 2017], generate anchor boxes in the first stage, and then classify them and regress their coordinates relative to the anchor. Anchor-free methods are designed to reduce the dependency on predefined anchors by directly predicting the locations of bounding boxes or keypoints without the need for explicit anchor boxes, like CenterNet [Duan et al., 2019], YOLOX [Ge et al., 2021]. However, all those methods rely on handcrafted components such as Non-Maximum Suppression and complex pipelines, which significantly limits their performance [Carion et al., 2020].

Transformer-based Detectors. DETR [Carion et al., 2020] is the first transformer-based detector that directly predicts objects by utilizing a fixed small set of learned object queries. Each query is responsible for the prediction of the class label and bounding box coordinates of a single object. Therefore, how to initialize and select these queries has become crucial research directions in DETR-like models [Zhang et al., 2022].

Many related works have focused on improving the query selection method. Instead of randomly initialing the query in original DETR, Conditional DETR [Meng et al., 2021] decouples queries into positional and content queries, and defines the positional query as 2D coordinates to learn a spatial query for better matching with image features. Efficient DETR [Yao et al., 2021] indicates that selecting TopK dense prior features is better than random initialization of queries. DAB-DETR [Liu et al., 2022b] constructs the positional query with 4D anchor box coordinates to join the object scales information, improving the query-to-feature similarity. DN-DETR [Li et al., 2022] introduces a denoising query part to reconstruct the original boxes, accelerating the training process. Based on DN-DETR, DINO [Zhang et al., 2022] designs a new query part for rejecting "no object" anchors, and proposes a mixed query selection method for initializing query better. Although these works enhance the query selection process and get improved performance, all of them ignores the issues of BS-ICP and the imbalanced spatial distribution of selected queries.

Detection with DeepMLPModels. Since the introduction of MLP-Mixer [Tolstikhin et al., 2021] into the field of computer vision, many MLP-based variants models [Liu et al., 2021a; Tang et al., 2022a; Tang et al., 2022b; Cao et al., 2023] have made significant advancements in image classification tasks. As an attention-free model, deep MLP model offers advantages such as simple structure and low computational complexity, achieving comparable accuracy to other approaches. Researchers have also observed improved accuracy and robustness in object detection tasks by incorporating the deep MLP model into CNN-based detection frameworks, such as Mask R-CNN [He et al., 2017] and Cascade Mask R-CNN [Cai and Vasconcelos, 2018].

However, the reliance on fully-connected (FC) layers in deep MLP models makes them highly resolution sensitive [Liu et al., 2022a], as the number of neurons in FC layer is related to the input size of the image, which means the model cannot accept the input image with arbitrary size. This sensitivity poses a significant challenge that these models cannot apply to dense prediction tasks like object detection. Indeed, some deep MLP methods, such as CycleMLP [Chen et al., 2021], Hire-MLP [Guo et al., 2022], and Wave-MLP [Tang et al., 2022b] have addressed this issue by employing techniques such as feature shifting or using smaller kernels within local windows. But, the method of feature-shifting operation relies on specific model designs and is not universally applicable to other MLP models. Using smaller kernels also limits the advantages of long-range information aggregation of MLP models. Moreover, their performance in downstream detection tasks is also limited. Consequently, it is valuable to address the resolution limitation of deep MLP models and extend their application within transformer-based frameworks.

3 Method

In this section, we first describe the overall architecture of our new MLP-DINO with deep MLP backbone model. Then, we introduce the QICS approach for the category modeling by decoupling the box regression process. Furthermore, we introduce a Graph-based Query Selection (GQS) method to balance the SDQ. Finally, we propose the Sharing Weight on Mini-Patch (SWMP) approach to accept input images with arbitrary size for MLP-based models.

3.1 Overall Architecture

Our MLP-DINO architecture, depicted in Fig. 4, extends the robust DINO framework and comprises three key components: a backbone for feature extraction, a multi-layer Transformer encoder for feature enhancement, and a multi-layer decoder with two feed-forward networks (FFN) for box prediction. The process begins with feeding an image of arbitrary size into the model. The backbone network extracts features from this image, resulting in backbone features represented as b1 ∼ b4. Then, these backbone features are flattened before being fed into the transformer-encoder along with the positional embedding [Vaswani et al., 2017] and get enhanced feature e1 ∼ e4. This step serves to boost these backbone features by leveraging attention mechanisms to disentangle the different objects within the image.

Next, we introduce the QICS approach to address the BSICP issue and introduce the Strip-MLP [Cao et al., 2023] model into detection framework to further boost the category predictions. Strip-MLP is a MLP-based model that aggregates long-range and short-range information simultaneously and efficiently improves the token interaction power, IJCAI showing promising performance in classification tasks. However, the main limitation of the Strip-MLP is that it cannot accept images with arbitrary input sizes, as the number of neurons in MLP layers must align with the size of the input feature, which is still the common challenge in MLP-based models [Liu et al., 2022a]. To overcome this limitation, we reformulate the Strip-MLP model with our SWMP approach to accept images with arbitrary size.

To address the issue of imbalanced query distribution, we propose a new graph-based GQS method on encoder features to select decoder queries in a discrete manner. In general, each object is directly detected by a single query. However, the number of queries is typically much larger than the number of objects within the image to be detected. The key insight behind this approach is that queries with higher hit-rate on objects tend to achieve better detection performance.

Finally, with selected positional queries, we update the positional bias of box predictions layer-by-layer with the deformable attention [Zhu et al., 2020] in transformer-decoder. The detection results of boxes and corresponding classes are finally predicted by two FFN networks, respectively.

3.2 Query-Independent Category Supervision

In DETR-like models, a single decoder query predicts the bounding box and category simultaneously, leading the boxsensitive category prediction. That means the category prediction is affected by the box regression process. As illustrated in Fig. 2, as the IoU of the box regression decreases, the category score also decreases. To address this issue, we propose a QICS approach, which contains a Global Classification Module (GCM) and a Global Classification Loss (GCL), aiming to decouple the box regression process and identify all potential object categories within the image.

In DETR, the backbone and encoder features contain more global semantic information rather than query-specific instance information. As shown in Fig. 4, the query is only explicit involved in the decoder part of the network for the box prediction. Accordingly, to enable query-independent global category prediction, we apply QICS solely to the backbone feature and encoder features, which contain a wealth of category information of all objects. Specifically, the GCM in QICS achieves query-independent category prediction by utilizing global average pooling (GAP) [Lin et al., 2013] to pool features from different levels with varying spatial dimensions. This process reduces the spatial dimension of the input features to 1, making them independent of any queries and effectively reducing the spatial complexity of the GCMmodule. Subsequently, we concatenate the pooled features from different levels along the channel dimension to generate multiple image-level category predictions.

Finally, a linear layer is adopted to predict all classes of the objects within the image. The GCM can be formulated as:

where X is the input feature, and X1 ∼ X4 are the input features at four levels with different spatial resolutions. Linear is a head of FC layer for predicting the categories of objects.

With those class prediction results, we achieve the queryindependent category supervision by adopting the binary cross-entrophy (BCE) loss, which can be defined as:

where yi is the binary label for each object category among N images. A label of 1 indicates that the image contains an object belonging to a specific category, while a label of 0 denotes the absence of the target category within the image. xi is the prediction from the GCM. σ(∗) is the sigmoid function. With such a light-weight module (nearly 0.02M), our QICS approach contributes to decoupling the box regression process and identifying all potential object categories within the image from a global perspective.

3.3 Graph-based Query Selection

In DETR-like models, the performance suffers from the spatial distribution of the decoder queries, as each detection result is directly predicted from the corresponding query. Consequently, a series of research works have focused on the query formulations [Meng et al., 2021; Zhu et al., 2020; Liu et al., 2022b; Zhang et al., 2022], including the number and pattern of queries, the composition of queries, the initialization method of queries, and so on. However, these existing works ignores the spatial distribution information of the queries, which significantly reduces the effectiveness of both the redundant queries and the model.

To deal with this concern, we propose a new GQS method to balance the distribution of positional queries by incorporating the spatial distribution information of queries into the Top-K method. To be specific, we design a two-step query selection method for positional queries, where Nq is the number of queries to be selected for the decoder. The output features from the transformer-encoder have different resolutions at 4 levels, denoted as Fe = {e1, e2, e3, e4}. In the first step, these features are flattened and concatenated in the spatial dimension. Then, this feature is fed into a head of linear layer to obtain the class predictions. We sort these predictions based on their scores and get the Top-K indices of the queries for the second step, where K = λNq (λ ⩾ 1). The first step can be formulated as:

where QNq means the selected candidate queries. Argmax(∗) gets the highest scores among all class predictions. Cat(∗) denotes the concatenation operation.

In our second step, we leverage the relative positional information to select Nq queries from the larger candidate pool of λNq queries, where each query corresponds to a specific location (or point) within the 2D spatial space. Considering that each query point can be located anywhere on the 2D spatial space, we construct a graph, denoted by G, to describe the relative positional distribution of the query points, with each query's spatial coordinate serving as a node, as illustrated in Fig. 3 (b). The edges between nodes reflect the spatial relative distance between each paired queries. Specifically, we adopt the Euclidean Distance (ED) as the distance metric to measure the relative distance of all paired queries. Based on these distances, we further create an adjacency matrix, referred as A ∈ RλNq×λNq for the graph G, where each row and column of A corresponds to a query point, and the distance value in each cell represents the weight of the edge between the corresponding pair of query points in G.

The objective of our approach is to distribute the query points in a discrete manner in the 2D spatial space of the feature, to cover a larger number of potential objects and increase the query hit-rate. Due to the varying sizes between images, it is challenging to only adopt the absolute distance information to accurately represent the distribution of queries in a uniform manner. To this end, we utilize the Coefficient of Variation (CV) [Abdi, 2010], also known as Normalized Root-Mean-Square Deviation (NRMSD), aiming to measure the discreteness of each query point. The CV denoted CV is a standardized statistical measure of the dispersion of distance distribution, which is defined as the ratio of the standard deviation σ to the mean µ:CV = σµ (4)

With the adjacency matrix A, we can calculate the standard deviation Aσ and mean Aµ of positional distribution along the second dimension of A for each query. By computing the ratio of Aσ to Aµ, we obtain the CV of A denoted as ACV . The values in ACV represent the level of discreteness in the distance distribution between query points. A lower value indicates a more concentrated distribution of distances between the current point and other points, implying that the current point is more scattered.

After obtaining the discreteness and classification prediction scores of each query, we design the criterion with these two matrix in Eq. 5 for query selection. In particular, we apply the Top-K strategy once again on this new criterion to select Nq queries with higher scores from the pool of λNq queries. Specifically, query points with higher confidence scores and larger discreteness will be selected as the final queries. This process can be formulated as:

where K = Nq, and σ means the sigmoid function. ⊙ means the Hadamard product of the matrix.

3.4 SWMP for Fixed Image Size of DeepMLP

As described in the previous section of Related Work about deep MLP Models, their reliance on FC layers for token interaction presents a significant challenge that MLPbased models pre-trained on image classification datasets are not easily transferable to downstream dense prediction tasks, which significantly limits their development and application.

To address this challenge, we propose a new SWMP approach, which involves applying MLP layers with shared weights on mini-patches within cross-regions. Initially, we assume that a pre-trained MLP model is available for image classification with a fixed input size of Ho × Wo. For the downstream task such as object detection, where input images have varying sizes of H ×W, we crop (or pad) the image into mini-patches with overlap, ensuring each patch has the same size as Ho × Wo. The overlap size of lh and lw is determined adaptively by the remainder of H and W divided by Ho and Wo, respectively. Afterwards, we utilize the shared MLP layer to process all these mini-patches. To restore those processed patches to their original size, we combine the overlapping regions of the patches with an average weight. The SWMP approach is a general method to transfer the MLP-based models into resolution-free models, making them suitable for a wider range of dense prediction tasks.

4 Experiments

4.1 Experiments Setup

The experiments are conducted on the benchmark dataset of COCO2017 [Lin et al., 2014], which contains 118k training images of train2017 and 5k validation images of val2017. We develop MLP-DINO model by reformulating the Strip-MLP [Cao et al., 2023] with our SWMP method to integrate it into original DINO [Zhang et al., 2022] framework, incorporating with the QICS and GQS to model the category and balance the query distribution. For a fair comparison, we follow the training recipe in DINO and train the model with AdamW [Loshchilov and Hutter, 2017] optimizer with weight decay of 1 × 10-4 using Tesla V100 GPUs, with the batch size of 8. The number of decoder queries and denoising queries are 900 and 100 for all of our models, respectively. The default value of λ is 2 for GQS. All MLP-DINO

models adopt 4-scale features from the backbone. All our models are trained for 12 epochs (1× training scheduler) in the ablation study, except for specially marked ones. The detection performance is measured by the standard average precision (AP) under different IoU thresholds and object scales.

4.2 Main Results

Our MLP-DINO enhances the DINO model by effectively modeling the category information and balancing the query distribution. Table 1 presents a comprehensive comparison of our MLP-DINO with other DETR-like detectors [Carion et al., 2020; Zhu et al., 2020; Yao et al., 2021; Meng et al., 2021; Liu et al., 2022b; Li et al., 2022; Zheng et al., 2023; Jia et al., 2023; Zhang et al., 2022], as well as Faster RCNN [Ren et al., 2015]. MLP-DINO achieves the best performance on all evaluation metrics. In particular, under the 12-epoch setting, MLP-DINO achieves 52.4% AP with fewer parameters and GFLOPs, higher than original DINO [Zhang et al., 2022] by +3.4%AP. In addition, in both 24-epoch and 36-epoch settings, our MLP-DINO consistently outperforms all other methods, demonstrating superior performance.

4.3 Ablation Study

Ablation on All Components of MLP-DINO. In Table 2, we present the results of ablation studies, using DINO as the baseline model with the backbone of ResNet50. The incorporation of the Strip-MLP-T-SWMP backbone leads to a significant improvement of +2.7%AP compared to the baseline, highlighting the importance of modeling long and short range information in the backbone network. Fig. 2 illustrates that both the category predictions and detection performance are improved with the MLP model and QICS approach. Additionally, the models show an additional performance boost of +0.6% AP when QICS or GQS is applied individually.

General Effectiveness of QICS and GQS. To verify the general effectiveness of the QICS and GQS approaches, we apply these two approach to different backbone architectures, including both CNN-based and Transformer-based models. Table 3 displays the ablation results, indicating that both QICS and GQS lead to the average improvement of +0.7%/+0.8% AP on ResNet50 and SwinT, respectively. These results demonstrate the general effectiveness of these approaches across different DINO models.

Ablation on Different Backbones in DINO. To verify the effectiveness of modeling the long-range and short-range information, we conduct experiments with different backbones in DINO, such as CNN-based, Transformer-based, and MLPbased models. Table 4 clearly demonstrates that the models incorporating long-range information, such as Swin-T [Liu et al., 2021b] and Strip-MLP-T, outperform ResNet50, which only models short-range information. Impressively, we find that Strip-MLP-T gets higher performance than ResNet50 and Swin-T by +2.0% and +0.4% AP, respectively, showing that the Strip-MLP model is more robust and efficient to aggregate features. The results highlight the advantage of incorporating both the long-range and short-range information, as it provides more robust feature and enhances the category predictions, as shown in Fig. 2.

Effectiveness of GQS on Hit-Rate. We conduct a statistical analysis of the hit-rate for each model with different backbones. In Table 5, all models with GQS consistently get better performance and higher hit-rate, indicating that GQS effectively leverages the spatial distribution of queries and thus improve the detection performance of the model.

Ablation of QICS on Different Features. The QICS approach aims to decouple the box regression process and identify all potential object categories within the image. To show the different impacts of QICS on different features, we apply the QICS to the different part of backbone features of b1 ∼ b4 and encoder features of e1 ∼ e4. As presented in Table 6, the ablation study reveals that the best results are achieved when applying QICS only to the high-level feature b4 and the enhanced features of e1 ∼ e4, as the low-level features of b1 ∼ b3 lack of necessary semantic category information. It would introduce additional noises and decrease accuracy when the QICS is applied to the low-level feature of b1 ∼ b3. Therefore, we apply the QICS on the features of b4 and e1 ∼ e4 for other experiments of this work.

Ablation of the Queries Pool Size for GQS. In the first step of GQS method, the hyper-parameter of λ determines the size of queries candidate pool. A smaller value, such as λ = 1, would result in the method degrading to the original Top-K approach. On the other hand, a larger value of λ indicating more candidate queries would introduce additional noise and complicate the training and optimization process. Consequently, choosing an appropriate value for λ is crucial to enhance the model's performance. We conduct additional experiments using various λ to identify the optimal values. Table 7 displays the results, indicating that λ = 2 achieves the best performance for the GQS method. Therefore, we set λ = 2 for the GQS method in our MLP-DINO models.

5 Conclusion

This paper introduces a novel MLP-DINO model, which is the first attempt to integrate deep MLP models into transformer-based detection framework, leveraging the strengths of both models. We emphasize the crucial roles of both category modeling and balanced query distribution in DETR-like models. To address this, we decouple the category prediction process by a novel QICS approach and further enhance the category predictions by introducing the deep MLP models into DINO with our general SWMP approach, enabling the MLP models to accept the input image of any size and greatly extending their application to downstream dense prediction tasks. In addition, we design a new GQS method to balance the query distribution, leveraging the spatial information of queries into query selection process. Our extensive experimental results demonstrate the effectiveness and potential of MLP-DINO in detection task. We hope that our results can spark further research based on MLP-DINO in computer vision community, such as the segmentation and other vision tasks.

Acknowledgments

This work is supported by National Key Research and Development Program of China (2021YFF1200800); Peng Cheng Laboratory Research Project (PCL2023A08); Shenzhen International Research Cooperation Project (Grant No. GJHZ20220913142611021).