Relation-enhanced DETR for Component Detection in Graphic Design Reverse Engineering

1 Introduction

Reverse engineering graphic design aims to detect the logical components from a pixel-based design and parse its layout structure. It plays an important role in design understanding and can potentially enable many downstream applications (e.g., digital prototyping from a mock-up artifact or screenshot). In Figure 1, we show some design examples annotated with component classes and bounding boxes, including mobile user interface (UI), poster and slide. As we can see, the layouts are diverse and complex with many free-form and overlapped components, thus bringing more challenges for component detection.

One major characteristic across different types of design is that objects usually follow certain relation patterns. For example in mobile UI, Icon is often created inside ToolBar as a navigation widget on top of the interface. As another example, freeform objects in a slide are usually clustered in a small region to form a meaningful shape. To better quantify this characteristic, we conduct a statistical analysis of relations in graphic layouts with some interesting findings.

∗Work done during internship at Microsoft Research Asia.

†Corresponding Author.

Recently MagicLayout [Manandhar et al., 2021] is an initial attempt to incorporate co-occurrence relations into UI component detection. It statistically calculates a cooccurrence matrix between object classes in the corpus and incorporates it as a fixed weight into Faster-RCNN [Ren et al., 2015] to enhance the proposal features. Although MagicLayout has shown the effectiveness of the co-occurrence relation, its matrix is only calculated once as a prior and fixed during the training process, which might not be flexible for model learning. Moreover, the relation is constrained to cooccurrence while other types of relations (e.g., spatial overlap) [Jiang et al., 2018] could also be useful in capturing complex interactions between objects in graphic layouts.

In this paper, we explore a more flexible solution of relation modeling for component detection. We build our method on the current state-of-the-art detection framework DETR [Carion et al., 2020]. DETR is an end-to-end pipeline which uses a sparse set of learnable object queries as input to a Transformer encoder-decoder. In the decoder, queries will interact with each other (self-attention) and attend to relevant regions of the image (cross-attention). Specifically, we propose a learnable relation matrix for capturing class-to-class correlations. For each query pair, we retrieve the corresponding weight from the relation matrix using their predicted class indexes from the previous decoder layer. This relation weight will then be added to the self-attention weight to update query-to-query interactions.

We conduct experiments on three publicly-available graphic design datasets, including RICO [Deka et al., 2017] (mobile UI), Crello [Yamaguchi, 2021] (posters) and InfoPPT [Shi et al., 2022] (slides). Experiment results show that such simple relation modeling works surprisingly well and achieves better performance than several strong baselines. Through matrix visualization and case studies, we can observe that the relation matrix has captured certain reasonable correlations between class pairs. Furthermore, we show a potential application of layout generation using the component detection outputs as augmented training data and achieve promising results.

To summarize, the main contributions of this paper are:

• We conduct an in-depth analysis of relations in graphic design layouts to emphasize their importance.
• We propose a simple but effective relation-enhanced self-attention based on the strong object detector DETR, and achieve currently the best performance on three datasets.
• We explore an application of leveraging component detection outputs as augmented training data in layout generation, which shows promising results.

2 RelatedWork

2.1 Component Detection in Graphic Design

Traditional approaches aggregate edge or contour features with heuristic rules to recover the interface structure [Yeh et al., 2009; Nguyen and Csallner, 2015; Dixon and Fogarty, 2010]. For example, REMAUI [Nguyen and Csallner, 2015] combines the OCR outputs of text boxes and the detected edges of elements to infer the final components on a design using simple merging algorithms. In recent years, as neural detection networks such as Faster RCNN [Ren et al., 2015] and Mask RCNN [He et al., 2017] have shown great performance across different domains, many works directly adopt these detection models for GUI component detection [Moran et al., 2018]. Some methods further incorporate specific features in graphic design. E3Nets [Ma et al., 2021] embeds edge features along with the RGB channels to train a segmentation model, as edges provide useful information of the object skeleton. The most closely related research to this paper is MagicLayout [Manandhar et al., 2021], which exploits common spatial relationships of components in UI layouts. It pre-calculates a co-occurrence matrix between object classes in the corpus and uses this fixed matrix as prior to update the proposal features in Faster RCNN. In this paper, we explore a more flexible solution of relation modeling and will show that our proposed learnable matrix can capture reasonable patterns.

Meanwhile, document layout analysis is a highly relevant task of identifying regions of interest in the scanned image of a document. It mainly focuses on textual documents such as forms [Harley et al., 2015] and scientific articles [Zhong et al., 2019], where text blocks are dominant and layouts are mostly simple rectangle grids. Detailed related works can be referred to [Binmakhashen and Mahmoud, 2019]. Being compared, graphic design layouts have more complex variations with free-form and overlapping objects, which makes its component detection more challenging.

2.2 Object Detection

Previous works on graphic design component detection mainly view it as a detection task. At the present stage, most object detection methods [Girshick, 2015; Ren et al., 2015; He et al., 2015; Lin et al., 2017; Wang et al., 2022a] are known as anchor-based, which rely on numerous handcrafted target assignments and non-maximum suppression post-processing, and thus are not fully end-to-end trainable and hard for parameter tuning.

DETR and its variants Recently DETR [Carion et al., 2020] has been proposed as a new paradigm for object detection which achieves promising results. It employs a Transformer architecture and replaces hand-crafted components with a set-based global loss. As DETR has suffered from slow and unstable convergence during training, many following works [Zhu et al., 2021; Liu et al., 2022; Yao et al., 2021; Meng et al., 2021; Wang et al., 2022b; Zhang et al., 2022; Gao et al., 2021] have been proposed to eliminate the issues. For example, Deformable DETR [Zhu et al., 2021] designs multi-scale deformable attention to only focus on a small set of relevant regions. DAB-DETR [Liu et al., 2022] explicitly assigns 4D box coordinates to queries as positional prior which obtain faster convergence.

Relation Modeling There are many pioneering works that have already verified the effectiveness of modeling relations in object detection [Jiang et al., 2018; Chen et al., 2019; Xu et al., 2019; Zhao et al., 2021; Bi et al., 2022; Manandhar et al., 2021]. Relations used in previous works can be mainly categorized into three types: (1) co-occurrence between object classes. For example, [Chen et al., 2019] uses conditional probabilities to represent co-occurrence; MagicLayout aggregates co-occurrence frequencies of objects in different regions; (2) spatial relation computes relative positions between objects. [Zhao et al., 2021] use self-attention to build position-wise spatial relation. SRRV [Bi et al., 2022] simply uses the distance between region proposals and achieves great performance improvement; (3) learnable relation introduces more model capacities. For example, SGRN [Xu et al., 2019] defines a sparse graph where the edge weights are tuned during training. In this paper, we follow the direction of learnable relation modeling which is more flexible with less human priors.

3 Methodology

In this section, we first conduct an analysis to emphasize the importance of relations in graphic design layouts. Then we revisit the DETR architecture and propose our relationenhanced attention mechanism.

3.1 Relations in Graphic Design Layouts

Relations between objects play quite an important role in graphic design layouts. For example in Figure 2, the combination of Toolbar and Icon (highlighted in red bold edge) exists in all three interfaces, as they compose a navigation widget which has a high frequency of co-occurring in the upper region.

To better quantify this characteristic, we measure the correlation degree of class pairs at corpus-level using Pointwise Mutual Information (PMI):

where P(cx, cy) denotes the probability of class cx and cy co-occurring in a layout. PMI = 0 indicates statistically independent between two classes, while a larger PMI value indicates a higher positive correlation.

We use the mobile UI dataset RICO for analysis and show its PMI distribution in Figure 3(a) by enumerating all possible class pairs. As for comparison, we also compute the PMI on the natural image dataset COCO [Lin et al., 2014] (Figure 3(b)). In our measurement, there are approximate 93% class pairs in RICO with PMI values larger than 0, and the distribution is negatively-skewed indicating the majority of class pairs have high positive correlations. While being compared, there are only 62% frequently co-occurred pairs in COCO and the PMI values are evenly distributed. Based on this statistic, we can observe different characteristics between natural image and graphic design layouts, where the latter contains more highly correlated class pairs.

3.2 Revisit DETR

Since our method is built on the current state-of-the-art detection model DETR, here we briefly introduce its architecture as shown in Figure 4. It contains a CNN backbone, a Transformer encoder-decoder, and two prediction heads to predict object classes and bounding boxes. For an input image of (H, W, 3),

• The CNN backbone extracts image features and produces a feature map of ( H 32 , W32 , C). C is the output channel size.
• The encoder takes in the summation of the flattened feature map and the positional embedding, then outputs the encoded image feature of ( H 32 , W32 , C).
• The decoder inputs a set of learnable object queries (N,C) and updates the queries with self-attention (query-to-query) and cross-attention (query-to-image).
• For each query, two prediction heads predict the class probabilities (N,Nc) and bounding boxes (N, 4) respectively. Nc is the number of classes.

DETR Variants As mentioned in Section 2, there are many following works trying to resolve the issues of slow convergence and unstable training in DETR. For example, Conditional DETR [Meng et al., 2021] projects the object queries of (N,C) to 2D reference points of (N, 2). Similarly, DAB-DETR [Liu et al., 2022] uses anchors of (N, 4) as queries. Our proposed relation-enhanced attention in the next subsection can be generally added to most of the DETR variants. For experiments, we verify our method's effectiveness using the two DETR variants mentioned above.

3.3 Relation-enhanced Attention in DETR

Based on the relation analysis in Section 3.1, we propose a relation-enhanced attention mechanism in the DETR decoder (Figure 5).

Specifically, we initiate a learnable class-to-class matrix A ∈ Nc × Nc for modeling correlations between classes in graphic design layouts. The self-attention in the L-th decoder layer is enhanced with an query-query relation weight R ∈ Nq × Nq:

where hi is the hidden state of the i-th object query, Wq,Wk are weights of the learned linear projections for query and key in self-attention, dk denotes the hidden size of Wqhi, and aij defines the probability distribution of attention over the queries which are computed from the un-normalized attention scores eij .

To obtain Rij, we use the class predictions from the previous (L - 1)-th layer as indexes to retrieve corresponding value from matrix A:

where cL-1 i is the predicted class of the i-th query with maximum probability in the previous layer, and Wc is the weight of the class prediction head.

Since the one-hot assignment operation via argmax is non-differentiable, we instead use Gumbel-Softmax [Jang et al., 2016] for sampling from class distribution:

where {γ} of classes {0, . . . , k, . . . ,Nc} are i.i.d random samples drawn from the Gumbel(0, 1) distribution. We apply the straight-through trick in [van den Oord et al., 2017] to compute the class assignment as

where sg is the stop gradient operator. With the straightthrough trick, ˆ cL-1 i has the one-hot value of assignment to a single class, and its gradient is equal to the one of cL-1 i , which makes the relation-enhanced self-attention differentiable and end-to-end trainable.

4 Experiments

4.1 Experiment Settings

Here we introduce the datasets, baselines and evaluation metrics used in our experiments.

Datasets. We consider three publicly-available graphic design datasets: RICO, Crello and InfoPPT 1 . (1) RICO [Deka et al., 2017] consists of approximately 70k unique UI screenshots from more than 9.3k Android mobile apps. We use its simplified annotations [Manandhar et al., 2020] of 25 UI component classes with in total of 65K images. (2) Crello [Yamaguchi, 2021] contains 23k images with 5 categories, covering a wide range of designs such as social media posts, banner ads and posters collected from crello.com. (3) InfoPPT [Shi et al., 2022] contains 23k information presentations collected from public websites. We parse the slides (.pptx) to obtain object annotations and then convert image inputs.

1 We do not consider DrawnUI [Ionescu et al., 2020] since it is

Baselines. We consider two detection backbones: Faster RCNN and DETRs (conditional DETR and DAB-DETR). Two baselines are applied to the detection framework: (1) SGRN [Xu et al., 2019] is a graph-based feature fusion method by learning a relational graph between region proposals in Faster RCNN; (2) MagicLayout [Manandhar et al., 2021] uses a pre-computed class co-occurrence matrix to update proposal features in Faster RCNN. When they are adapted to DETRs, we use their matrices to update the object queries similar to the region proposals in Faster RCNN.

Evaluation Metrics. Following MagicLayout, we use the standard metrics from COCO detection evaluation criteria [Lin et al., 2014]: mean Average Precision (mAP) across different IoU thresholds (IoU= 0.5:0.95, 0.5, 0.75) and scales (small, medium, large). We also report Average Recall (AR) with different numbers of detection (1, 10, 100) and scales.

not currently available.

4.2 Implementation Details

We use open-source implementations of Faster-RCNN 2 , Conditional DETR 3 and DAB-DETR 4 . Models are initialized using parameters pretrained on COCO. We run all the models on 8 Tesla V100 GPUs with batch size 8 for 40 epochs and AdamW [Loshchilov and Hutter, 2017] is used for training with weight decay 10-4 . We set different learning rates for backbone and other modules to 10-5 and 10-4 respectively. CosineAnnealing optimizer is used with Tmax of 40 and decays it by a factor of 0.05 by the end of training. During training, images are resized such that the short side is at least 480 and at most 800 pixels and the long size is at most 1333 pixels.

4.3 Overall Results

We show the overall results in Table 1 (RICO), Table 2 (Crello) and Table 3 (InfoPPT) respectively. As we can see, DETR-based models (Conditional DETR and DAB-DETR) generally outperform Faster RCNN by a large margin. Except that in RICO, DETRs have lower accuracy in terms of APs, APm and ARs. We argue that as RICO has much more classes and larger size variation of objects, Faster RCNN with multi-scale features would handle some specific classes better. Furthermore, our relation-enhanced attention mechanism has boosted the performance in both DETR variants and obtained better results than SGRN and MagicLayout. For example, under the setting of DAB-DETR, the mAP has been improved from 55.3% to 57.6% on RICO, achieving state-ofthe-art performance. Similar gains can be observed in Crello (+1.5%) and InfoPPT (+2.5%).

2 https://github.com/open-mmlab/mmdetection

3 https://github.com/Atten4Vis/ConditionalDETR

4 https://github.com/IDEA-Research/DAB-DETR

4.4 Qualitative Results

Here we show some predicted results in Figure 6 on RICO. We keep all the detected objects with confidence scores larger than 0.6. With the use of relation matrix, our model outputs objects with generally higher confidence and more reasonable bounding boxes. For example, our model successfully detects all the Text objects with confidence over 0.96 in Figure 6(a). Also, our model can correctly recognize objects that are missed by the baseline DAB-DETR, such as the two white TextButton in Figure 6(b). For the above analysis, we use the DAB-DETR setting on RICO.

4.5 Matrix Visualization

To better understand what the relation matrix has learned, we further visualize the relation matrix heatmap in Figure 7(a). We only highlight the top-5 frequent classes in each row for better viewing. The diagonal of the matrix has the largest weight, which indicates that each class pays the most attention to its own. We can also observe high correlations between some class pairs in the learned matrix, such as <Toolbar, Icon>, <Input, TextButton>. As an example shown in Figure 7(b), the baseline DAB-DETR failed to detect a TextButton object ("SIGN IN" button in blue) while our relation-enhanced model can correctly identify the object with the surrounding context of Input. This demonstrates that our relation matrix has captured useful contextual information between class pairs for better detection accuracy.

4.6 Relation Type Ablation

To verify the effectiveness of the learnable relation, we experiment with different types of relations into the selfattention, including (1) class co-occurrence: we use the same pre-computed co-occurrence matrix as in MagicLayout; (2) spatial distance: center-point distance between two object queries using their predicted bounding boxes from previous decoder layer. As shown in Table 4, there is a relatively small improvement using the relation of class co-occurrence and spatial distance. We argue that the learnable matrix can provide more capacities for capturing relevant information and therefore shows more performance gain than the other two types of relations. Due to the space limitation, more ablation studies will be provided in the Supplementary.

4.7 Data Augmentation for Layout Generation

Here we show a potential usage of component detection, which uses the detection outputs as augmented training data for graphic layout generation. This task aims to synthesize a set of diverse and realistic layouts. Assuming that we only have a small amount of labeling data, this experiment verifies if the detection outputs can be used as high-quality training data to improve the layout generation results. In our setting, we sample 500 layouts as the initial set and increasingly add percentages of detection outputs (in total 2k layouts) to train the layout generator. We use the state-of-the-art model LayoutTransformer [Gupta et al., 2021] for generation. Three commonly-used evaluation metrics are reported: FID, Overlap and Alignment. As shown in Table 5, the generation performance gets steadily boosted with increasing amounts of augmented data, which indicates the high quality of our detection outputs. We expect better and more stable results will be obtained with a larger scale of data, which could be possibly collected by parsing layouts from numerous pixel-based designs available online.

Due to the space limitation, we show more analysis and case studies in the Supplementary.

5 Conclusion

In this paper, we study the problem of component detection in reverse engineering graphic design. We conduct a statistical analysis to demonstrate the importance of relations in graphic layouts. Built on the strong detector DETR, we present a relation-enhanced self-attention mechanism in DETR decoder for better interactions between object queries. Our method has achieved the best results on three public datasets. Moreover, we leverage the detection outputs as augmented data to improve the layout generation performance. In the future, we plan to explore better solutions to handle the overlapping objects which are not solved well by the current model. Also we will try to use component detection to enable more applications in graphic design intelligence.