Learn and Sample Together: Collaborative Generation for Graphic Design Layout

1 Introduction

Graphic layout generation is the process of determining the position and size of each object on a page, which plays a crucial role in creating a successful design (e.g., user interface, articles, presentation slides). It establishes the relationships between elements as well as the overall coherent appearance for better content display. Layout generation is a challenging research topic that dates back to the 1980s when dominant approaches were constraint-optimization-based with heuristic rules or templates [Hurst et al., 2009; Kumar et al., 2011; O'Donovan et al., 2014]. In recent years, an increasing number of works try to tackle this problem with powerful generative models such as Variational AutoEncoders (VAE)

∗Work done during internship at Microsoft Research Asia.

†Corresponding Author.

[Jyothi et al., 2019; Lee et al., 2020], Generative Adversarial Networks (GAN) [Li et al., 2019; Kikuchi et al., 2021] and AutoRegressive models (AR) [Gupta et al., 2021; Arroyo et al., 2021], which have achieved promising results.

Considering downstream applications (e.g., novel layout suggestion and layout retrieval), user specifications including element attributes and their relationships are useful preconditions to constrain the generated layouts. For example in Figure 1, given the user specification "put the image above the button", the generated layouts need to satisfy such spatial constraint. Nevertheless, most of the current systems fail to handle the relationship constraints. A natural way to encode the relationships is by using a graph, where the nodes represent element categories and the edges represent spatial relationships. The graph condition for layout generation can not only provide users the flexibility to specify requirements, but also serve as an interpretable intermediate representation for better model control.

Recently, Neural Design Network (NDN) [Lee et al., 2020] is an initial attempt to incorporate graphs into layout generation with multiple components in series, including graph generation, layout synthesis and refinement. Each component is dependent on the output of the previous one. NDN uses a training pipeline in which components are separately optimized. In our initial experiment, we observe that the system has a high frequency of generating out-of-distribution graphs which are unseen for the succeeding layout generation component during training. Errors in each component could be propagated and thus decreasing the overall performance (e.g., it is less likely to generate promising layouts if conditioned on an unreasonable graph). We will give a more detailed analysis of this issue in the later section.

In this paper, we aim to better utilize graph conditioning in layout generation. We simplify the components in NDN and present a two-stage framework consisting of a VAEbased graph generator followed by an autoregressive layout decoder. To prevent error propagation in the pipeline, we propose a novel collaborative generation strategy to jointly optimize these two generative networks. During training, one network is fed with the output from the other and conducts a round-way knowledge transfer. Similarly in inference, we apply cyclic sampling to refine the layout for several iterations. Additionally, we conduct a comprehensive study of graph conditioning such as graph sparsity, the ratio of user constraints in the graph, and its consistency to generated layouts.

We evaluate our model on three public datasets related to graphic design: RICO [Deka et al., 2017], PubLayNet [Zhong et al., 2019] and InfoPPT [Shi et al., 2022]. Experiment results show that our model outperforms current baselines by a large margin in terms of both quantitative and qualitative evaluations.

In summary, the main contributions of this paper include:

• We in-depth analyze the issue of the separate training pipeline in graph-conditioned layout generation.
• To better utilize the two highly correlated generative networks, we propose a novel collaborative generation strategy that boosts the model performance to the stateof-the-art.
• We conduct extensive studies to verify the effectiveness of graph conditioning. Our model is robust against the input constraint ratio in the graph as well as the sequence order in the autoregressive layout generation.

2 RelatedWork

2.1 Layout Generation

Early works [Hurst et al., 2009; Kumar et al., 2011; O'Donovan et al., 2014; Tabata et al., 2019] are mostly based on heuristic rules or predefined templates with constraint optimization. These methods usually ensure high-quality outputs but with very limited variations, thus restricting the applications of layout generation in complicated scenarios.

Unconstrained Layout Generation. In recent years, deep generative models have shown great power in learning the complex distribution from given data and generating samples with high fidelity and diversity, such as GAN [Goodfellow et al., 2014] and VAE [Kingma and Welling, 2013]. They have been also adapted to layout generation and achieved promising results. LayoutGAN [Li et al., 2019] applies GAN to synthesize the layout bounding box and proposes a differential wireframe rendering module to enable the training of discriminator, and LayoutGAN++ [Kikuchi et al., 2021] extends LayoutGAN with Transformer backbone. LayoutVAE [Jyothi et al., 2019] trains two VAEs separately, one to predict the element categories and the other to generate the layouts given the category condition. Several methods also follow the VAE-based generative framework [Patil et al., 2020;

Lee et al., 2020; Jiang et al., 2022]. Recent works [Gupta et al., 2021; Kong et al., 2021; Arroyo et al., 2021; Jiang et al., 2022] build the generative backbone based on Transformer [Vaswani et al., 2017] to model the long-distance dependency and yield better performance. There are also some content-aware generation methods [Zheng et al., 2019; Wang et al., 2022; Cao et al., 2022; Zhou et al., 2022; Li et al., 2022; Vaddamanu et al., 2022] that further considers the element content into modeling, which we will leave for future exploration.

Layout Generation with Constraints. Spatial constraints are shown to be crucial in this task which allows users to specify the desired spatial relations between elements. Previous works mainly encode the constraints as auxiliary losses or optimization functions. Attribute-conditioned LayoutGAN [Li et al., 2020b] considers element attributes (e.g., area, aspect ratio) and incorporates them by forcing the model to meet attribute conditions with extra training objectives. LayoutGAN++ [Kikuchi et al., 2021] views it as a constrained optimization problem in post-processing. Neural Design Network (NDN) [Lee et al., 2020] is an initial attempt to represent constraints as graph condition and incorporate it into model learning. It is a graph-based system with a separate training pipeline that optimizes different components separately, which raises error propagation in a series of components. In the next sections, we will analyze this issue in more detail and improve the graph condition modeling with a collaborative generation strategy.

As a similar task, related works in floorplan generation commonly use graph to model floorplan [Wang et al., 2019; Hu et al., 2020; Nauata et al., 2020; Nauata et al., 2021; Para et al., 2021; He et al., 2022], exhibiting great capability of graph representation in layout modeling.

2.2 Graph Generative Networks

Graph generative models [Kipf and Welling, 2016; Li et al., 2020a; Hasanzadeh et al., 2019; Zhang et al., 2019] mainly adopt graph convolutional networks to learn the distribution of graphs. One of the most representative models is Variational Graph AutoEncoder (VGAE) [Kipf and Welling, 2016], which embeds each node to a random variable in the latent space and uses an inner-product decoder to generate the adjacency matrix. In this work, we extend the framework of VGAE to multi-class edge matrix prediction and propose a variational graph autoencoder for generating spatial graphs with user constraints.

3 Approach

In this section, we first present our two-stage framework for graph-conditioned layout generation, including a graph generator and a layout decoder. Next, we analyze the issue of the separate pipeline for training the two networks, and further propose our collaborative generation framework.

3.1 Graph-Conditioned Layout Generation

Layout generation can be viewed as a sequence generation of s [Gupta et al., 2021]:

where vi is the category label of the i-th element in the layout (e.g., title, text, figure), xi, yi, hi,wi represent the position and size which are converted to discrete tokens. [bos], [eos] are special tokens for beginning and end.

To consider adding user specifications of element categories and their spatial relationships, it is natural to introduce a graph as a condition to constrain the generated layouts. Moreover, a graph serves as an intermediate representation in the generation process to improve the model's interpretability. We define a graph G = (V, E) where the node vi ∈ V is the element category and the edge ei ∈ E encodes the spatial relationship between two nodes. The edge can be one of the nine types: overlap, above-left, above, above-right, right, below-right, below, below-left, left. The spatial graph so far is a complete directed graph, which is dense and contains many spatially-true but redundant edges. For example, given above(A, B) and above(B, C), the edge above(A, C) may be not necessary to be stored. This edge redundancy problem will be likely to introduce noise in (1) predicting self-consistent edges during graph generation; (2) capturing the key structural information in graph conditioning. Therefore, we apply a heuristic pruning strategy to obtain a sparse graph: delete the edges which the distance of the two connected nodes is longer than a pre-defined threshold. We show later in the experiment that such a simple pruning strategy works surprisingly well.

Our generation process consists of two networks: (1) graph generator, producing a graph given partial user constraints; (2) layout decoder, synthesizing layouts conditioned on the graph. We briefly describe them in the following.

Network 1: Graph Generator. We extend Variational Graph Autoencoder (VGAE) [Kipf and Welling, 2016] for multi-classes edges generation. In VGAE, a graph is encoded to the latent code z by a GCN, and the edges are generated by the dot product of z. In our model, the latent code zi for each node is encoded as the concatenation of the input graph embedding by GCN and the node embedding with a learnable embedding matrix. The probability of an edge eij ∈ E belonging to a relation type k between nodes vi, vj ∈ V can be given by an inner product between the latent of two nodes zi, zj :

where ρk and ϕk is the non-linear transformation corresponding to k. During training, the ground truth of spatial graphs can be extracted from real layouts in the dataset.

Network 2: Layout Decoder. Conditioned on a generated graph from the previous step or an extracted graph from a real layout, our layout decoder autoregressively generates the layout sequence s:

where hi is the hidden representation of the i-th token. In each decoder layer, tokens will attend to the graph using a cross-attention.

In our observation, one major limitation of the previous autoregressive models is that the generated results are very sensitive to the input element order [Lee et al., 2020; Gupta et al., 2021]. For example, LayoutTransformer sorts the elements from top left to bottom right as the input order in both training and inference. The performance drops significantly when the order is set randomly. In our experiments, we show that our model is robust against different input order variations, in which the graph condition plays a crucial role.

3.2 Issue of Separate Training Pipeline

Similar to NDN, it is natural to connect the two networks using a separate training/inference pipeline. During training, the graph generator and layout decoder are separately trained with ground-truth extracted graphs from real layouts. Instead in inference, the layout decoder receives generated graphs from the graph generator as input. There exists a gap in the layout decoder's conditioned graphs between training and inference time. In our initial experiment, we observed huge performance reduction when the layout decoder input is changed from the extracted graph to the generated graph. Similar results can also be found in Lee [2020].

To better understand this issue, we visualize the latent space of extracted/generated graphs in Figure 3. As we can see on the left (separate training), the generated graphs have a distribution shift from the extracted graphs. With limited-scale and highly-imbalanced training data, it is difficult to train a fully generalized graph generator and can easily yield out-of-distribution graphs that are likely to be selfinconsistent among edges with increasing node size. Moreover, the layout decoder, only trained with ground-truth extracted graph inputs, is not robust to the unseen generated graphs. Being compared, our proposed learning strategy in the next subsection alleviates the distribution shift of generated graphs as shown in Figure 3 (right), and therefore reduces the error propagation from the graph generator to the layout decoder.

3.3 Collaborative Generation

To resolve the issue mentioned above, we propose a strategy including collaborative training and cyclic sampling for better unifying the graph generator and layout decoder. The overall pipeline is shown in Figure 2.

Collaborative Training. As shown in Algorithm 1, two networks teach each other alternatively during training iterations. On one hand, besides ground-truth extracted graphs, the layout decoder accepts generated graph input from the graph generator to update its parameters. On the other hand, a graph can be derived from the layout decoder output and is used as the training data for the graph generator. This training framework has two major benefits. First, it can be viewed as an approach of curriculum data augmentation, which enforces model robustness to more unseen data. Second, it provides instant communication to perform round-way knowledge transfer between the two networks, and adjusts their learning pace when training together.

Cyclic Sampling. We apply several rounds of decoding in the inference. After the first round of the generation process from graph generation to layout generation, the layout decoder will input the graph derived from its sampling layout in the previous round for iterative refinement. Empirically, we apply the cyclic sampling only on the layout decoder side as it improves sampling time efficiency while performing considerably well.

Algorithm 1 Collaborative Generation.

Require: Target layouts Lt, extracted graphs Gt fromLt

• 1: Initialize Graph generator θg, layout decoder θl
• 2: for training iteration i = 1, · · · , Tt do
• 3: Update θg with Gt, generate graphs Gg
• 4: Update θl with Lt conditioned on either Gt or Gg
• 5: Sample layouts Lg conditioned on Gg with θl
• 6: Update θg with graphs derived fromLg
• 7: end for
• 8: for sampling iteration i = 1, · · · , Ts do
• 9: if i == 1 then
• 10: Generate graphs Gi with θg
• 11: end if
• 12: Sample layouts Li conditioned on Gi
• 13: Derive Gi+1 fromLi
• 14: end for

4 Experiments

In this section we show our experiment details, system comparison results and the in-depth analysis of graph modeling.

4.1 Experiment Setup

Datasets. We conduct our experiments with three public datasets for graphic design, RICO [Deka et al., 2017] and PubLayNet [Zhong et al., 2019] and InfoPPT [Shi et al., 2022]. RICO consists of over 66k unique UI layouts from Android mobile apps. Following previous works, we exclude elements whose labels are not in the 13 most frequent sets and exclude layouts with more than 9 elements. After filtering there are 20,507 layouts in total. PubLayNet is a large collection of over 360k scientific documents crawled from PubMed Central. Similarly, layouts with more than 9 elements are excluded, totaling 173,225 layouts in the final set. InfoPPT contains 23k information presentations collected from the Internet. We exclude several unnecessary categories such as footnote and decorator, and select layouts with element numbers ranging from 4 to 20, which are in total 46,654 layouts.

Baselines. We consider the following recent works as baselines. LayoutVAE [Jyothi et al., 2019] takes the latent code and category labels (optional) as input and generates the element bounding boxes in an autoregressive manner. LayoutGAN++ [Kikuchi et al., 2021] improves LayoutGAN with Transformer backbone and applies several beautification post-process for alignment and non-overlap. NDN [Lee et al., 2020] is a pipeline system with graph generation, layout synthesis and refinement. LayoutTransformer [Gupta et al., 2021] autoregressively generates a sequence of element tokens.

Evaluation Metrics. There are 4 metrics commonly used to measure the generated layout quality:

• Maximum IoU. Given the generated layouts and the references, this metric computes the intersection over the union of the two sets with a permutation to maximize the IoU as a similarity measurement.
• Alignment. Layout elements are usually aligned with each other to create an organized composition. Alignment calculates on average the minimum distance in the x- or y-axis between any element pairs in a layout.
• Overlap. It is assumed that elements should not overlap excessively. Overlap computes the average IoU of any two elements in a layout. Layouts with small overlap values are often considered to be high quality.
• FID. Compared to the above heuristic metrics, FID is a sample-based metric for image generation [Heusel et al., 2017] and has been adopted in layout generation. It pretrains a feature network to classify real or fake layouts which is then used to extract features of two data sets and calculate the Fr´echet distance.

For all the evaluation metrics, we use the implementation from LayoutGAN++ 1 .

ImplementationDetails. For model architecture, the graph generator stacks 3 GCN layers with hidden dimension 128 and each node with hidden size 32. The layout decoder contains 6 attention layers each with 8 heads. The attention dropout rate is set to 0.5. The token embedding size is 512. The cyclic step for sampling is set to 2. We train the model for 100 epochs with a learning rate 3 × 10-4 and batch size 64. We use early stopping based on validation error. Adam [Kingma and Ba, 2014] is used as optimizer with β1 = 0.9 and β2 = 0.99.

1 https://github.com/ktrk115/const layout

4.2 Unconstrained Generation Results

Quantitative Comparisons. For the three heuristic metrics, the closer value to real data, the better performance. For FID score, we pursue the lowest absolute value. Table 1 shows the overall results of unconstrained layout generation. In this setting, our pipeline generates graphs without any constraints to initialize the graph as the same in NDN (labeled as "none"). Our model performs significantly better than the baselines with a large margin on most metrics, especially on the FID. Except that it does not perform well in terms of Overlap on RICO. We argue that elements in UI layouts are more frequently to be overlapped which might encourage the model to learn strong overlap behavior. To highlight, NDN is the most similar approach to ours which also synthesizes layouts conditioned on the generated graph but mainly differs in training strategy. Our model beats NDN across most metrics, indicating the effectiveness of our collaborative generation framework to improve the final performance of generated layouts.

Qualitative Comparisons. As the case study shown in Figure 4, our model generates layouts with better alignment and less overlap. With the aid of spatial graphs, we find that our model can learn to better capture the element relationships. For example, it frequently places icons inside the toolbar as navigation widget in UI layouts (RICO), and text captions on the top of tables while in the bottom of figures in scientific document layouts (PubLayNet).

Ablation of Different Components. We show the results by removing different components of our model in Table 2. All experiments in the following are conducted on PubLayNet. As we can see, coupled with all three components it achieves the best result. Especially with collaborative training, the performance is significantly improved, with FID optimized from 52.70 to 6.64. Graph pruning is also useful, meaning that sparse graphs can help the model better capture the key structural information by removing redundant edges.

Robustness to Input Sequence Order. The order of the input elements is also an important factor in previous autoregressive generation methods. Table 3 shows the result of different element orders on RICO. Without graph condition, the result is sensitive to the input element order (the random order performs much worse than the sorted order). After adding the graph, our model is robust against the order, with even FID score improved under the random setting. It may probably be the reason that random order serves as an approach to data augmentation with graph conditioning.

4.3 Conditioned Graph Analysis

Here we investigate the impact of different graph settings on the overall performance.

Effect of Input Constraints Ratio in Graph. With a smaller input constraint ratio to initialize a graph, the graph generator has to complete more missing edges, which increases the task difficulty. As shown in Figure 5, when training the graph generator and layout decoder separately (i.e., NDN and Separate), we observe great performance reduction with less constraint input. With our collaborative generation (purple line), the performance becomes stable across different constraint ratios. This indicates that collaborative generation improves the generated graph quality as well as the robustness of the layout decoder, minimizing the performance gap among graphs with different input edge ratios. We also show the performance of LayoutGAN++ with a postprocessing optimization of user constraints (red line). Interestingly, it shows a different trend of performance decreasing with a higher constraint ratio, which is expected since more constraints might increase the optimization problem difficulty.

Graph Consistency with Layouts. To evaluate how well the generated layouts conform to the conditioned graphs, we Matching Degree (k) RICO PubLayNet InfoPPT

60.61

41.47

79.24

63.38

87.90

80.57

measure the graph consistency in Table 4. Specifically, given a graph g and the corresponding generated layout l where a graph g′ which can be derived from, the consistency value λ can be calculated by matching the edges between g and g′ :

where the matching degree k indicates the direction closeness between two edges. For example, the degree of <above, above-left> is 1 and <above, left> is 2. From the table, we can see that the exact match of the conditioned graphs and generated layouts (k = 0) is challenging, with only 40%-60% of samples satisfying this hard matching. While setting loose the degree to k = 1, 2, the consistency values can be reached up to around 80%, which indicates that most generated layouts can reasonably conform to the conditioned graph.

Effect of Graph Sparsity. As wementioned in Section 3.1, complete graphs usually contain redundant edges and are likely to increase the learning difficulty for both the graph generator and layout decoder. Here we investigate the effect of graph sparsity on the overall performance. As shown in Figure 6, pruning the redundant edges to a reasonable percentage (up to 40%) can improve the quality of generated layouts. As the number of pruning edges continues to get increased, the performance deteriorates, which is expected since there will be more information lost in the pruned graph. This conclusion consistently holds when given different amounts of input constraints (0/25%/50%).

Generation Diversity. Given the user specification, a wellperformed system should generate diverse layouts satisfying the same input constraints. As shown in Figure 7, in terms of graph diversity, our model outputs different reasonable spatial graphs. Furthermore, for each conditioning graph, our model generates diverse layouts with varying sizes and positions.

5 Conclusion

User specifications are commonly used to constrain graphic layout generation. In this work, we introduce a graphconditioned layout generation system that accepts flexible user constraints. Our system consists of a graph generator and a layout decoder. Instead of the separate training pipeline, we propose a novel collaborative generation strategy to better utilize the two networks. Experiment results show the effectiveness of our approach. In the future, we aim for better solutions for generation control. Also, content-aware layout generation would be another promising direction.

Acknowledgements

This work was funded in part by the Major Research plan of the National Natural Science Foundation of China under number 92267203, in part by the National Key Research and Development Program of China under number 2019YFA0706200, in part by the National Natural Science Foundation of China grant under number 62076102, U1813203, and U1801262, in part by the Guangdong Natural Science Funds for Distinguished Young Scholar under number 2020B1515020041, in part by the Science and Technology Major Project of Guangzhou under number 202007030006, in part by the Science and Technology Program of Guangzhou under number 202002030250, in part by The Program for Guangdong Introducing Innovative and Entrepreneurial Teams (2019ZT08X214), in part by Guangdong-Hong Kong-Macao Greater Bay Area Center for Brain Science and Brain-Inspired Intelligence Fund (NO. 2019016).