Scan-and-Print: Patch-level Data Summarization and Augmentation for Content-aware Layout Generation in Poster Design

1 Introduction

Integrating artificial intelligence (AI) with art and creativity has emerged as a pivotal trend in the design domain. According to a survey of a mainstream design platform with over 185 million monthly users, 90% of respondents agreed that AI has improved their work [Canva, 2024]. Among these advancements, content-aware layout generation plays a crucial role in automating poster design [Lin et al., 2023b; Yang et al., 2023; Weng et al., 2024; Wang et al., 2024] by indicating the arrangement of visual-textual elements, e.g., logo, text, and underlay, on the background image, as shown in Fig. 1.

Despite the increasing attention to this valuable task [Zhou et al., 2022; Hsu et al., 2023b; Xu et al., 2023; Horita et al., 2024], existing methods faced a high computational bottleneck in perceiving images. Taking the current state-of-theart (SOTA) method, RALF [Horita et al., 2024], as an example, even when neglecting the time required for imageretrieval augmentation [Fu et al., 2024], it took an average of 385 ms per single inference on an NVIDIA A40 GPU. The saliency detection in preprocessing accounted for 55 ms, while more critically, the image encoders during generation occupied over 69.5% of the model parameters. This not only poses significant challenges for real-time performance, but the scarcity of training data relative to the model's capacity also impairs the model's generalization ability.

∗Corresponding author.

To relieve the inefficiency and overfitting issues in the field, we proposed Scan-and-Print, an auto-regressive model accompanied by patch-level data summarization and augmentation. Specifically, (a) the scan procedure selects only the few patches from an input image that are predicted to have a high probability of placing element vertices, thereby concentrating computational resources on the most applicable areas. Then, (b) the print procedure synthesizes augmented samples by mixing the patches and vertices from two imagelayout pairs, effectively increasing both the size and diversity of datasets. We also introduced a new (c) vertex-based layout representation to facilitate vertex-level mixup operations.

We conducted extensive experiments on the public benchmarks [Zhou et al., 2022; Hsu et al., 2023b] and demonstrated that Scan-and-Print has achieved new SOTA performance. Compared to RALF [Horita et al., 2024], it has drastically reduced 95.2% of the FLOPs in image encoders. The synthesized data have consistently shown positive impacts even when the augmentation rate reached more than 100%, highlighting their plausibility and usability. Besides, we demonstrated the adaptability of Scan-and-Print to user-specified constraints, particularly beneficial for the real-world poster design workflow.

The contribution of this work are summarized as follows:

• A data summarization approach (Scan) selects only the few patches suitable for arranging vertices of layout elements to efficiently perceive input image content.
• A data augmentation approach (Print) mixes the patches and vertices from two image-layout pairs to synthesize extensive new plausible samples at a low cost.
• A vertex-based layout representation (VLR) models fine-grained geometric properties to facilitate delicate vertex-level mixup operations across layouts.

• Comprehensive evaluations verify the practical application value of Scan-and-Print, being the first in the field to focus on reducing computational complexity.

2 RelatedWork

2.1 Content-aware Layout Generation

Different from general layout tasks [Li et al., 2020a; Weng et al., 2023], content-aware layout generation additionally takes into account the given background image, thus possessing high application value in the field of AI-empowered design [Wang et al., 2024; Weng et al., 2024]. Pioneered in CGLGAN [Zhou et al., 2022] and PKU PosterLayout [Hsu et al., 2023b] to establish the heatmap-based paradigm, i.e., utilizing object saliency [Li et al., 2021; Qin et al., 2022] or spatial density [Hsu et al., 2023a] maps to enhance the awareness of image composition. Although CGL and PKU contributed valuable datasets, the total size of training data remains scarce around 70.5k samples, urging for heuristic techniques or data augmentation approaches to improve model performance.

Inspired by the prior design experiences [Guo et al., 2021; Li et al., 2020b], DS-GAN [Hsu et al., 2023b] organized layout elements in a motivation-aligned order to mine patterns in the data more effectively during GANs' training. [Chai et al., 2023b] employed a general diffusion model [Chai et al., 2023a] for content-aware tasks by incorporating predefined aesthetic constraints and a saliency-aware layout plausibility ranker. In another way, RALF [Horita et al., 2024] incorporated retrieval augmentation by searching for the nearest neighbors of the input image [Fu et al., 2024] and using their layout features as additional input for autoregressive models. LayoutPrompter [Lin et al., 2023a] coarsely extracted the minimum bounding rectangle from the saliency maps and retrieved layout examples to enable the in-context learning of LLMs. PosterLlama [Seol et al., 2024] synthesized new image-layout samples by a depth-guided image generation with refined text descriptions [Zhang et al., 2023; Li et al., 2023], while maintaining the corresponding layouts unchanged, to fine-tune DINOv2 [Oquab et al., 2023; Zhu et al., 2023] and CodeLlama-7B [Roziere et al., 2023; Hu et al., 2021] successively.

However, along with these efforts, the size of model parameters, especially image encoders, often exceeds that of the available training data. This leads to significant challenges in real-time performance and generalization ability. In light of this, we devote this work to a compact model with selective scan and efficient data augmentation- print.

2.2 Patch-level Data Augmentation

Data augmentation is a crucial regularization method to enhance model generalization by artificially increasing the size and diversity of training data. In image understanding, traditional operations such as random cropping and flipping have been widely used. An advanced field is mixing-based data augmentation, which synthesizes new data by combining multiple samples. E.g., Mixup [Zhang et al., 2018] randomly drew two image-label pairs (xi, yi), (xj, yj) and linearly interpolated them to obtain the convex combination (˜x, ˜ y).

Upon the success of Mixup, patch-level mixing approaches have been developed to create more realistic samples that preserve the spatial structure of the images. CutMix [Yun et al., 2019] cropped a rectangular region of xi and filled the corresponding part of xj, showing particular advantages in localization tasks. As random selection sometimes results in mixed patches lacking supervisory information, saliency detection has been introduced into the process. Puzzle Mix [Kim et al., 2020] formulated an optimization problem, maximizing the exposed saliency, to jointly determine the size of the mixing mask and its spatial offset between the samples. SaliencyMix [Uddin et al., 2021] straightforwardly selected the peak salient area within xj and pasted it on xi. In contrast, Co-Mixup [Kim et al., 2021] sophisticatedly paired the samples from the mini-batch to obtain the largest possible accumulated saliency regions. GuidedMix [Kang and Kim, 2023] further sped up this complicated process by splitting apart the pairing and mask determination.

These efforts have consistently enhanced model robustness and generalization ability in deep learning-based computer vision methods [Bochkovskiy et al., 2020; Bang et al., 2022]. However, the labels y in previous tasks are often simplistic, e.g., one-hot vectors, which fail to serve the complex, hierarchical structure of layout elements. In light of this, we dis-

Algorithm 1 Vertex-based layout representation (VLR)

Input: Layout L = {ei}ni = {(ci, (xci , yci , wi, hi))}ni

Parameter: Category ID of underlay cund

Output: Start-end vertex tensor V

• 1: (xl, yt, xr, yb) ← CXCYWH-TO-XYXY((xc, yc,w, h))
• 2: G ← GROUP-ELEMENT-ID(cund, c, (xl, yt, xr, yb))
• 3: C ← REPEAT(c, n→ (n 2))
• 4: for i = 1 to n do
• 5: C2i ← 2C2i
• 6: C2i+1 ←C2i +1 ▷ Increment for end vertex
• 7: end for
• 8: X ← REARRANGE((xl, xr), n (2 d) → (n 2) d)
• 9: Y ← REARRANGE((yt, yb), n (2 d) → (n 2) d)
• 10: W ← 0.01×X +Y ▷Weights for top-left ordering
• 11: A←ARRANGE-SEVERTEX-ID(G,W)
• 12: V ← {(cA[i], (xA[i], yA[i]))}2n i
• 13: return V

3.2 Scan Procedure: Data Summarization

Image encoder with patch selection. In the target context, image understanding essentially boils down to a binary determination of whether specific areas are suitable for placing layout elements. Relative to its limited complexity, existing methods are prone to employing disproportionately heavy image encoders. This insight naturally leads us to a compact encoder focusing on a small proportion within the input image I. Preliminarily, a parameter-efficient density mapping network is pre-trained semi-supervised as in [Hsu et al., 2023a] to identify top-k patches {P(x,y) | (x, y)∈{P}} with the highest scores for placing element vertices, where |{P}|=k≪the number of patches p2 . This selection is performed after the positional embedding, so the subsequent encoder retains the original spatial information of P(x,y) as well, which is crucial for understanding global relationships.

cussed the vertex-level label mixup to mitigate this gap.

3 The Proposed Approach: Scan-and-Print

Considering the computational overhead and generalization issues of current content-aware layout generation approaches, we propose Scan-and-Print. It is an autoregressive model that achieves efficiency in parameter count, image perception, and data augmentation. An overview is shown in Fig. 2. Briefly, we propose (a) vertex-based layout representation (VLR) to capture the fine-grained structure of elements and facilitate the two procedures, namely, (b) scan that identifies and perceives a small size k of image patches applicable for element placement, and (c) print that synthesizes new training samples (˜ I, ˜L) by mixing patches and vertices across two imagelayout pairs (Ii, Li) and (Ij , Lj ) within the given mini-batch.

3.1 Vertex-based Layout Representation

Conventionally, a layout L is represented as a set of bounding boxes, each described by its center coordinates and size, i.e., {ei}ni = {(ci, (xci , yci ,wi, hi))}ni . With its simplicity comes the difficulty of modeling fine-grained spatial relationships and manipulating geometric properties precisely, leading to poor graphic quality, such as misalignment [Zhou et al., 2022]. To this end, we propose a more direct representation based on the vertices of boxes. As depicted in Algorithm 1, VLR first derives the top-left and bottom-right coordinates (xl, yt, xr, yb) of elements, and then performs GROUPELEMENT-ID to construct ID trees, which explicitly reflects the hierarchical structure in L. Subsequently, the categories and coordinates are transformed into the attributes of startend vertices. Inspired by [Hsu et al., 2023b], ARRANGESEVERTEX-ID is performed to sort and obtain the vertex tensor V ={(ci, (xi, yi))}2n i but only considers the reading order and grouping relationship. The detailed definitions of the two functions are presented in the supplementary material.

Image-to-layout alignment and autoregressive decoder. Following [Horita et al., 2024], a layout tokenizer is utilized to quantize x and y into 128 bins. To bridge the modality gap [Chen et al., 2024] between visual and geometric features, a two-layer FFN is inserted after the last layer of the encoder. Finally, an autoregressive decoder, trained through the next token prediction objective, sequentially predicts 6n tokensdetokenized as n elements afterward.

3.3 Print Procedure: Data Augmentation

Based on the concept of Mixup [Zhang et al., 2018], imagelayout pairs (Ii,Li) and (Ij,Lj) are picked from the given mini-batch to construct a convex combination (˜ I, ˜L), considering their patch indices {Pi} and {Pj}. In total, α new samples are created per mini-batch during training, effectively improving the model's generalization ability.

Image patchmixing. Ii and Ij are combined using a binary mask M ∈ {0, 1}p×p and its complement M ′ , defined as:

where ⊙ denotes patch-wise multiplication. This removes the applicable patches of Ii and pastes on the corresponding patches of Ij , making ˜ I the challenging case containing fewer patches suitable for placing elements than the sources.

Element vertex mixing. A stricter mask Sp×p considering {Pi ∩ Pj} is initiated to constrain this process, ensuring that ˜L is a plausible layout for the input ˜ I, and then, all continuous regions R with at least three available patches in S is found by depth-first search. Before mixing points in (Vi, Vj), the longest common subsequence (LCS) of their categories (Ci,Cj) is obtained as ˜ C of ˜L by dynamic programming. With indices vi/j = {INDEX-OF(LCSl)IN(Vi/j)}LENGTH(LCS) l and regions R, the points ( ˜X, ˜ Y ) are abstracted as:

This forms ˜ V , where all starting points come from Vi, and all ending points come from Vj. To increase randomness, vi/j and R are shuffled carefully before this process. Finally, the shifting operation is defined to move each pair (Vs, Ve) of start-end points into the corresponding region r while keeping their relative positions in the original patch, as:

where (Iw, Ih) is the input image size, and LEFT-TOP, RIGHTBOTTOM return (xl, yt) coordinates of the specified patch.

4 Experiments

4.1 Datasets and EvaluationMetrics

To evaluate the proposed Scan-and-Print, we conduct experiments on widely used e-commerce poster datasets, PKU PosterLayout [Hsu et al., 2023b] and CGL [Zhou et al., 2022]. Their train/annotated test/unannotated test splits are allocated following [Horita et al., 2024] to ensure a fair comparison with existing work. Concretely, PKU PosterLayout contains 8,734/1,000/905 samples with three element types, which are logo, text, and underlay. CGL contains 54,546/6,002/1,000 samples with four element types, where the extra one is embellishment.

Following the above work, we evaluate layouts in two aspects, including (a) graphic metrics: overlay Ove↓, alignment Ali↓, loose, and strict underlay effectiveness Undl ↑, Unds ↑, and (b) content metrics: space utilization Uti ↑, salient object occlusion Occ↓, and readability Rea↓.

4.2 Implementation Details

We implement the image encoder and layer decoder with the first 8 layers of ViT-S [Touvron et al., 2022] and a 4-layer Transformer Decoder, respectively. The size of the input image I is (224, 224) and the embedding dimension is 384. We set the batch size, epoch, learning rate of the encoder, and of the others as 128, 15, 5e-5 , and 5e-4 . Considering the available data, the scan size k is 96 and 48 for PKU PosterLayout and CGL, and the augmented sample size α is 256 and 16, creating approximately 17,644 and 6,816 samples per epoch. All experiments are carried out on an NVIDIA A40 GPU.

4.3 Comparison with State-of-the-arts

We select approaches with open-sourced implementation as baselines, including the GAN-based CGL-GAN [Zhou et al., 2022], DS-GAN [Hsu et al., 2023b], autoregression-based ICVT [Cao et al., 2022], AutoReg [Horita et al., 2024], RALF [Horita et al., 2024], diffusion model-based LayoutDM † [Inoue et al., 2023], and LLM-based PosterLlama ‡ [Seol et al., 2024], LayoutPrompter § [Lin et al., 2023a].

Baseline comparison. As reported in Tab. 1 and Tab. 2, Scan-and-Print consistently achieves new SOTA performance across different benchmarks, especially on the severely datascarce PKU PosterLayout. Compared to existing LLM-based methods that represent layouts as structured language and utilize LLMs' coding abilities, ours shows unprecedentedly comparable graphic effectiveness with only 26M parameters while significantly improving content metrics. Particularly, it outperforms the SOTA approach, e.g., PosterLlama, by 28.7% and 28.4% in Occ ↓ across two benchmarks. On the other hand, when compared to the non-LLM-based SOTA approach, e.g., RALF, it shows an overall superiority, especially improves Unds ↑ by 12.2% and 4.5%. These observations demonstrate that the proposed Scan-and-Print can generate visually appealing layouts, ensuring that (1) salient objects in the input image are not occluded and (2) complex structural elements are correctly organized.

† The extended version presented in [Horita et al., 2024].

‡ With the released CodeLlama-7B weight tuned on [Hsu et al., 2023b], [Zhou et al., 2022], and depth-guided augmented data.

§ With the Llama3.1-8B weight [Dubey et al., 2024].

Results on the annotated splits are also reported, as in Tab. 3 and Tab. 4. We found the layouts generated by Scan-andPrint are already very close to the quality of ground truth data and even show better performance in Ali ↓ and Occ ↓. While some metrics on CGL are slightly behind RALF, the overall performance of Scan-and-Print presents a good trade-off considering its reduced computational complexity.

Computational cost. The parameter counts and FLOPs of different approaches are reported in Tab. 1. As observed, the total #Params in Scan-and-Print is only 61% of that in RALF, even less than #Params of RALF's image encoders, which is 2× the Scan-and-Print's image encoder. This efficiency is further strengthened by the proposed scan procedure. Specifically, when the size k is set to 96, our image encoder consumes only 1.46G FLOPs, which is 17% of the 8.51G FLOPs required by RALF. More exploration of k values will be reported in the ablation study.

Visualized results. Fig. 3 shows the layouts generated by different methods. The results illustrate that Scan-and-Print specializes in organizing combinations of elements rarely or never seen in datasets to suit applicable areas of diverse sizes and distributions, which is the charm of Print- the mixing-based data augmentation, while Scan ensures these elements are properly placed and reduces undesirable occlusions. Specifically, seeing the images in the first column, i.e., (a) and (f), although their only objects leave enough available spaces, layouts generated by existing methods still have minor flaws. In contrast, ours generates nearly perfect ones that actively utilize most spaces, creating more and betterorganized elements. Moving on to the second column, as the complexity of objects in the image increases, the negative effect of overfitting shows and tends to place elements at the upper center, just like most training data. When the distribution of objects becomes dispersed and not centered, as in the third and fourth columns, or when there is very little available space, as in the fifth column, Scan-and-Print consistently generates visually appealing layouts, making full use of all available element types, even those that constitute a very small fraction of the training data, e.g., embellishment.

More visualized results of (1) constrained generation task, discussed in the next paragraph, and (2) mixed-up samples are presented in the supplementary material.

User-specified constraint. Following RALF, we also explore constrained generation and demonstrate the adaptability of Scan-and-Print to the Category → Size + Position task. As reported in Tab. 5, ours maintains a leading position in most metrics compared to RALF, showcasing its versatility to meet real-world needs. Moreover, Scan-and-Print achieves an average inference time of 267.8 ms per single inference, which is only 70% that of RALF (384.5 ms). This efficiency enables users to quickly explore a variety of high-quality layout options for their materials.

4.4 Ablation Study

To gain insight into our implementation choices, including (1) each component, (2) the scan size k, (3) mixed-up pairs selection strategy, and (4) the augmented sample size α per mini-batch, we conduct extensive ablation studies with PKU PosterLayout, unannotated test split.

Effectiveness of each component. As the comprehensive analysis provided in Tab. 6, all components of the Scan-andPrint have contributed positively to its performance. Starting from the first row, which is our baseline with a compact architecture, it has already outperformed the current SOTA approach, e.g., RALF, in almost all metrics. More concretely, it falls behind only in Occ ↓ by a negligible 0.0003. This finding remarkably confirms the point we made in Introduction that the model complexity of existing methods has exceeded the size of available data supports, leading to counterproductive outcomes. More findings are as follows:

Row 1 → 2. When replacing conventional representation with the proposed VLR, an overall improvement is witnessed, especially in Unds ↑ by 5.7%. While this is as expected, since VLR captures the fine-grained structure of layout elements, what impresses us is the coherent improvements in all content metrics, proving the value of the new representation.

Row 1 → 3. When joining the scan procedure, only a slight performance degradation comes with substantial computational cost savings. This strongly supports our view that the complexity of image perception in the target task is limited, verifying our decision to reduce its cost in pursuit of inference speed.

Row 2, 3 → 4. When involving both VLR and the scan procedure, their advantages are well combined, showing a good trade-off between effectiveness and computational efficiency.

Row 4 → 5. Finally, introducing the print procedure obtains the best results. It compensates for the slight visual information loss caused by the patch selection and further improves most graphic metrics through the diverse augmented samples. The only noticeable decrease in Uti ↑ is attributed to the design of image mixing (Sec. 3.3) that tends to synthesize difficult cases of very few applicable patches, hence the Uti ↑ in augmented samples is lower than the sources. Notably, the print procedure incurs only a time cost of 3.2 s per mini-batch and 54.8 min over the entire training period.

Exploration of different scan size k. Tab. 7 observes the impact of varying the perceived patch number on the performance. We experiment with candidates k ={96, 48, 24} and also the full-size scenario. Surprisingly, compared to the full size, k=96 leads to an overall improvement in content metrics, which suggests that perceiving less applicable patches can introduce undesired noises. In contrast, if k becomes smaller, where insufficient informative patches are selected, a downward trend of content metrics appears as expected. Another insight is that when the model pays less attention to visual information, layout features become more dominant and improve graphic metrics. Furthermore, it is important to highlight that all these results consistently outperformRALF. When k = 24, the cost saving significantly reaches 95.2% compared to RALF, which requires 8.51G FLOPs.

Different ways of selecting mixed-up pairs. Selecting the mixed-up sources can be crucial [Kim et al., 2021] to affect the quality and diversity of the augmented data, which in turn the model's performance. Therefore, we experiment with three different strategies, including two based on patch indices ({Pi}, {Pj}), Pearson correlation coefficient (PCC) and cosine similarity (COSIM), as well as a data-agnostic random strategy. As reported in Tab. 8, PCC achieves the best overall results, particularly excelling in Uti ↑, which is most impacted by the synthesized challenging cases, as analyzed in previous studies. Since PCC and COSIM tend to select pairs with similar patch indices, their leading position is expected. Nevertheless, the random strategy achieves amazingly good results in other metrics. This finding demonstrates the robustness of the proposed mixup operations, which regains the advantages of their original concepts [Zhang et al., 2018], namely, high adaptability and no stringent prerequisites for the existing training data.

Impacts of augmented sample size α. Last but not least, Tab. 9 provides insights into the augmented sample size per mini-batch. We experiment with candidates α={32, 64, 128, 256} given a fixed mini-batch size 128. As observed, the best outcomes are found at α=128, whereas additional improvement is witnessed in all content metrics when α keeps increasing, demonstrating the potential of involving more augmented data for further advancement. It is worth reiterating that the cost of creating these samples is extremely low. Even at α=256, it takes only 3.2 s per mini-batch, as reported in the previous study. However, the additional samples do consume more GPU memory and result in longer training times, which is a trade-off to consider. By investing more efforts in the training stage, the proposed Scan-and-Print has successfully demonstrated excellent real-time performance and generalization capabilities in the inference stage.

5 Conclusion

This work discussed the common pitfalls of current contentaware layout generation methods, i.e., high computational costs and low generalization ability. These defects are attributed to the large number of parameters relative to the limited available training data, and the image encoder is further identified as the culprit. To address these challenges, we presented a compact autoregressive model accompanying the proposed patch-level data summarization and augmentation approach, Scan-and-Print. Through extensive experiments, we demonstrated that it has achieved new SOTA results across various benchmarks. Moreover, compared to the previous SOTA method, it has saved 95% computational cost in image perception, required only 61% of the parameters, and reduced inference time to 70% in generation. To advance the field further, we discuss potential directions for future work in the supplementary material.

Acknowledgments

This work was supported by the grants from the National Natural Science Foundation of China (62525201, 62132001, 62432001) and Beijing Natural Science Foundation (L247006).