GarmentDiffusion: 3D Garment Sewing Pattern Generation with Multimodal Diffusion Transformers

1 Introduction

Digital garment modeling has emerged as a pivotal research area for fashion design, with garment sewing patterns being essential in transforming design concepts into tangible garments. Many studies have been conducted on sewing pattern modeling and generation, either by scaling up the size of datasets [Korosteleva and Lee, 2021; Korosteleva et al., 2024] or by proposing new parametric and learning-based approaches [Korosteleva and Lee, 2022; Korosteleva and Sorkine-Hornung, 2023; Liu et al., 2023; Chen et al., 2024a; He et al., 2024].

The first attempt [Korosteleva and Lee, 2021] was made to build a synthetic garment sewing pattern dataset using a parametric approach, with ˜20K sewing patterns. However, both the complexity of sewing pattern geometries and the quantity of sewing patterns are insufficient to meet the increasing data demands of advanced data-driven models. Subsequently, in [Korosteleva et al., 2024], the authors built a large-scale garment dataset, i.e., GarmentCodeData, using component-oriented garment programs [Korosteleva and Sorkine-Hornung, 2023]. It introduces more complex design prototypes and scales up the number of sewing patterns by 5×, with ˜115K sewing patterns in total. However, learning-based models typically require paired samples for conditional training, such as (sewing pattern, modality-specific input). The unimodal nature of these datasets restricts the development of generative modeling approaches. SewFactory [Liu et al., 2023] has recognized this problem and provides approximately 1M imageand-sewing-pattern pairs for training. Since its design prototypes are derived from [Korosteleva and Lee, 2021], the complexity of sewing pattern geometry is still limited.

Another challenge lies in efficiently modeling the generation of sewing patterns to make it more applicable to realtime scenarios. SewFormer [Liu et al., 2023] introduces a DETR-like [Carion et al., 2020] discriminative model to map 2D images to sewing patterns. The discriminative training of SewFormer leads to the deterministic predictions of sewing patterns given input images, which restricts the diversity of garment designs compared to the generative approaches. A pioneering work, DressCode [He et al., 2024], introduces a GPT-like autoregressive model (SewingGPT) to generate vector-quantized sewing patterns, conditioned on the text descriptions via cross-attention. While this method is effective in generating simple sewing patterns, it faces significant challenges when applied to GarmentCodeData [Korosteleva et al., 2024]. For example, the token sequence length of SewingGPT increases from ˜2K to over 18K, making its training and inference infeasible in practice. Another issue is the coarse sewing pattern descriptions generated by GPT-4V using its data annotation pipeline, which lacks the precise control for text-conditioned sewing pattern generation.

In this paper, we rethink the modeling paradigm of sewing patterns, questioning whether the vector-quantized encoding scheme and autoregressive next-parameter prediction in DressCode are efficient for sewing pattern generation. Inspired by BRepGen [Xu et al., 2024], we encode edge-related parameters (such as 3D coordinates, stitch tags, and free edge scores) into the embedding dimension, while denoising all edge tokens in parallel along the temporal axis. By leveraging the parallel processing nature of diffusion transformers [Peebles and Xie, 2023], our approach accelerates the generation process by approximately a hundredfold without sacrificing the precision (in centimeters) of sewing pattern geometries. Specifically, with parameters set to #edge parameters/edge = 9 (endpoints, control points, arc), #edges/panel = 39, and #panels/pattern = 37, SewingGPT requires 18, 135 + 2(SOS,EOS) tokens and steps to autoregressively generate a sewing pattern. In contrast, our model only needs 1,443 tokens for generation, with a constant denoising step independent of dataset statistics. Furthermore, following [Khan et al., 2024], we redesign the data annotation pipeline for both DressCodeData [Korosteleva and Lee, 2021; He et al., 2024] and GarmentCodeData, to provide both brief and detailed text descriptions for sewing patterns. To support the image modality as input, we employ commonly used garment sketches as the interactive interface between users and models. As a benefit of our modeling paradigm, we also support sewing pattern completion using user-provided incomplete patterns as input for controllable generation.

To sum up, our contributions to the community are as follows:

1. We present a new generative model, GarmentDiffusion, pushing the limits of diffusion-based modeling paradigm for multimodal sewing pattern generation.
2. We propose an efficient edge encoding scheme that significantly reduces the token sequence length of sewing patterns, achieving a substantial speedup compared with the autoregressive approach, i.e., DressCode.
3. We validate the effectiveness of our model on SewFactory, DressCodeData, as well as the largest and most challenging GarmentCodeData, and establish a strong baseline with comprehensive and quantitative evaluation metrics.
4. We provide new multimodal data annotation pipelines that can generate both brief and detailed text descriptions, as well as garment sketches for sewing patterns, enabling multimodal sewing pattern generation.

2 RelatedWork

2.1 Garment Sewing Pattern Generation

Existing research on garment generation can mainly be divided into 3D-based and sewing pattern-based approaches. The 3D-based methods generates garment models through Gaussian splatting guidance [Li et al., 2024], unsigned distance function regression [Moon et al., 2022; Zheng et al., 2024], neural volumetric rendering [Chen et al., 2024b], or latent representation learning [Su et al., 2023; Shen et al., 2020; Srivastava et al., 2025; Shao et al., 2024]. However, these garment models often have topological imperfections that make them unsuitable for manufacturing.

To generate production-ready sewing patterns, early methods included scanned model flattening [Bang et al., 2021] and iterative panel parameter optimization [Wang et al., 2018]. With the release of large sewing pattern datasets [Korosteleva and Lee, 2021; Korosteleva et al., 2024], there has been a shift toward data-driven approaches. SPnet [Lim et al., 2024] predicts sewing patterns from generated T-pose images, while Neural Sewing Machine [Chen et al., 2022] utilizes principal component analysis to create sewing pattern masks. To reduce reliance on templates, Korosteleva and Lee [2022] predict edges and stitches directly from 3D point clouds. Liu et al. [2023] and Chen et al. [2024a] use hierarchical transformers to recover sewing patterns from images. He et al. [2024] generate garment sewing patterns with textures, guided by natural language descriptions. At the same time as our work, Design2GarmentCode [Zhou et al., 2024] proposes a DSL-oriented multimodal agent and leverage the garment programs to generate sewing patterns. ChatGarment [Bian et al., 2024] fine-tunes a VLM to generate garment specification files. AIpparel [Nakayama et al., 2024] builds on top of LLaVA-1.5 [Liu et al., 2024a] and proposes to train a multimodal pattern generation model using both discrete and continuous training objectives. SewingLDM [Liu et al., 2024b] applies a latent diffusion model with two-stage training to generate patterns, incorporating various handcrafted losses. Our model adopts a single-stage training with MSE loss, achieving improvements in modality diversity, performance, and efficiency.

2.2 Conditional DiffusionModels

A large portion of current research on diffusion models originates from [Ho et al., 2020; Rombach et al., 2022], with a forward chain that perturbs data into noise and a reverse chain that converts noise back into data. Diffusion Transformers (DiT) [Peebles and Xie, 2023] explore replacing the U-Net backbone with a transformer that operates on latent patches, achieving better scalability. IP-Adapter [Ye et al., 2023] introduces the decoupled cross-attention mechanism to achieve the image-prompt conditional generation.

Diffusion models have also proven successful in generating CAD models. For example, Xu et al. [2024] present a diffusion-based approach that unconditionally outputs boundary representation of CAD models. Wang et al. [2024] utilize a vector-quantized diffusion model to generate command sequences from design concepts represented by text or sketches. The success of diffusion models in synthesizing structured CAD models suggests that similarly structured sewing patterns could also be generated through a reverse diffusion process.

3 Method

3.1 Sewing Pattern Representation

Edge representation. A 3D pattern consists of variable number of panels, with each panel being a closed shape made up of multiple edges. Since the edges are connected end-toend, we only need a start point ej ∈ R3 and control points cj ∈ R3 of B´ezier curves to represent the geometry of j th edge Ej, where ej := (ex, ey, ez)j and cj := (cx, cy, cz)j are the 3D point coordinates. The number of control points depends on whether the B´ezier curve is quadratic or cubic. We represent the circular arc using another three parameters aj ∈ R3 , where aj := (ax, al, as)j represents the radius, major or minor arc and sweeping orientation, respectively. Note that if Ej is a linear curve, cj is identical to ej. Furthermore, we use a per-edge stitch tag sj ∈ R3 and a binary stitch flag fj ∈ {0, 1} to encode the stitch information of edges. sj := (sx, sy, sz)j is calculated as the averaged 3D midpoint between matched edge pairs. All coordinates are calculated after performing 3D rotation and translation for each panel. Ej is thus represented as ej ⊕ cj ⊕ aj ⊕ sj ⊕ fj with the appropriate zero padding, where ⊕ represents the concatenation along the embedding dimension. Note that the number of parameters of Ej, denoted as |Ej|, could be variable length depending on the dataset.

Pattern representation. Suppose that a dataset contains at most M panels for all patterns, and each panel contains at most N edges. We pad the pattern P ∈ R|m×n|×|Ej| that has m panels (m ≤ M) and n edges per panel (n ≤ N) to the uniform M × N sequence length, which is denoted as P′ ∈ R|M×N|×|Ej| . That is, all panels {Pi}Mi>m are set to 0, and all edges {Ej}Nj>n are set to 0 as well, where i and j denote the index of panel and edge, respectively. The hierarchical pattern representation is illustrated in Figure 2. This edge-oriented pattern representation largely shortens the token sequence length compared to the coordinate-oriented representation in DressCode. Different from the "token-bytoken" causal generation, all these tokens can be processed by subsequent diffusion transformers in parallel.

3.2 Pattern Generation with GarmentDiffusion

Our model follows the design of DiT [Peebles and Xie, 2023] architecture. It accepts multimodal inputs to control the generation of sewing patterns. During the training phase, all edges are converted into token representations, followed by random panel shuffling and noise corruption. The model is trained to predict the noises added to the edge tokens. In the generation phase, the edge tokens are initialized as random Gaussian noises, and are iteratively denoised using the predicted noise. The entire framework is illustrated in Figure 3.

Pattern preprocessing. Suppose we have P′ ∈ R|M×N|×|Ej| , which consists of {Pi}Mi=1 with {Pi}Mi>m being panel-level padding. Each Pi consists of {Ej}Nj=1 with {Ej}Nj>n being edge-level padding. We shift and scale each dimension of Ej using respective parameter statistics to ensure that the value range is between -1 and 1. To support the pattern completion, we randomly shuffle {Pi}mi=1 to break the predefined panel order, while keeping the order of edges {Ej}nj=1 within a panel unchanged. We always place the padding tokens after the shuffled edge tokens.

Token embeddings. To distinguish the edge tokens among different panels, we construct a panel-level look-up embedding table EmbP ∈ RM×C , where C is the embedding dimension of the model. We also construct an edge-level lookup embedding table EmbE ∈ RN×C to encode the sequential edge order within a panel. To represent each time step t, we use the conventional sine and cosine positional encoding to construct the time embedding t ∈ RC . We employ a DDPM noise scheduler to obtain the noise-corrupted edge token, that is ddpm scheduler.add noise(Ej, ϵ, t) → ˜Ej with ϵ ∼ N(0, I). Finally, we compute the edge token embedding as:

where φ(·) and T (·) are the projections, each comprising two linear layers with a non-linear activation function, to match the model's dimension.

Conditional training. To achieve both text-to-pattern and image-to-pattern generation with fine-grained control, we inject the conditions using cross-attention layers rather than adaLN-Zero layers [Peebles and Xie, 2023]. Specifically, we follow the practice [Ye et al., 2023] to employ decoupled cross-attention layers that use separate key and value projection matrices to process text and image features, while using shared query projection matrices among different modalities to process edge features. The multimodal cross-attention operation is defined as:

where Z′ is the fused multimodal latent features; Q is the edge features from Z after query projection; KT , VT , KI , VI are the text and image features after key and value projection. The text and image features are extracted using CLIP's text and image encoders [Radford et al., 2021]. We use both class and patch embeddings before the last projection layers of CLIP for the conditional training. The image features are projected into the same dimension as the text features by a two-layer MLP before fed into the diffusion transformer. All the parameters of the diffusion transformer are trainable. The training objective of our model is a simple L2 loss, which minimizes the mean-squared error between the sampled Gaussian noise and the predicted noise. That is:

where xt is the edge token embeddings at time step t; cT and cI are the text and image conditions, respectively.

4 Experiments

4.1 Datasets

We use SewFactory [Liu et al., 2023], DressCodeData [Korosteleva and Lee, 2021; He et al., 2024] and GarmentCodeData (V2) [Korosteleva et al., 2024] for training and evaluation. For SewFactory, we employ off-the-shelf rendered garments superimposed on diverse human poses as image prompts (without text prompts). For DressCodeData and GarmentCodeData, we designed multimodal data annotation pipelines (depicted in Figure 4) to generate both text and image prompts for sewing patterns. SewFactory consists of 13,707 sewing patterns, featuring a maximum of 14 panels and 12 edges per panel. DressCodeData contains 19,683 patterns, each with up to 10 panels and 10 edges per panel. GarmentCodeData offers 115,195 patterns, with a maximum of 37 panels and 39 edges per panel. Since the official splits of SewFactory are not available, we use our own version that 90% of randomly selected data points are used for training, with the remaining 10% evenly divided for validation and testing. For DressCodeData and GarmentCodeData (V2), we adhere strictly to the official splits provided by the authors for training, validation, and testing. Note that SewingGPT cannot be trained on the entire GarmentCodeData due to its long context length. To address this problem, we created a subset of GarmentCodeData by filtering out those patterns with #edges/panel > 12 and #panels/pattern > 10.

4.2 Multimodal Data Synthesis

Text-prompt generation. To enhance our model's comprehension of multi-level design concepts from users, we design two-level text descriptions for the sewing patterns: a concise category-level summary with basic design features, and detailed component-level descriptions. Leveraging LLMs' incontext understanding, our annotation pipeline first filters irrelevant information from garment specification files, then prompts Llama-3.1-8B-Instruct [Grattafiori et al., 2024] using a unified prompt to generate the multi-level descriptions automatically, as shown in Figure 4.

Image-prompt generation. We select the garment sketches as the interactive interface between users and our model. To generate the garment sketches that closely emulate the hand-drawn style of professional garment designers, we initially render the 3D garment models (in .obj and .ply formats) of DressCodeData and GramentCodeData into 2D garment images using Blender's APIs [BlenderAuthors, 2024]. Subsequently, we utilize MistoLine [Zhang et al., 2023] and Anything-XL fine-tuned from SD-XL [Podell et al., 2023] to extract the garment sketches, followed by a binarization operation.

4.3 EvaluationMetrics

We adopt the same evaluation metrics as [Liu et al., 2023] and [Korosteleva and Lee, 2022] to assess the fidelity of the generated sewing patterns. Specifically, Panel L2 denotes the L2 distance of the coordinates of 2D panels (converted from 3D coordinates) between predictions and ground truths, with the centroids of panels shifted to the origin. #Panel and #Edge represent the accuracy of correctly predicted patterns within all patterns, based on the number of panels in each pattern and the number of edges in each panel, respectively. Rot L2 and Trans L2 represent the L2 distances of x, y, z rotation Euler angles and universal x, y, z translations of panels between predictions and ground truths. Precision, Recall, and F1 Score are used to measure the false positives and false negatives of paired stitches with respect to all edge relations.

4.4 Implemention Details

Architecture. We adopt OpenAI ViT-H/14 (336×) [Radford et al., 2021] as our text and image encoders. Since the embedding dimensions of the text and image features are 1,024 and 768, we project the image features into 768dimensional vectors to match the text feature dimension. The main body of our model consists of 12 DiT blocks. Each block consists of a self-attention layer, a multimodal crossattention layer and a feed-forward layer, all utilizing pre-layer normalization [Ba et al., 2016; Xiong et al., 2020]. The number of heads for each attention layer is set to 8. The embedding dimension of the DiT blocks is 768, while the feedforward layers have an embedding dimension of 1, 024.

Look-up embedding tables. The panel-level and edgelevel embedding tables contain M and N learnable positional embeddings, where M is the maximum number of panels and N is the maximum number of edges per panel in each dataset. For SewFactory, M = 14, N = 12. For DressCodeData, M = N = 10. For GarmentCodeData, M = 37, N = 39.

Training details. We adopt a DDPM noise scheduler for diffusion training, with a maximum of 1, 000 denoising steps and a linear beta scheduler (beta start = 1×10-4 , beta end = 2 × 10-2 ). We use the AdamW optimizer [Loshchilov and Hutter, 2019] with betas = (0.95, 0.999), a constant learning rate of 1×10-4 and the weight decay of 1×10-2 . The training epoch is set to 1, 000 with an early-stop criterion. We evaluate the model at denoising steps of 50, 200, 500, and 1000 every 10 epochs. Based on the results shown in Figure 5, we select 50 denoising steps for inference. The multimodal training is performed in a round-robin fashion, following the order of image prompts, text prompts and imageand-text prompts. Our model is distributedly trained across 8 A10 GPUs (24GB) with the Hugging Face Accelerate library [Gugger et al., 2022].

4.5 Comparison with State-of-the-Art Methods

Compared with SewingGPT. Our evaluation metrics assess the accuracy of geometric structures, panel placement in 3D space, and stitching relations. Table 1 shows the evaluation results for SewingGPT and GarmentDiffusion, using text prompts from two different annotation pipelines. The first three rows show that the text descriptions generated by GPT4V with DressCode's pipeline result in worse performance than ours. This is expected, as we prompt the Llama3.1-8BInstruct (text-only) [MetaAI, 2024] with precise design specifications, which are used to generate sewing patterns through programs. Moreover, our method outperforms SewingGPT when trained with the same captions, highlighting the advantage of our approach.

We also evaluated SewingGPT on the subset of GarmentCodeData. As shown in Table 2, our method outperforms SewingGPT in panel and edge number accuracies, geometrical shapes, and 3D placement of panels. However, it lags in stitching edge prediction, likely due to the lack of stitching information in the prompt.

As shown in Figure 7, our model also supports pattern completion with incomplete patterns provided by users. It achieves strong control even though the text descriptions are not precise.

Compared with SewFormer. We also trained SewFormer and GarmentDiffusion using our training split as described in Section 4.1, while maintaining the same evaluation protocols and test set to ensure fair comparison. Table 3 demonstrates that our generative model achieves comparable performance to SewFormer in terms of the geometrical shapes and 3D panel placement, while it surpasses SewFormer in terms of panel and edge accuracies. These results confirm the effectiveness of our method, even when using multi-pose rendered images as prompts.

4.6 Ablation Study

Inputs of different modalities. To assess the impact of training on model performance under different combinations of input modalities, we trained three models using text, image, and multimodal prompts on the GarmentCodeData, respectively. As shown in Table 4, under the multimodal training setting, evaluation metrics are gradually improved when fine-grained conditions are incorporated for generation. Specifically, the combination of detailed text descriptions and sketches achieve the best performance. This conclusion is also supported by the middle section of the table. Compared with the model trained using text-only prompts, the model trained with image-only prompts exhibits slightly better performance. This is reasonable, as images may convey more information than texts.

Condition injection method. In addition to employing cross-attention for conditional training, we also experimented with adaLN-Zero conditioning proposed in DiT [2023]. We trained multimodal models (text & image) with both conditioning methods on the augmented DressCodeData. For cross-attention, we evaluate the model under text, image, and text-and-image conditions, while for adaLN-Zero, we used text and image conditions with equal probability. As shown in Table 5, although adaLN-Zero exhibits advantages when evaluated with brief text descriptions, it becomes less effective with detailed text descriptions and image inputs due to insufficient conditional information extraction. Cross-attention achieves the best overall performance when both detailed text and image inputs are provided.

4.7 Limitations and FutureWorks

While current annotations provide detailed sewing pattern descriptions, they still lack stitching information on edge and panel connectivity, which can compromise garment simulation. The annotation engine thus remains improvable. Additionally, current methods offer limited control via numerical parameters (e.g., panel/edge count) or human body measurements. From an efficiency standpoint, reducing denoising steps is also desirable. Future work will target these challenges by enhancing controllability and generation efficiency.

5 Conclusion

In conclusion, we introduced GarmentDiffusion, a much under-explored research direction for sewing pattern generation. Our model incorporates the design of diffusion transformers with an efficient edge encoding scheme. The architecture and training of our model are simple yet efficient, enabling the end-to-end generation of centimeter-precise and vectorized 3D sewing patterns. Our experimental results demonstrate the effectiveness of our method, which bridges the gap between creative garment design and manufacturing through scalable, precise, and efficient generative modeling. This work lays the foundation for advancing AI-driven fashion technology, seamlessly connecting digital design with practical garment production.