SketchAgent: Generating Structured Diagrams from Hand-Drawn Sketches

1 Introduction

Hand-drawn sketches are a natural and powerful medium for rapidly conveying ideas, serving as a universal language in creative, technical, and educational workflows [Zhao and Lai, 2022; Zhao et al., 2024; Tan et al., 2024]. From rough brainstorming sessions to preliminary engineering designs, sketches offer an intuitive way to externalize concepts. However, translating these informal and ambiguous drawings into structured, machine-readable diagrams remains an open challenge. Unlike natural image generation tasks [Cao et al., 2024; Huang et al., 2024; Li et al., 2019], which have seen remarkable progress in recent years through techniques such as controllable natural image generation, the sketch-to-diagram task demands more than just visual fidelity-it requires understanding and formalizing the underlying structural and semantic relationships inherent to diagrams.

†Corresponding author.

We introduce a new task, sketch-to-diagram generation, which involves converting a hand-drawn sketch into a structured, machine-readable diagram. As shown in Figure 1, this task differs fundamentally from controllable natural image generation, as it focuses not on generating aesthetically pleasing visuals but on synthesizing a precise, semantically meaningful diagram that adheres to specific structural rules. This transformation requires solving several core challenges: (1) handling the inherent ambiguity and variability in freehand sketches, (2) preserving the spatial and structural relationships between diagram components, and (3) producing an output that is both syntactically valid and semantically aligned with the user's intent. These challenges make sketch-to-diagram generation a highly specialized and underexplored problem, distinct from existing works.

To address the lack of standardized resources for sketch-todiagram research, we introduce the Sketch2Diagram Benchmark, a comprehensive dataset and evaluation framework designed to support the development and assessment of models for this task. The dataset spans eight diverse diagram categories, including flowcharts, directed graphs, and model architectures, and consists of over 6,000 high-quality examples. Each example includes a hand-drawn sketch paired with its corresponding structured diagram representation. The dataset is meticulously curated, featuring token-level annotations, standardized preprocessing, and rigorous quality control, ensuring its reliability for both training and evaluation purposes.

Building on this benchmark dataset, we propose SketchAgent, an end-to-end system for automating the transformation of hand-drawn sketches into structured diagrams. The system begins by converting an input sketch into a symbolic code representation, which abstracts its structural and spatial properties into a machine-readable format, bridging the gap between informal freehand drawings and precise computational diagrams. From there, SketchAgent performs iterative refinement to improve the accuracy, coherence, and validity of the code representation, ensuring the final diagram adheres to the user's intent while satisfying all structural constraints.

Our main contributions are as follows:

• We formally define the task of converting hand-drawn sketches into structured diagrams, distinguishing it from related tasks such as controllable image generation.
• We introduce a benchmark dataset of hand-drawn sketches and their corresponding structured diagram representations, offering a standardized resource for training and evaluation.
• We propose SketchAgent, a modular system that automates the sketch-to-diagram transformation process, integrating sketch recognition, symbolic reasoning, iterative refinement, and verification into a unified pipeline.

2 RelatedWork

2.1 Controllable Image Generation

Controllable image generation aims to synthesize images that adhere to specific constraints [Cao et al., 2024; Huang et al., 2024]. Existing methods can be categorized into three main approaches: GAN-based, diffusion-based, and multimodal fusion techniques. Early GAN-based methods, such as ControlGAN [Li et al., 2019], introduced fine-grained text-conditioned image synthesis but suffered from instability and mode collapse. Diffusion models have since become the dominant paradigm, offering more stable and highquality generation. Diffusion Self-Guidance [Epstein et al., 2023] and MultiDiffusion [Bar-Tal et al., 2023] enable explicit control over object positioning and spatial structure, while Control-GPT [Zhang et al., 2023b] leverages GPT-4generated sketches for improved spatial consistency. Other approaches integrate LLMs or additional modalities for enhanced control, such as MoMA [Song et al., 2025], which fuses textual and visual embeddings, and MM-Diff [Wei et al., 2024b], which refines personalization through CLIPbased representations. Furthermore, PALP [Arar et al., 2024] enhances alignment with complex textual prompts by optimizing cross-modal score matching. Despite these advancements, existing approaches primarily focus on photorealistic image synthesis, making them insufficient for structured and logic-constrained generation tasks like diagrams [Cao et al., 2024; Huang et al., 2024]. While diffusion-based models offer control over spatial attributes [Epstein et al., 2023; Bar-Tal et al., 2023], they lack explicit structural reasoning capabilities required for diagram generation.

2.2 Controllable Code Generation

Controllable code generation aims to produce structured and executable code while adhering to specific constraints [Shin and Nam, 2021; Wei et al., 2025]. Language model-based approaches leverage pre-trained models to improve code synthesis. Magicoder [Wei et al., 2024a] enhances multilanguage code generation through OSS-INSTRUCT, while VeriGen [Thakur et al., 2024] tailors language models for Verilog synthesis by curating specialized training datasets. Structure-aware methods refine code generation by integrating abstract syntax trees and data flow graphs. StructCoder [Tipirneni et al., 2024] introduces a structure-aware self-attention mechanism, and CoTexT [Phan et al., 2021] applies multi-task learning to enhance text-to-code understanding. Planning-based techniques decompose complex tasks into stepwise solutions, as seen in Self-Planning Code Generation [Jiang et al., 2024], while reinforcement learningbased approaches such as CodeRL [Le et al., 2022] optimize model adaptation through reward-based fine-tuning. Execution-enhanced methods ensure generated code correctness by leveraging runtime validation. MBR-EXEC [Shi et al., 2022] employs minimum Bayesian risk decoding based on execution, whereas CODET [Chen et al., 2022] generates test cases to filter invalid code. Besides, real-world integration studies, such as In-IDE Code Generation [Xu et al., 2022], evaluate the practical utility.

While these methods advance code generation in terms of syntax, semantics, and execution fidelity, they remain constrained to text-based inputs, lacking the capability to synthesize code from sketch-based conceptualizations. Furthermore, existing controllable code generation approaches do not inherently support structured diagram generation, limiting their applicability in domains requiring logical and hierarchical visual representations. [Ghosh et al., 2018; Almazroi et al., 2021] have focused on extracting structured representations from textual descriptions, but these methods do not generalize to sketch-driven workflows.

3 Method

The system consists of three modules: the Sketch-to-Code Agent, the Editing Code Agent, and the Check Agent, each responsible for specific tasks. Given a sketch S and a userspecified instruction set Q, SketchAgent generates an initial code representation, refines it based on additional instructions, and verifies the final output before rendering the structured diagram. The overall workflow is illustrated in Figure 2.

3.1 Sketch-to-Code Agent

The Sketch-to-Code Agent maps a hand-drawn sketch S and an instruction set Q to an initial code representation Ck, capturing the structural semantics of the sketch. This process is formulated as:

where Fk represents the transformation function. The output Ck is modeled as a sequence of tokens, where each token corresponds to a diagram component or an attribute.

To ensure the generated code aligns with the expected structure, we define the objective as minimizing the negative log-likelihood of the sequence:

where T is the sequence length, C (t) k is the t-th token, and P(·) denotes its conditional probability given the preceding sequence. Optimizing this objective ensures Ck remains structurally valid and semantically aligned with S.

3.2 Editing Code Agent

The Editing Code Agent refines the initial code representation Ck based on an additional instruction set Q′ , generating an updated version Ce. This process is formalized as:

where Fe represents the refinement function. The objective is to minimize the discrepancy between Ce and the expected output, ensuring modifications align with the intended diagram structure. To achieve this, we minimize the negative log-likelihood of the sequence:

where T is the sequence length, C (t) e is the t-th token, and P(·) denotes its conditional probability given the preceding sequence and input conditions. Optimizing this objective ensures Ce effectively integrates the modifications specified in Q′ while preserving the structural integrity of Ck.

3.3 Check Agent

The Check Agent verifies and refines the generated code Ce to produce the final executable representation Cf . This process is formalized as:

where Ff denotes the verification and debugging function. The goal is to ensure Cf is syntactically correct, executable, and aligned with the original sketch S and instructions Q.

To achieve this, the Check Agent begins by performing syntax validation and debugging to ensure that the generated code representation Ce is syntactically correct and compilable. If Ce fails to compile, it is sent back for regeneration by either the Sketch-to-Code Agent or the Editing Code Agent. Once compilation succeeds, we use GPT-4o to compare the compiled diagram D with the original sketch S and the user instructions Q. If the generated diagram aligns with the expected structure, the process is finalized; otherwise, the Check Agent triggers a fallback mechanism, prompting the responsible agent to regenerate Ce. This iterative validation process guarantees that only structurally coherent and semantically correct diagrams are finalized.

4 Sketch2DiagramBenchmark

The Sketch2Diagram Benchmark introduces a comprehensive dataset and evaluation framework for sketch-to-diagram tasks. It features eight diverse diagram categories, standardized with rigorous quality control and token-level statistics. Clear metrics evaluate sketch-to-code and code editing tasks, ensuring thorough assessment. The data collection and processing pipeline for the Sketch2Diagram Benchmark is meticulously designed to ensure a comprehensive and highquality dataset. This process is divided into three key stages: Data Collection, Data Processing, and Human Inspection, as illustrated in Figure 3.

Data Collection. The first stage involves gathering opensource .tex files of logical diagrams from multiple repositories, including datikz-v2, GitHub, and Overleaf. These sources provide a diverse range of diagrams across various domains, ensuring the dataset's broad applicability. The collected .tex files are then compiled into diagram images using standard LaTeX compilers. This step guarantees that the diagrams accurately reflect their logical structures as intended by their original creators.

Data Processing. We standardize the compiled diagrams to ensure uniformity and facilitate sketch-to-diagram tasks. Images are cropped to remove blank spaces and resized to 800×600 pixels. Colors and intricate details are stripped from diagrams to produce simplified sketch representations. GPT-4O generates corresponding sketch codes, ensuring consistency with the original diagrams. For model evaluation, we create two query types: (1) User Queries, which pair sketch images with codes to add supplementary details; and (2) Editing Queries, which capture differences between sketch and original diagram codes to guide refinement.

Human Inspection. The final stage involves a rigorous manual inspection process to ensure data quality. Human annotators adhere to three strict guidelines: First, images must be clear, free of blurriness or distortion, to ensure accurate interpretation. Second, images must be complete, with no missing sections or truncations. Third, queries are reviewed to exclude code snippets and focus on descriptive elements relevant to the task. These measures ensure the dataset's reliability and suitability for model evaluation.

4.1 Data Analysis

Diversity and Imbalance Challenges. Figures 5 and 4 illustrate the dataset's diversity and distribution. Figure 5 showcases examples from eight diagram categories, including undirected graphs, model architecture diagrams, and flowcharts, highlighting their real-world relevance. Figure 4 reveals an imbalance, with model architecture diagrams (52.34%) and directed graphs (42.5%) dominating, while categories like flowcharts and mind maps are underrepresented.

Token Length Statistics. Table 1 summarizes token length statistics for the Sketch2Diagram dataset, categorized by sketch-to-code (S2C) and code-editing (C2C) tasks. The dataset contains a total of 4824 training samples and 1206 test samples. Query lengths range from a minimum of 28 tokens for S2C tasks to a maximum of 170,894 tokens for C2C tasks. Answer lengths exhibit similar variability, with maximum values reaching 170,371 tokens. These diverse token lengths reflect the dataset's ability to evaluate models across tasks with varying complexities and input-output structures.

4.2 EvaluationMetrics

We employ a comprehensive set of evaluation metrics. For sketch-to-code, Pass@1 measures functional correctness, while ROUGE-L, BLEU, CodeBLEU (C-BLEU), chrF, Edit Distance (ED), BLEURT, and RUBY evaluate textual and semantic similarity. For code editing, we extend these metrics to include diagram fidelity measures such as FID, KID, CLIPFID (C-FID), Inception Score (IS), LPIPS, and SSIM.

5 Experiment

Setup The Sketch-to-Code Agent is based on Qwen2-VL7B [Wang et al., 2024], while the Editing Code Agent utilizes Qwen2.5-Coder-7B [Hui et al., 2024]. Both agents were finetuned over four epochs on a 4×80GB A100 GPU setup. The input token length for both agents is set to 4096 tokens.

Model For sketch generation, the Sketch-to-Code Agent is compared against several state-of-the-art models, including Yi-VL [Team, 2025c], Qwen2-VL [Wang et al., 2024], Internlm-Xcomposer2.5 [Zhang et al., 2023a], Llama-3.2Vision [AI, 2025b], Phi-3.5-Vision [Abdin et al., 2024], Llava-v1.6 [Liu and Others, 2025], Cogvlm2-Llama3 [Hong et al., 2024], and DeepSeek-VL [Team, 2025f], with closesource models such as GPT-4o [Achiam et al., 2023], GLM-4-plus [Team, 2025g], and Gemini-1.5-Pro [Team et al., 2024] also implemented. For the sketch editing task, the Editing Code Agent is assessed alongside specialized code models, including Qwen2.5-Coder [Hui et al., 2024], DeepSeek-Coder-Instruct [Guo et al., 2024], CodeLlama [Roziere et al., 2023], WizardCoder [Luo et al., 2023], CodeGeeX4-All [Team, 2025d], Starcoder2 [Lozhkov et al., 2024], Yi-Coder [Team, 2025b], Llama 3.1 [AI, 2025a], Baichuan2 [Yang et al., 2023], Internlm2.5 [Cai et al., 2024], Yi-1.5 [Team, 2025a], and Qwen2 [Team, 2024], and close-source models like GPT-4o [Achiam et al., 2023], DeepSeekV2.5 [Team, 2025e], GLM-4-Plus [Team, 2025g], and Gemini-1.5-Pro [Team et al., 2024].

5.1 Sketch generation

Main Results SketchAgent leverages compiler principles and integrates GPT-4o as a feedback mechanism to achieve state-of-the-art performance in translating logical structure diagrams into executable code. As demonstrated in Table 2, SketchAgent significantly outperforms both open-source and close-source models across key code generation metrics. Notably, it achieves a Pass@1 score of 82.34, substantially surpassing powerful large language models such as GPT-4o (51.12), Gemini-1.5-pro (61.94), and GLM-4V-Plus (66.09). In addition, SketchAgent excels in other critical metrics, including CodeBLEU (86.02), BLEU (29.61), and BLEURT (46.34), indicating its strong ability to generate syntactically and semantically accurate code. These evaluation results underscore SketchAgent's superior capability in producing high-quality, executable code from sketch representations.

Ablation Study We conduct an ablation study as detailed in Table 2. The results reveal the critical role played by both the compiler module and GPT-4o feedback mechanism. Removing the compiler results in a 1.49-point drop in Pass@1, indicating the compiler's significant contribution to precise code generation. Removing the GPT-4o feedback mechanism leads to a 1.16-point decline in Pass@1, demonstrating GPT-4o's role in validating outputs. More importantly, when both the compiler and GPT-4o are removed, the Pass@1 score experiences a substantial decrease of 3.82 points, falling to 78.52, highlighting their effect on SketchAgent.

Human Evaluation We employed three professional evaluators to assess the outputs. A score of 1 indicated the lowest quality, while 5 represented the highest quality. As shown in Figure 6, SketchAgent consistently achieved the highest quality based on objective assessment metrics. Furthermore, SketchAgent surpassed the accuracy of other models. Interestingly, the results revealed that SketchAgent performed better on the editing task compared to the generation task.

5.2 Sketch Editing

Main Results SketchAgent's Editing Code Agent demonstrates high performance across several key metrics, as shown in Table 3. It achieves the highest Pass@1 score of 93.12, surpassing other models such as Qwen2.5-Coder (62.19) and DeepSeek Coder (81.92). Moreover, SketchAgent excels in CodeBLEU and BLEU, with scores of 98.63 and 87.38, respectively, surpassing WizardCoder (96.78 and 76.39) and Code Llama (96.57 and 77.20). In terms of visual fidelity metrics, SketchAgent performs well with a FID of 130.1494, outperforming models such as DeepSeek Coder (222.42) and Qwen2.5-Coder (246.79), demonstrating its strong visual coherence and competitiveness.

Ablation Study The ablation study in Table 3 evaluates the impact of feedback module and compilation module on SketchAgent's performance. The full model achieves a Pass@1 score of 93.12, slightly lower than the 94.69 of the model without feedback, suggesting that the removal of feedback marginally improves performance. However, both models outperform the one without compilation, which scores 91.21, highlighting the importance of the compilation module. For ROUGE-L, the full model leads with 97.68, followed by the model without feedback at 97.65, while the model without compilation drops to 96.26, emphasizing the contribution of the compilation module. In FID, the full model outperforms both ablated versions with a score of 130.15, demonstrating the importance of both components in maintaining visual coherence. The model without both feedback and compilation shows higher KID and CLIP-FID scores but sacrifices performance in other key metrics, indicating the necessity of both components for optimal performance.

5.3 Error Analysis

As shown in Figure 7, SketchAgent encounters errors in three key areas: misaligned structures, misidentified elements, and misconnected relationships. Misaligned structures occur when the model fails to accurately capture the underlying structure of the objects in the sketch. This leads to incomplete or overly simplified outputs.

Misidentified elements refer to errors in recognizing individual components of a sketch. For instance, the model may distort characters, fail to reproduce their correct proportions, or misrepresent attributes such as font style or orientation. These errors stem from the variability and ambiguity in the sketches, where subtle variations in shape can confuse recognition. Misconnected relationships arise when the system misinterprets the spatial or relational connections between elements. For example, arrows may connect incorrect components, omit intermediate nodes, or bypass key elements, leading to diagrams with logical inconsistencies. Such errors undermine the semantic integrity of the generated diagram.

6 Conclusion

In this work, we present SketchAgent, a modular and endto-end system for transforming hand-drawn sketches into structured diagrams. SketchAgent demonstrates its ability to produce accurate and semantically coherent diagrams with minimal human intervention. Moreover, we introduce the Sketch2Diagram Benchmark, a comprehensive dataset featuring over 6,000 high-quality examples spanning eight categories. Extensive experiments demonstrate that SketchAgent outperforms state-of-the-art models across key metrics, achieving superior accuracy and visual coherence.

Acknowledgements

This work was supported by National Science and Technology Major Project (No. 2022ZD0115101), National Natural Science Foundation of China Project (No. 624B2115, No. U21A20427), Project (No. WU2022A009) from the Center of Synthetic Biology and Integrated Bioengineering of Westlake University.

Contribution Statement

The first four authors are equal contribution.