The Graph’s Apprentice: Teaching an LLM Low-Level Knowledge for Circuit Quality Estimation

1 Introduction

Rapid technological advancements in computing power has taken an increasingly important role in the past decades in driving scientific research in biology , chemistry , physics and especially artificial intelligence, where it has been estimated that at least half of all performance gains in the past ten years have stemmed from hardware improvements alone [Dorner, 2021; Karpathy, 2022; Erdil and Besiroglu, 2022]. This everrising demand for compute power means that efficient and effective electronic chip design has become increasingly critical.

Modern electronic chip design is a complex, multi-stage endeavor that begins with a chip architect specifying the digital circuit's functionality in a Hardware Description Language (HDL), such as Verilog [Thomas and Moorby, 2008] or VHDL [Coelho, 2012]. This HDL code is then subjected to a series of transformations and optimizations, ultimately yielding a physical circuit design that can be manufactured [LaMeres, 2023]. The quality of the resulting circuits are usually measured using physical characteristics only available in the later stages, such as circuit area or delay [Brayton and Mishchenko, 2010]. However, the computational cost of logic synthesis makes the iterative improvement according to the resulting circuit quality metrics prohibitively expensive. Optimizations are best made early on in this pipeline, ideally at the RTL level, so as to leave maximal flexibility in circuit design. Thus, efficient feedback methods that can estimate the quality of results of the HDL, can improve the resulting circuit and reduce the overall chip design time.

This discrepancy has led to interest in using artificial intelligence methods in the circuit design process [Huang et al., 2021]. In this literature, machine learning models are trained to provide feedback on HDL code without running the actual logic synthesis process. This is done using supervised learning on a training set of circuits for which logic synthesis has been run and from which quality-of-result (QoR) metrics such as circuit area and delay have been computed. Although this approach seems straightforward, finding an representation of the RTL code appropriate for machine learning models has proven a challenge. The few works that have approached this topic did so by extracting graphical information about the code and using hand-designed statistics of those graphs as features [Zhou et al., 2019; Sengupta et al., 2022; Fang et al., 2023]. Despite encouraging results, the performance of these methods has ultimately been limited by the relatively shallow understanding of the semantics of the code that these statistics can provide.

Recently, Large Language Models fine-tuned on code, such as Code-T5 [Wang et al., 2021], Codex [Chen et al., 2021], CodeGen [Nijkamp et al., 2023], CodeLlama [Roziere et al., 2023] and DeepSeek-Coder [Guo et al., 2024], have proven to be remarkably successful on a wide range of tasks [Zheng et al., 2023], most notably as code assistants such as Github Copilot 1 . This raises the question as to whether their internal representations could be used as inputs to machine learning to predict circuit quality estimates.

In this work, we propose to feed Verilog code to the stateof-the-art Large Language Models, and train an inexpensive decoder neural network that uses the LLM's hidden states as features to predict area and delay. In addition, and critically, we regularize this decoder to encourage its embeddings to resemble those of a graph neural network model trained on Look-Up Table (LUT) graph, an intermediate representation used during the logic synthesis process. The resulting decoder is shown to strongly outperform state-of-the-art baselines in RTL-level circuit quality estimation, while keeping training and inference costs practical.

Our work makes the following main contributions:

1. We develop the first truly end-to-end machine learning model in the literature, named VeriDistill, which can take raw Verilog code, without any preprocessing, and produce accurate estimates of circuit area/delay metrics.
2. Moreover, we apply during training a novel knowledge distillation method which allows to transfer low-level insights about the circuit, in the form of LUT graphs, back into the machine learning predictor model.
3. We demonstrate through experiments that the combination of those two elements outperforms previous state-of-the-art baselines in a large-scale Verilog dataset and enhances the model's ability to transfer to out-ofdistribution data.
4. Finally, we also demonstrate that both using LLM representations and the knowledge distillation are essential, in that removing any one of these components brings the performance back below the previous baselines.

The remainder of this paper is structured as follows. Section 2 provides an overview of the relevant literature and background information. In Section 3, we present a detailed description of our proposed methodology, including its key components and underlying assumptions. The efficacy of our approach is then demonstrated through a series of experiments, which are reported in Section 4. Finally, Section 5 summarizes our main findings, discusses their implications, and outlines potential avenues for future research.

2 RelatedWork

2.1 Quality-of-Result Prediction fromHDL Code

Closest to ours is the work of [Sengupta et al., 2022]. Their approach consists in computing the Abstract Syntax Tree (AST) induced by Verilog code, and extracting from this free vector- and graph-based features. They then train several machine learning models to predict from these features the total negative slack and dynamic power of the circuit. Among all the models evaluated, the XGBoost Regressor performs best and achieves 95% R2-score. The analysis was however limited to different runs of a single circuit and it is not clear how the performance would generalize to different circuits. Since the Abstract Syntax Tree is essentially the raw Verilog code with extra syntactic information, which can be obtained at little cost at inference time by a grammar parser, we include it (along with variants) as baselines in our experimental section.

1 https://github.com/features/copilot

Further related is the work of [Fang et al., 2023] and [Fang et al., 2024b]. They propose to process Verilog code into a new representation called Simple Operator Graph (SOG), and test several machine learning models (Transformers, Random Forests, Graph Neural Networks and XGBoost regressors) to predict path delay, module-level power and combinatorial area. Although achieving promising results, computing the SOG requires expensive conversion of linguistic data into bit-level operators using logic synthesis tool Yosys [Wolf et al., 2013], and a Verilog-to-graph parser, which is outside the scope of this work.

Finally, some works take a step further and try to assist circuit design by annotating which parts of HDL is most critical to achieve quality-of-result metrics. For example, [Sengupta et al., 2023] attempts to identify timing critical components based on path delay prediction. The AST of each Verilog design is extracted and converted into a graph, with nodes representing IO ports, registers or behavior logic. Behavioral paths are extracted from the graph and used for path-level feature generation. Delay labels of timing paths are generated using commercial synthesis tools, and are assigned to corresponding behavior paths with the same start and end points. By training an XGBoost model on the resulting features, the authors achieve an average classification accuracy of 91%. Also similar is RTL-Timer [Fang et al., 2024a], which ensembles four bit-level circuit representations to predict the post-logic synthesis endpoint arrival time. Such predictions can then be mapped to registers in HDL code to identify critical code paths. Just as in the work of [Fang et al., 2023], however, these representations are bit-level rather than word-level and require pre-processing by logic synthesis tools like Yosys.

2.2 LLMs for Verilog

Large language models (LLMs) such as GPT [Ouyang et al., 2022] and Llama [Touvron et al., 2023] have achieved exceptional success in various natural language tasks and have expanded their success to programming languages as well. Although excellent on generalist programming languages like Python or C++, these models have been trained on the relatively small amount of HDL code that is publicly available on the internet, and therefore have performed poorly on Verilog benchmarks like VerilogEval [Liu et al., 2023b] and RTLLM [Lu et al., 2024]. This has motivated further work to build LLMs with a higher-degree of knowledge of hardware description languages. Both CodeGen-Verilog [Thakur et al., 2023] and VeriGen [Thakur et al., 2024] used a combination of customized Verilog datasets from code repository website GitHub 2 and various textbooks to fine-tune code LLMs. Finally, RTLCoder [Liu et al., 2023c] used the GPT 3.5 language model [Brown et al., 2020] to generate further Verilog data, in a form of data augmentation, while CodeV [Zhao et al., 2024] used the same model to generate natural language description of real world Verilog code through multilevel summarization.

2 www.github.com

Besides Verilog code generation from natural language description, LLMs were also explored for other EDA-related tasks. RTLFixer [Tsai et al., 2023] employed RetrievalAugmented Generation (RAG) and ReAct prompting techniques to interactively debug syntax errors in Verilog code, and achieved remarkable improvement in success rates in the VerilogEval benchmark. ChipNemo [Liu et al., 2023a] explored the application of LLMs in chip design process and adopted several domain adaptation techniques to train an LLM for various applications including assistant chatbots, EDA script generation, and bug summarization and analysis. Finally, ChatEDA [Wu et al., 2024] used code LLMs as an agent to autonomously complete the entire chip design flow from HDL code to the Graphic Data System Version II (GDSII) by managing task planning, script generation and task execution. We refer the reader to the extensive survey of [Zhong et al., 2023] for more details on the application of LLMs in electronic design automation and future research directions in this field.

2.3 Alignment of LLMand GNN Embeddings

The multimodal alignment regularizer we propose during training also relates to the broader literature on tuning large language models to align with a pre-trained graph neural network, to incorporate its capabilities.

The work closest to ours is that of [Mavromatis et al., 2023], who train a language model to perform a node classification task while adding a regularizer that encourages the predictive distributions to match a pre-trained graph neural network model. The language model makes predictions by passing the graph as input, and extracting the representation corresponding to a final [CLS] classification token. Also similar is [Zou et al., 2023], which jointly trains a language model and a graph neural network on a common "context graph prediction" task which encourage alignment of their representations. They then discard the graph neural network and only keep the language model, so that topological characteristics best captured by graph convolutions can be said to have been incorporated in the language model.

More generally, there is a large literature on integrating pretrained graph neural networks with language models by training an adaptive module [Liu et al., 2024; Liu et al., 2023d; Chai et al., 2023; Tang et al., 2024; Cao et al., 2023], allowing the language model to receive inputs from the graph neural network. Alternatively, multiple works have interlaced graph neural network layers and language model layers [Yasunaga et al., 2021; Zhang et al., 2022; Jin et al., 2023]. In either case, some kind of training is necessary to allow for interactions between the graph neural network and the language model, although the result is not distillation of the graph neural network's perspective into the language model per se.

3 Methodology

We now present our VeriDistill approach in detail. As described in the introduction, turning a high-level description of a circuit in a Hardware Description Language like Verilog into a physical description ready for manufacturing is a computationally expensive process involving several steps, each with an associated intermediate representation describing progressively lower-level elements of the circuit. Our goal is to predict low-level quality-of-result metrics, like area and delay, from the high-level representation, namely the HDL code to allow for fast iterative improvement of the RTL design. Figure 1 provides and overview of logic synthesis as well as the training and inference pipelines.

3.1 Model

Our model takes as input Verilog code, which is fed to a Large Language Model (LLM). This LLM has been specifically fine-tuned on Verilog code generation. The code is first split into a sequence of tokens, which are then fed in parallel in the LLM. As an output, the LLM produces a sequence of high-dimensional "hidden state" vectors, one for each token that is inputted to the LLM. We average these hidden states, producing a single vector. This vector is then fed to a feedforward neural network, composed of several linear layers with nonlinear activations, which finally outputs the QoR estimate.

3.2 Training

We produce a training set of circuits with Verilog code for which the expensive logic synthesis process has been performed, so we know their QoR metric (such as area or delay). In addition, as an intermediate product of the logic synthesis process, an LUT graph is produced immediately following the logic optimization phase, which we save. This yields a collection of training triples D = {(XVerilog, XLUT, yQoR)}.

Supervised learning: Given such a dataset, we treat our problem by supervised machine learning. The LLM, which has been pretrained on Verilog code, is kept frozen, so that only the FNN gets updated. In a training step, the Verilog code XVerilog is fed to the VeriDistill model to produce a prediction ˆ yQoR. This prediction is compared in mean-squared error loss with the true QoR metric yQoR as a supervised learning loss

Low-level knowledge distillation: In practice, training only with the supervised learning loss leads to limited performance. One potential explanation is that there is too much of a gap between a high-level circuit description like Verilog and the low-level metrics we purport to predict. Intuitively, to perform high-quality predictions, we would want the model to possess some degree of understanding of lower-level circuit design while still only taking Verilog code as input.

We propose the following approach to address this problem. Prior to training, we pretrain a Graph Neural Network (GNN) to predict the same QoR metric as VeriDistill, but from the Look-Up-Table (LUT) graph XLUT of the circuit obtained after optimization using Yosys [Wolf et al., 2013]. This graph, which can be seen as an alternative to the more popular And-Inverter Graph (AIG) format, is particularly suitable for GNN training as it is compact with rich node information. Moreover, as a circuit representation, it sits intermediate between a high-level description of the circuit encoded in the Verilog code, and a physical circuit description. Prediction from LUT graphs is thus easier than prediction from Verilog code, but not completely trivial either.

The GNN architecture we adopt is composed of a sequence of graph convolutions, followed by joint mean and max pooling, and a sequence of linear layers. We pretrain it using the supervised learning loss (1) until good predictive performance is achieved. Then, during the VeriDistill training, we keep the GNN weights frozen and we propose to encourage the last-layer activations of the VeriDistill model z(-1) VeriDistill to resemble those of the GNN model z(-1) GNN , despite these models operating on different inputs. We perform this simply by adding a mean-square error loss

in the total loss. As the weights of the GNN are pretrained and kept frozen while the VeriDistill model is being trained, this is effectively a form of knowledge distillation from the GNN to the VeriDistill model.

Total loss: We balance the importance given to the knowledge distillation compared to the supervised learning objective using a hyperparameter factor α, yielding the final loss

A diagram describing the VeriDistill training process is provided as Figure 2.

4 Experiments

This section is organized as follows: We begin by presenting the implementation details of our experimental setup in Section 4.1, including hardware, model, and training hyperparameters. Next, we describe the dataset used and the data preprocessing steps for training and evaluation in Section 4.2. We then introduce the baseline methods and their implementation details in Section 4.3. Finally, we present the results on the main datasets and a study on unseen out-of-distribution circuits in Sections 4.4 and 4.5.

4.1 Experimental Setup

We use the following implementation of the model. We employ DeepSeek-Coder-V2-Lite-Base [DeepSeek-AI et al., 2024] and CodeV-7B (based on CodeLlama) [Zhao et al., 2024] as the Verilog LLM, and three layers with ReLU activations in the feedforward neural network. The model takes as input strings, which are broken into a sequence of tokens in the LLM's vocabulary. The language model processes these inputs into a sequence of the same length, made up of 512-dimensional vectors. After mean pooling, the resulting vector is passed to the feedforward neural network, which uses 512-dimensional activations, before making the final prediction. In particular, this architecture means that the last-layer activations z(-1) VeriDistill are 512-dimensional. Results for other variants of CodeV-7B [Zhao et al., 2024] as the Verilog LLM are included in the Appendix.

The auxiliary GNN teacher model takes a LUT graph with 16-dimensional node attributes, and passes it through three 64-dimensional graph convolutional layers interleaved

VeriDistill Model

with batch normalization layers. After concatenation of the mean and max pooling outputs, the 128-dimensional vector is passed through three 512-dimensional linear layers with ReLU activations before the final prediction. Thus, in particular, the last-layer activations z(-1) GNN are 512-dimensional, matching with those of the VeriDistill model.

We implement VeriDistill and the baselines using the PyTorch and PyG libraries. Models which do not use our knowledge distillation procedure are trained using the ReduceLROnPlateau scheduler with initial learning rate 1e-3, patience set to 30 epochs and factor set to 0.5. In contrast, models involving our knowledge distillation procedure are trained using the CosineAnnealingLR [Loshchilov and Hutter, 2017] scheduler, with an initial learning rate of 1e-3 and number of iterations set to 50. We start the training process with α = 0.5, and increase α to 0.75 and 1 at epochs 150 and 250. The idea is put less emphasis on knowledge distillation at every warm re-start. We find that this approach results in marginal gain compared to other optimization methods.

All models are trained until full convergence. Since the LLM is kept frozen during training, it was possible to save training time by extracting the forward pass through the LLM only once and saving it. We performed this phase on a machine with 8 Nvidia V100 GPUs with 32GB of memory and 32 Intel(R) Xeon(R) Gold 6140 CPUs. Once the hidden states are saved, we then trained each model following the procedure detailed in the paper on the same machine using a single V100 GPU with 1024 minibatch sizes. The training times for each model are summarized in the Appendix.

4.2 Datasets

We train and evaluate on two separate datasets. The first dataset is used for training, validation, and testing of all the methods, while OpenABCD contains out-of-distribution circuits aiming to challenge VeriDistill and determine its ability to generalize.

Customized Dataset To train and evaluate our proposed method, we collect 18.4k Verilog examples provided by [Pei et al., 2024] and 5.8k from [Thakur et al., 2022]. These Verilog examples are obtained from open-source GitHub repositories and textbooks and have been verified for syntax correctness. We use an open-sourced EDA platform OpenROAD [Ajayi et al., 2019] with 7nm technology PDK provided to conduct logic synthesis and record post-synthesis labels of area and delay. We convert the AIG graphs obtained after logic optimization into LUT graphs and save them for training the auxiliary GNN model.

Note that a substantial fraction of the code snippets end up being functionally incorrect and failing some stage of the logic synthesis pipeline. Since we require functionally correct examples for their QoR metric to be well-defined, we removed such examples during the preprocessing. In addition, although not strictly a problem for our method, one of the competing baselines requires extracting the Abstract Syntax Tree (AST) of the Verilog, which is obtained by running a parser on the code. The parser was unable to produce AST representations for a small fraction of the instances (FRACTION%), which we removed from consideration. The resulting dataset, after filtering bad examples, ended up having 16k examples, which we split into training, validation, and test sets with a ratio of 0.75/0.1/0.15, respectively. Details about the dataset and label distributions can be found in the Appendix.

We note that OpenROAD provides two optimization recipes for the logic synthesis process: "ABC AREA=1" for area optimization and "ABC SPEED=1" for timing optimization. The results reported under Section 4.4 are produced under the speed optimization. We report the results under area optimization in the Appendix. We find that our approach works as well under different recipe optimization settings.

OpenABCD Additionally, we consider data provided by [Chowdhury et al., 2021] to evaluate the transferability of our method to unseen circuits. The OpenABCD dataset consists of functionally diverse designs such as bus communication protocols, computing processors, digital signal processing cores, cryptographic accelerators and system controllers.

4.3 Baselines

While many prior works have attempted to predict postsynthesis circuit quality at the RTL-stage, none of them perform prediction directly from source Verilog files. Several works rely on lower-level circuit representation that requires extra processing using logic synthesis tools [Zhou et al., 2019; Fang et al., 2023]. Utilizing low-level circuit representations as input is advantageous for the circuit quality prediction. However, in some cases, obtaining the low-level description can be prohibitively expensive. In addition, obtaining the low-level description relies on external processing tools which are prone to errors. As such, we compare our method to approaches which take the raw Verilog or the AST representation of the circuit.

We adopt the method proposed by [Sengupta et al., 2022] as our baseline. It relies on AST representations that can be easily converted from Verilog source files. We implement the method based on description in [Sengupta et al., 2022]. Verilator [Snyder, 2004] is used to convert each source Verilog into its respective AST representation, which can be represented as a graph. The nodes in the graph represent one of the following five semantic categories from the source Verilog (root, variable, operation, constant, edge), while edges are created between nodes with connections. We implement three variants of the AST-based method:

AST-XGBoost We compute the following features: (i) the total number of input bits, (ii) the total number of output bits, (iii) the longest path in the AST, (iv) the frequency of each node type in the graph and (v) the frequency of each logic type in the graph. The features are concatenated to form a feature vector with 108 features 3 . We perform a thorough hyper-parameter selection using grid search and employ early stopping to prevent over-fitting.

AST-GNN w/o KD The AST-GNN model takes in the following features per node: (i) the total number of input bits,

3 108 = 1 + 1 + 1 + 5 + 100 features coming from feature categories (i)...., (v)

(ii) the total number of output bits, (iii) the node semantic type and (iv) the node operation type. Each feature is represented via a one-hot vector and is projected to a 4dimensional space via a linear layer. The final node features consist of a (4 × 4) = 16-dimensional vector. We cap the number of input/output bits to 200, since 99.9 percent of the nodes in the dataset have less than 200 input/outputs. The AST-GNN model utilizes the same hyperparameters and architecture as the auxiliary GNN model used for the knowledge distillation objective in VeriDistill.

AST-GNN w/ KD We propose a third baseline, where the AST-GNN model is guided by the LUT GNN model. The baseline utilizes the same student-teacher knowledge distillation as our method. We introduce this baseline to demonstrate the effectiveness of utilizing an LLM in the student network.

4.4 Main results

We first summarize the results of our main experiment, where we train and test the model on the large Customized Dataset (see Section 4.2). Table 1 outlines the performance of different models on the test set. As can be seen, our proposed method, utilizing both the LLM as an encoder and knowledge distillation, outperforms other baselines across all the metrics, especially with area prediction. Interestingly, simply using a decoder on the LLM representation performs worse than the previous state-of-the-art, while knowledge distillation on the AST-GNN model has almost no effect. Only when both are used together is there substantial impact on performance, which suggests our knowledge distillation procedure is crucial in fully exploiting the richness of the LLM representations.

The CodeV + Decoder model greatly outperforms the DeepSeek-V2-Lite + Decoder model. We hypothesize this performance gap is due to the fact that CodeV representations are more aligned with Verilog semantics. While DeepSeekV2-Lite is pre-trained on a variety of programming languages, the CodeV is specially fine-tuned on Verilog data after being pre-trained on various programming languages.

However, the performance gap between the two VeriDistill models is much smaller. This results hints that using knowledge distillation can, to a high degree, mitigate the lack of additional fine-tuning of the base LLM on Verilog. The result hints that VeriDistill can achieve similar results when applied on top of various code LLMs without the need for further fine-tuning the base LLM on Verilog data.

We gain further insight on the benefits of our approach by analyzing scatter plots of the predictions against the targets. As can be seen in Figure 3, the baseline models perform well primarily on circuits with small delay and area but struggle with larger circuits, likely due to their lower representation in the training set. In contrast, our model achieves consistently strong performance across circuits of all sizes. This contrast is particularly pronounced when comparing against the same model without knowledge distillation (LLM+Decoder), which indicates that our knowledge distillation procedure is crucial in allowing our model to perform well across the whole range of circuit sizes. The impact of knowledge distillation on the AST model is relatively minimal compared to its effect on the LLM-based model. This can be attributed to the enhanced alignment between the teacher representations and the LLM when used as the encoder. A visualization of the t-SNE projection of the final hidden space representations is provided in the Appendix to verify the above claim.

4.5 Additional Out-of-Distribution Results

Finally, we evaluate how our knowledge-distillation procedure can impact the ability of the trained model to generalize to new out-of-distribution circuits. For this, we take our model, trained with and without knowledge distillation on our Customized Dataset, and apply it to instances in the OpenABCD benchmark (see Section 4.2). We outline the OpenABCD circuit statistics in the Table 2. Due to the scarcity of large circuit Verilog data, the circuits from the OpenABCD benchmark are larger than the majority of the circuits present in our dataset (that is, the Verilog files contains more lines and the circuits have generally larger area and delay). As can be seen in Table 2, our knowledge distillation procedure systematically improves the LLM-based model's ability to transfer prediction performance on out-of-distribution instances, which differ significantly from those seen during training. A comparison between the performance of VeriDistill with the AST based approaches on the OpenABCD benchmark can be found in the Appendix. where VeriDistill outperforms the baselines on the well majority of the circuits.

5 Conclusion

In this work, we propose a novel procedure to predict qualityof-result electronic circuit metrics from Verilog code, by training a small neural network model on Verilog LLM representations with a knowledge distillation regularizer which align its internal activations with those of a low-level GNN model. We show that this new model outperforms previous approaches in predicting the post synthesis QoR labels of a circuit. Beyond the potential of our method for future practical applications, our results underscore the value of the information encoded in the LLM's representations for predicting circuit quality. Additionally, they highlight the crucial role of our knowledge distillation procedure in enabling downstream models to effectively leverage this information.