ADELT: Transpilation between Deep Learning Frameworks

1 Introduction

The rapid development of deep learning (DL) has led to an equally fast emergence of new software frameworks for training neural networks. Unfortunately, maintaining a deep learning framework and keeping it up-to-date is not an easy task. Many deep learning frameworks are deprecated or lose popularity every year, and porting deep learning code from a legacy framework to a new one is a tedious and error-prone task. A source-to-source transpiler between DL frameworks would greatly help practitioners overcome this difficulty.

Two promising solutions to source-to-source transpilation between deep learning frameworks are unsupervised neural machine translation (NMT) [Artetxe et al., 2018] and large language models (LLMs). NMT treats deep learning code as a sentence for training sequence-to-sequence [Sutskever et al., 2014] models, but its applicability is limited due to the scarcity of parallel corpora and its notable data hunger. On the other hand, LLMs like GPT-3 [Brown et al., 2020], pretrained on web crawl data, offer potential, performing translation tasks in a few-shot or zero-shot manner. Our early experiments with GPT-4 show its potential in few-shot transpilation of deep learning programs. However, such models struggle with API-specific details, inaccurately handling function names and parameter mappings.

∗The appendices can be found at https://arxiv.org/abs/2303.03593.

That said, most deep learning framework code is structured: each type of layers has its own constructor, and constructing a network involves calling each layer's constructor in a chaining manner. By leveraging the structures of programming languages, we can decouple the transpilation of skeletal codes from the mapping of API keywords. The transpilation of skeletal codes is the easier part, and LLMs already do a great job. We only need a separate algorithm to translate the API keywords, i.e., the function and parameter names to complete the transpilation.

In this paper, we present ADELT (Figure 1), a method motivated by this insight to transpile DL code. The canonicalized source code is decoupled into two parts: the code skeleton and the API keywords. ADELT transpiles the code skeleton using a pretrained LLM by few-shot prompting. Each API keyword occurrence is then embedded into a vector by PyBERT, a BERT pretrained on Python code. This vector is both the textual and the contextual representation of the API keyword. ADELT then leverages domain-adversarial training to learn a generator that maps the vector to an aligned embedding space. The alignment is enforced by a two-player game, where a discriminator is trained to distinguish between the embeddings from the source DL framework and those from the target DL framework. The API keyword embeddings are trained jointly with the generator as the output embedding matrix of a softmax classifier on the aligned embedding space. After generating a synthetic API keyword dictionary from the embeddings using a two-step greedy algorithm, ADELT then looks up each API keyword occurrence in the dictionary and puts them back into the transpiled code skeleton.

In summary, this paper makes the following contributions:

• We introduce ADELT, a robust solution for transpilation between deep learning frameworks without training on any labeled data. Outperforming large language models, ADELT excels across various transpilation pairs, achieving pass@1 rate of 73.0 and 70.0 for PyTorch-Keras and PyTorch-MXNet transpilations, respectively. These scores surpass those of the state-of-the-art LLM, GPT-4, by 16.2 and 15.0 points respectively.
• For training, we construct a PyTorch-Keras-MXNet corpus

of deep learning code from various Internet sources, containing 49,705 PyTorch modules, 11,443 Keras layers/models, and 4,785 MXNet layers/models. We then build an evaluation benchmark for PyTorch-Keras and PyTorch-MXNet transpilation. The benchmark evaluates both our API keyword mapping algorithm and the overall source-to-source transpilation.

2 Method

ADELT (Adversarial DEep Learning Transpiler) is an algorithm that transpiles code from a source deep learning framework into an equivalent one in a target framework, by transpiling the skeletal code using a pretrained large language model, and then looking up each keyword in a dictionary learned with unsupervised domain-adversarial training. ADELT applies the following steps to each piece of input code, which we illustrate using the example shown in Figure 1:

1. Extract API calls from the source code. Such API calls can be automatically extracted with the Python's built-in ast library. We then convert each API call into its canonical form, where each layer/function has a unique name, and all of its arguments are converted to keyword arguments. Finally, we extract all API keywords from the canonicalized API call, where an API keyword is the name of a layer/function or the name of a keyword argument.
2. Transform the program into its code skeleton by replacing each API keyword occurrence with a distinct placeholder.
3. Transpile the code skeleton, where all API keywords are replaced by placeholders, into the target DL framework using a pretrained big LM (e.g., Codex).
4. Look up each API keyword in the API keyword dictionary, and replace each keyword with its translation. To generate the API keyword dictionary, we first learn the API embeddings using domain-adversarial training based on contextual embeddings extracted by PyBERT (a BERT pretrained on Python code and then fine-tuned on deep learning code). Next, we calculate the cosine similarity between the embedding vectors. Then we generate the API keyword dictionary using a hierarchical algorithm.
5. Put each API keyword back into the transpiled code skeleton to generate the final output.

We describe each of these steps next in detail.

2.1 Canonicalization &API Keyword Extraction

We first parse the source code into an abstract syntax tree (AST) with the Python ast module. Then, canonicalization and API call extraction are applied to the AST.

Canonicalization. We canonicalize each API call using the following steps during both domain-adversarial training (Section 2.3) and inference. Each step involves a recursive AST traversal.

1. Unify the different import aliases of each module into the most commonly used name in the training dataset. For example, torch.nn is converted to nn.
2. Unify different aliases of each layer/function in a DL library into the name in which it was defined. We detect and resolve each alias by looking at its __ name __ attribute, which stores the callable's original name in its definition. 1 For example, layers.MaxPool2D is converted to layers.MaxPooling2D.
3. Convert each positional argument of an API call into its equivalent keyword argument. Sort all keyword arguments according to the order defined in the function signature. This is done by linking the arguments of each API call to the parameters of its API signature using the bind method from Python's inspect module. 2

API keyword extraction. We define API keyword as the name of a layer/function or the name of a keyword argument. Once the input code is canonicalized, we locate each API keyword in the AST and then unparse the AST into the canonicalized source code.

2.2 Skeletal Code Transpilation

After canonicalizing the source program, ADELT then replaces all API keywords with a placeholder, turning the source program into its code skeleton. Each placeholder has textual form PLACEHOLDER _ i, where i = 1, 2, . . . . The code skeleton is then translated by Codex using few-shot prompting. The full prompt for is shown in Appendix A.4 [Gong et al., 2024].

1 https://docs.python.org/3/reference/datamodel.html

2 https://docs.python.org/3/library/inspect.html#inspect.Signatu re.bind

Algorithm 1 Pseudo-code for domain-adversarial training.

1 for (x _ 1, y_ 1), (x _ 2, y_ 2) in loader: 2 # N samples from X _ 1, X _ 2 respectively 3 # y_ 1, y_ 2: API keyword ids 4 5 h _ 1 = B(x _ 1).detach() # contextual embedding 6 h _ 2 = B(x _ 2).detach() # no gradient to PyBERT 7 z _ 1 = G(h _ 1) # generator hidden states 8 z _ 2 = G(h _ 2) # z _ 1, z _ 2: N x d 9 10 # dot product of z _ l and output embeddings 11 logits _ 1 = mm(z _ 1, E _ 1.view(d, m _ 1)) 12 logits _ 2 = mm(z _ 2, E _ 2.view(d, m _ 2)) 13 L _ CE _ 1 = CrossEntropyLoss(logits _ 1, y_ 1) 14 L _ CE _ 2 = CrossEntropyLoss(logits _ 2, y_ 2) 15 16 # discriminator predictions 17 pred _ 1 = D(z _ 1) 18 pred _ 2 = D(z _ 2) 19 labels = cat(zeros(N), ones(N)) 20 L _ D = CrossEntropyLoss(pred _ 1, labels) 21 L _ G = CrossEntropyLoss(pred _ 2, 1 - labels) 22 23 # joint update of G and E _ l 24 # to minimize L _ CE _ l 25 optimize(G + E _ 1 + E _ 2, L _ CE _ 1 + L _ CE _ 2) 26 optimize(D, L _ D) # train the discriminator 27 optimize(G, L _ G) # train the generator

• B: PyBERT used as the contextual embedder.
• G, D: the generator G and the discriminator D.

E _ l: a d by ml matrix, where the i-th column vector is the output embedding of API keyword w(l) i .

mm: matrix multiplication; cat: concatenation.

2.3 Domain-Adversarial Training

Once the code skeleton is transpiled, we then transpile API keywords. We train the aligned embeddings of API keywords in a domain-adversarial setting. In Section 2.4, the embeddings will be used to generate a dictionary that maps an API keyword of the source deep learning framework X (1) to an API keyword in the target DL framework X (2) .

Figure 2 illustrates the domain-adversarial approach of ADELT, and Algorithm 1 shows the pseudocode. A generator maps the contextual representations extracted by PyBERT into hidden states (line 5-8). The alignment of hidden states from different DL frameworks is enforced by the adversarial loss induced by the discriminator (line 17-21), so that output embeddings learned with these hidden states (line 11-14) are also aligned. Next, we describe each step in detail:

Each training example is a pair of API keyword occurrences with their context in the training corpus, denoted by (x(1) , x(2) ). Each keyword occurrence x(l) is tokenized and encoded as multiple byte pair encoding (BPE) [Sennrich et al., 2016] tokens. In our unsupervised setting, x(1) and x(2) are independent samples fromX (1) and X (2) in the training dataset, respectively, and they are not necessarily translations of each other.

PyBERT. PyBERT is our pretrained Transformer [Vaswani et al., 2017; Devlin et al., 2019] for Python code [Feng et al., 2020; Kanade et al., 2020; Roziere et al., 2021]. Given a sequence of BPE tokens that represent an API keyword with its context x(l) , PyBERT outputs a sequence of vectorsone vector in Rdb for each token, where db is the hidden dimension size of PyBERT. We average-pool all BPE tokens of the keyword and get a single db-dimensional vector as the contextual embedding PyBERT(x(l) ) of the API keyword. We denote the contextual embedding of x(1) , x(2) by h(1) , h(2) respectively.

Generator and discriminator. We define two multi-layer perceptrons, a generator and a discriminator. A generator G encodes the contextual embeddings h(1) , h(2) into hidden states z(1) , z(2) ∈ Rd , and a discriminator D is trained to discriminate between z(1) and z(2) . The generator is trained to prevent the discriminator from making accurate predictions, by making G(PyBERT(X (1) )) and G(PyBERT(X (2) )) as similar as possible. Our approach is inspired by domain-adversarial training [Ganin et al., 2016], where domain-agnostic representations of images or documents are learned for domain adaptation. In our case, a domain is represented by a DL framework.

Formally, we define the probability PrD(pred = l|z) that a hidden state z is from the DL framework l predicted by the discriminator. Note that z(1) = G(h(1) ) and z(2) = G(h(2) ). The discriminator loss and the generator loss are computed as the binary cross entropy against the true label and the reversed label, respectively, as shown in Equation (1).

Output embeddings. Our goal is to learn an embedding for each API keyword, but the contextual embedding of each keyword occurrence varies with its context. So we instead train a d-dimensional vector e(l) i for each API keyword w(l) i , such that e(l) i is similar to the generator hidden states z(l) j of this keyword's occurrences and dissimilar to the hidden states z(l) k of any other keyword's occurrences. e(l) i is considered the output embedding of the API keyword w(l) i . With similarity computed using dot product, our optimization objective is shown in Equation (2), equivalent to the cross-entropy loss of m(l) -way softmax-based classification.

Adversarial training. During each training iteration, the generator and discriminator are trained successively to minimize LG and LD respectively with mini-batch stochastic gradient descent. Minimizing the adversarial loss equals to minimizing the distance between two distributions of hidden states [Goodfellow et al., 2014]. Therefore, the API keywords from the different DL frameworks will be mapped to an aligned embedding space.

Also, we jointly update the generator and the output embeddings to minimize L(l) CE with mini-batch SGD. The joint optimization is crucial, as updating the generator to minimize L(l) CE ensures that each generator hidden state z(l) preserves enough information to recover its original API keyword. As a result, the output embeddings {e(1) i }m(1) i=1 and {e(2) j }m(2) j=1 are also aligned, as they are trained with vectors z(l) from the aligned embedding space.

We do not fine-tune PyBERT during domain-adversarial training, as fine-tuning PyBERT makes the generator disproportionally strong that results in training divergence.

2.4 Hierarchical API Dictionary Generation

ADELT calculates a scoring matrix using the aligned API keyword embeddings trained in Section 2.3. The entry in the ith row and the j-th column of the matrix is the cosine similarity between w(1) i and w(2) j , denoted by si,j. Given the scoring matrix, we need to generate an API keyword dictionary that maps each API keyword in one deep learning framework to an API keyword in another DL framework.

Greedy match. Greedy match has been used to generate a dictionary in word translation of natural languages [Conneau et al., 2018], where each source word is matched to the target word with the highest similarity score.

Structure of API keywords. Unlike words in NL, API keywords are structured: API keywords can be classified into two types based on their associated AST node: callables names (names of functions or classes), and parameter names (names of keyword arguments). In dictionary generation, we do not allow callable names to be translated to parameter names. We only allow parameter names to be translated to callable names in a special case when the weight passes a threshold. In this case, this parameter will be dropped and generate a new API call (the last case in Table 2). Another structural property is that the matching of parameters depends on the matching of callables.

Hierarchical API dictionary generation. The algorithm leverages the structure of API keywords to generate a dictionary: Step 1. Consider each callable and its parameters as a group and compute the group similarity between each pair of groups, by summing up similarity scores in the greedy matching of parameter names, plus the similarity between two callable names. Step 2. Match groups greedily based on group similarity scores calculated in step 1.

3 Experiments

We evaluate the effectiveness of ADELT on the task of transpilation between PyTorch, Keras, and MXNet and compare our method with baselines.

3.1 Skeletal Code Transpilation

We use Codex [Chen et al., 2021], a LLM trained on public GitHub code, to transpile code skeletons. As an autoregressive language model trained on massive web data, Codex can handle translation tasks via prompting with few-shot demonstrations. Our prompt design aligns with Codex's code translation setup, comprising a single input-output example and three instructions to keep placeholders unchanged. Appendix A.4 provides further details on this.

3.2 Training Setup

DL corpus. We consider 3 data sources GitHub, JuiCe, Kaggle to build our DL corpus:

• GitHub: The GitHub public dataset available on Google BigQuery. 3 We keep py and ipynb files that contain torch, keras, or mxnet in the main and master branch of the repository. (69GB of clean Python code before filtering, 2.5GB after filtering).
• JuiCe: A code generation dataset [Agashe et al., 2019] based on ipynb files from GitHub. JuiCe contains many files absent in the public dataset on Google BigQuery, since the latter is a selected subset of GitHub (10.7GB of clean Python code).
• Kaggle: All files in KGTorrent [Quaranta et al., 2021], a dataset of Jupyter Notebooks from Kaggle 4 (22.1GB of clean Python code).

We tokenize all Python source code and extract subclasses of torch.nn.Module, keras.layers.Layer, or keras.Model. Then, we canonicalize (section 2.1) the code of each class definition. We byte-pair encode [Sennrich et al., 2016], merge, and deduplicate codes from all sources. Finally, we collect all files into our DL Corpus containing 49,705 PyTorch modules, 11,443 Keras layers/models, and 4,785 MXNet layers/models.

PyBERT. PyBERT is our Transformer encoder pretrained with masked language modeling (MLM) [Devlin et al., 2019] on all open-source Python files from the GitHub dataset. We consider two model sizes: PyBERTSMALL (6-layer, 512-d) and PyBERTBASE (12-layer, 768-d). Pretraining hyperparameters are detailed in Appendix A.1 [Gong et al., 2024].

3 https://console.cloud.google.com/marketplace/details/github/gi thub-repos

4 https://kaggle.com

Adversarial training. The generator and discriminator of ADELT are multilayer perceptrons. We search the learning rate and batch size according to the unsupervised validation criterion "average cosine similarity" [Conneau et al., 2018], which measures the consistency between learned API keyword embeddings and generated keyword translations. Other hyperparameters are set based on previous studies [Conneau et al., 2018] with details described in Appendix A.2.

3.3 Evaluation Benchmark

Our method is evaluated through the task of transpiling code snippets from one DL framework to another. Our benchmark consists of two parts: first, we use heuristics to identify potential matching pairs in the corpus, which were then refined through manual curation to ensure a solid evaluation benchmark; the second part of the benchmark includes a set of expert-transpiled examples, each accompanied by unit tests. For detailed methodology and statistics, please refer to Appendix A.3.

We report results in three evaluation metrics:

• F1 score quantifies the overlap between the predicted and ground truth outputs. In this context, we treat each prediction or ground truth as a bag of function calls. For each test case, we determine the number of exactly matched calls nmatch, predicted calls npred, and ground truth calls ntruth. We define the F1 score for a particular example as 2nmatch/(npred + ntruth), and report the average F1 scores across all test cases.
• Exact Match (EM) score is a more rigorous metric that evaluates whether a model's transpilation is exactly equivalent to the ground truth for each code snippet. It's calculated as the proportion of exact matches to the total number of examples in the eval set.
• Pass@1 assesses the proportion of examples for which the first transpilation attempt by the model successfully passes all the unit tests. These unit tests, created by experts for each benchmark example, evaluate the correctness of transpilations by execution.

3.4 Evaluation of Skeletal Code Transpilation

Transpiling code skeletons of DL programs is an easy task, and Codex easily learned transpilation patterns via few-shot prompting. In our evaluation benchmark, the exact match score of skeletal code transpilation using Codex is 100%.

3.5 Comparison with Other Methods

We compare ADELT using PyBERTSMALL and ADELT using PyBERTBASE with the following baselines. We run all methods 5 times with random seeds [10, 20, 30, 40, 50], and report the arithmetic average of all metrics.

End-to-end language models. We compare ADELT with end-to-end few-shot LLM baselines, including GPT-3, Codex, and GPT-4, where the entire piece of source code, instead of the code skeleton, is fed into the LLM to generate the transpiled target program. For source-to-source translation, we use the "completion" endpoint of code-davinci-002 version for Codex and the "chat" endpoint of gpt-4-0314 for GPT-4. In both cases, we give the LLM a natural language instruction and 5 examples as demonstrations. Details of the prompts are shown in Appendix A.5 [Gong et al., 2024].

Edit distance. We consider a rule-based baseline where we use edit distance [Levenshtein, 1966] as the similarity measure between API keywords, in place of the similarity measures calculated from learned embeddings. We apply hierarchical API dictionary generation exactly as what we do in ADELT. We report the result of both cased and uncased setups for edit distance calculation.

Results. Table 1 shows that ADELT consistently outperforms other methods across all metrics. Notably, ADELT outperforms GPT-4 by significant margins, achieving a 16.2 pts lead in pass@1 of PyTorch-Keras translations and a 5.6 pts lead in Keras-PyTorch. The difference is due to ADELT being based on an encoder-only PyBERT model that leverages larger corpora of the source DL framework (PyTorch) more effectively, unlike GPT-4, a decoder-only LLM that benefits from larger corpora for the target framework. Therefore, ADELT complements traditional end-to-end LLMmethods. Additionally, ADELT runs much faster than GPT-4, as it uses a smaller LLM for transpiling code skeletons and a dictionary lookup step that requires only a tiny fraction of the time needed for full LLM inference.

3.6 Case Studies

Table 2 shows four examples of PyTorch-Keras transpilation together with hypotheses of Codex and ADELT (Base).

Both Codex and ADELT transpile the nn.Conv2d to Keras correctly by dropping the first argument in _ channels. ADELT does not translate the parameter names of nn.Embedding to input _ dim and output _ dim correctly, while Codex does. However, we notice that Codex sometimes relies on the argument ordering heuristic. In the example of nn.MultiheadAttention, where parameters have a different ordering in Keras than in PyTorch, Codex generates the wrong translation, but ADELT successfully constructs the correct mapping between parameters.

Also, in the nn.Embedding example, Codex continues to generate code about "positional embeddings" after finishing transpilation. The extra code generated by Codex is relevant to the context. 5 Still, the extra code should not be part of the translation. We have tried various ways to make Codex follow our instructions (see Appendix A.5 [Gong et al., 2024]). However, because Codex is an end-to-end neural language model, our means of changing its predictions are limited, and the result is highly indeterministic. In the end, Codex still occasionally generates extra arguments or unneeded statements.

On the other hand, we decouple neural network training from the transpilation algorithm. ADELT transpiles between deep learning frameworks using deterministic keyword substitution based on a learned API keyword dictionary. The transpiled code is always syntactically correct. If a mistake is found in the dictionary (e.g., the nn.Embedding example in Table 2), it can be corrected by simply modifying the dictionary.

5 The definition of positional embeddings usually follows the definition of word embeddings (nn.Embedding(vocab _ size, ...)) in the source code of a Transformer model.

Correcting the API keyword dictionary by humans requires much less effort than building the dictionary manually from scratch, as ADELT generates a high-quality dictionary. Developers can even add additional rules to the transpiler. The flexibility of our decoupled design makes ADELT far easier to be integrated into real-world products than end-to-end neural translators/LMs are.

The last case in Table 2 shows an example where an API call (layers.Dense with activation="relu") should be transpiled to two calls (nn.Linear and nn.ReLU). One-to-many mapping is rare in transpilation between deep learning frameworks, but the capability to model such mapping reflects the generality of a transpiler to other APIs. Both ADELT and Codex fail to solve this example because this usage is rarely seen in the training data. Still, if we train ADELT on an additional synthetic dataset ("ADELT +" in Table 2. See Appendix A.8 [Gong et al., 2024]), it successfully solves this case, showing that our method can model one-to-many mappings when enough training data is available.

3.7 Ablation Studies

We conduct ablation studies on PyTorch-Keras transpilation to validate the contribution of each part of ADELT. As illustrated in Figure 1, ADELT consists of five steps. Steps 1 (canonicalization and extraction), 2 (code to skeleton), and 5 (skeleton to code) are deterministic and always yield 100% accuracy as they do not rely on machine learning models. The accuracy of Step 3 (code skeleton transpilation) has been discussed in Section 3.4. Therefore, this section focus on the primary error source: the API keyword translation in Step 4. This crucial step involves mapping API keywords between frameworks. To evaluate its accuracy, we construct a high-quality dictionary by manually translating the top 50 most frequent API keywords from PyTorch to Keras. We then evaluate the translation's efficacy using standard metrics: precision@k (for k = 1, 5) and the mean reciprocal rank (MRR) of correct translations. The results are shown in Table 3, where we study the following variants of ADELT:

Necessity of contextual embeddings. In "w/o PyBERT", we replace PyBERT with Word2Vec [Mikolov et al., 2013] embeddings of the same dimensions db trained on the same corpora. The result in Table 3 shows that this change significantly harms the performance of ADELT. This justifies the use of PyBERT, a high-quality pretrained representation of API keywords that can capture their contexts.

Contribution of adversarial loss. In "w/o Adv Loss", we remove the adversarial loss during training. Instead, we only train the generator and the output embeddings with the crossentropy loss in Equation (2). The result in Table 3 shows that adversarial training contributes ∼6 pts in source-to-source transpilation, showing the effectiveness of adversarial training.

Comparison of similarity measures. By default, ADELT uses cosine similarity as the similarity measure for API dictionary generation. Table 3 shows the results of using dot product (inner). Cosine similarity outperforms dot product by a small margin. This fact implies that the performance of ADELT is insensitive to the choice of similarity measure.

4 RelatedWork

Source-to-source transpilation. Classical source-to-source transpilers use supervised learning. Nguyen et al. [2013] and Karaivanov et al. [2014] develop Java-C# transpilers using parallel corpora of open-source code. The dependency on parallel corpora renders these methods inapplicable to transpilation between deep learning frameworks, as parallel corpora are difficult to get.

Drawing inspiration from unsupervised neural machine translation (NMT) [Artetxe et al., 2018], recent advancements have made unsupervised programming language translation possible [Lachaux et al., 2020]. Such approaches, however, require vast amounts of in-domain unlabeled corpora, as evidenced by Lachaux et al. [2020] and Roziere et al. [2022], who used 744GB of GitHub source code and 333k curated Java functions respectively. The scarcity of DL code hinders their effectiveness for transpilation between DL frameworks.

Metalift [Bhatia et al., 2023] is a transpiler generator for various domain-specific languages (DSLs) [Cheung et al., 2013; Ahmad et al., 2019; Ahmad et al., 2022; Ahmad and Cheung, 2018; Qiu et al., 2024]. It synthesizes target code validated through formal verification. Metalift requires users to explicitly define the target DSL's semantics, while ADELT automatically learns transpilation mappings from data.

Language models are few shot learners. GPT-3 [Brown et al., 2020] is a language model with 175B parameters trained on massive web crawl data. GPT-3 can be applied to many NLP tasks without any task-specific training, with instructions and few-shot demonstrations specified purely via text interaction with the model. Codex [Chen et al., 2021] is a GPT-3 fine-tuned on publicly available code from GitHub, specialized for code generation tasks. GPT-4 is a LLM proficient in both code and NL trained using instruction finetuning. In constrast, the code generation step of ADELT is keyword substitution instead of autoregressive generation. ADELT outperforms GPT-3, Codex, and GPT-4 in PyTorch-Keras transpilation and PyTorch-MXNet transpilation.

Adversarial learning & cross-lingual word embedding. Conneau et al. [2018] uses domain-adversarial [Ganin et al., 2016] approach to align the distribution of two word embeddings, enabling natural language word translation without parallel data. The domain-adversarial training in ADELT is inspired by their approach, but we align the distributions of the hidden states of keyword occurrences.

5 Conclusion

In this work, we present ADELT, an novel code transpilation algorithm for deep learning frameworks. ADELT decouples code transpilation into two distinct phases: code skeleton transpilation and API keyword mapping. It leverages large language models (LLMs) for the transpilation of skeletal code, while using domain-adversarial training for the creation of an API keyword mapping dictionary. This strategic decoupling harnesses the strengths of two different model types. Through comprehensive evaluations using our specially curated PyTorch-Keras and PyTorch-MXNet benchmarks, we show that ADELT significantly surpasses existing state-of-theart transpilers in performance.

Acknowledgements

We thank all reviewers for their feedback and support. This work is supported in part by gift fromMeta, the U.S. National Science Foundation through grants IIS-1955488, IIS-2027575, ARO W911NF2110339, ONR N00014-21-1-2724, and DOE awards DE-SC0016260, DE-SC0021982.