Improving Code Translation with Syntax-Guided and Semantic-aware Preference Optimization

1 Introduction

Code translation, the process of migrating functionality from a source programming language to a target language, is a crucial task in modern software engineering. It promotes code reuse, facilitates legacy system modernization, and enables cross-language interoperability in an increasingly heterogeneous software ecosystem [Nguyen et al., 2014; Zhu et al., 2022a; Yan et al., 2023]. While recent advances in pre-trained language models have shown immense potential [Lu et al., 2021; Zhu et al., 2022b; Zheng et al., 2023; Roziere et al., 2020], their practical application is frequently hindered by a key challenge: ensuring that the translated code is not only syntactically correct but also semantically equivalent to the source. Even state-of-the-art large language models (LLMs) often make errors due to inadequate understanding of the syntactic and semantic nuances across different programming languages [Pan et al., 2024; Yang et al., 2024].

∗Corresponding author

To address this, a natural evolution is to move beyond standard supervised finetuning and adopt preference-based learning. By leveraging reward signals derived from user judgments or task-specific criteria, reinforcement learning from human feedback (RLHF) [Ouyang et al., 2022] has been widely employed to align model outputs with human preferences. However, applying this paradigm to code translation raises a fundamental question: How can we define and ob-

and semantic consistency? For syntax, compiler feedback serves as an infallible oracle, providing a deterministic and binary signal of syntactic correctness [Shojaee et al., 2023]. For semantics, however, the path is fraught with challenges.

As illustrated in Figure 1, prevailing strategies for semantic rewards are built on flawed foundations. The most common strategy is to derive rewards from test cases, treating test outcomes as a proxy for functional correctness [Le et al., 2022; Gee et al., 2025; Zhang et al., 2025]. However, real-world test suites are often sparse and exhibit low coverage, which can lead to reward hacking [Ma et al., 2025], where the model may overfit to these limited test cases without preserving the intended semantics of the source code. In the context of code translation, relying on test cases leads to a critical entanglement of syntax and semantics. A single syntactical divergence in the target language often invalidates the entire execution, yielding a zero pass rate regardless of semantic accuracy. Consequently, the model receives a binary reward that fails to decouple semantic correctness from syntactic validity, making it impossible to quantify the magnitude of semantic deviation. An alternative is to use the reference translation as a semantic anchor. However, this approach gives rise to two flawed strategies. The first strategy compares the generated code to the reference using text-based similarity metrics such as CodeBLEU [Ren et al., 2020]. This strategy is fundamentally superficial, as it rewards lexical similarity over true functional equivalence. The second, more sophisticated strategy introduces a monolingual semantic model to evaluate the similarity between candidate translations and the reference. However, it implicitly assumes that the reference translation is a perfect and complete representation of the source semantics. This assumption introduces a semantic bottleneck that restricts output diversity and may even propagate errors in the reference. We argue that a robust semantic reward should be disentangled from such flawed proxies and instead be derived directly from the source code itself. We propose CTO, a novel approach that improves code translation via syntaxguided and semantic-aware preference optimization.

We reformulate code translation as a multi-objective preference optimization problem, explicitly modeling both syntax and semantics as distinct objectives. It leverages compiler feedback to construct syntax-aware preference pairs. For semantic alignment, it trains a cross-lingual semantic model via contrastive learning to assess functional equivalence directly between the source and translated candidates. It injects the semantic reward difference of a preference pair directly into the learning process, biasing the preference strength to unify syntactic and semantic objectives within the direct preference optimization (DPO) [Rafailov et al., 2023] framework.

We conduct extensive experiments on pairwise code translation between C++, Java, and Python. Results demonstrate that CTO outperforms existing approaches and alternative preference optimization strategies, achieving a superior alignment with the intended functionality of the source code.

To summarize, our main contributions are as follows:

• We propose CTO, a novel approach that improves code translation via syntax-guided and semantic-aware preference optimization. It introduces a reward-biasing mechanism within DPO, enabling unified optimization of both syntactic correctness and semantic consistency.
• We train a cross-lingual semantic model as a robust reward source. It directly evaluates functional equivalence between the source code and translation candidates, overcoming the fundamental limitations of reward signals derived from other proxies.
• Our experiments show that CTO improves translation accuracy by up to 3.66% on the TransCoder-Test dataset and 4.27% on HumanEval-X with the finetuned CodeT5 model. With a larger finetuned CodeLlama-7B model, it yields accuracy gains of 5.60% and 6.70%, respectively.

2 RelatedWork

Code Translation. Early works [Roziere et al., 2020; Roziere et al., 2022] on code translation primarily focus on self-supervised learning. Building upon this, several works [Szafraniec et al., 2023; Huang et al., 2023; Liu et al., 2023] explore incorporating code structure information to enhance code representations for unsupervised code translation. These methods demonstrate the feasibility of learning crosslanguage representations without parallel data. Despite their effectiveness, unsupervised approaches often struggle with collecting enormous amounts of code corpora and high computational resource consumption.

With the availability of high-quality parallel datasets [Zhu et al., 2022a; Yan et al., 2023; Khan et al., 2024; Yan et al., 2024] and pre-trained code models [Wang et al., 2021; Guo et al., 2021; Zheng et al., 2023; Feng et al., 2020], supervised finetuning has become a paradigm for code translation. Building on this foundation, recent studies explore incorporating reinforcement learning (RL) techniques [Shojaee et al., 2023] and variational inference techniques [Du et al., 2024] to further enhance the performance. Besides, a few recent works [Ibrahimzada et al., 2024; Wang et al., 2024] focus on repository-level code translation, aiming to facilitate complex codebase migration. Our work follows the supervised finetuning paradigm, further advancing code translation performance without relying on any auxiliary datasets.

Reinforcement Learning fromHuman Feedback. RLHF has significantly improved the performance of downstream tasks by aligning models with human preferences. While the reward-based methods like proximal policy optimization [Schulman et al., 2017] and group relative policy optimization [Shao et al., 2024] demonstrate success, they suffer from increased memory overhead and substantial computational burdens due to their requirement for a large number of samples during each policy update cycle.

Reward-free methods address these limitations from a different perspective. Direct preference optimization [Rafailov et al., 2023] eliminates the need for an explicit reward model by implicitly estimating reward signals from preference data. Identity preference optimization [Azar et al., 2024] introduces a general non-decreasing bounded function to mitigate overfitting, while simple preference optimization [Meng et al., 2024] further simplifies the process by removing the reference model altogether. Additionally, some studies [Park et al., 2024; Meng et al., 2024] explore the regularization terms as a margin to improve reward modeling. Despite these advancements, directly applying these techniques to code translation remains suboptimal due to inadequate syntax representation and incomplete semantic modeling. To address this challenge, CTO enhances preference optimization by jointly incorporating syntax and semantics, leading to improved accuracy and robustness in code translation.

3 Methodology

3.1 RewardModel Training

The overall reward comprises both syntactic and semantic components. The syntactic reward is readily accessible, as it can be directly derived from the compiler feedback. This binary and rule-based signal indicates whether the translation code adheres to the target language's grammar, following standard practice in prior work [Shojaee et al., 2023].

In contrast, designing an effective semantic reward presents a more challenging problem, as code semantics encompass not only functional equivalence, but also logical fidelity to the source code's intent. Prior studies have approached this from different perspectives. Execution-based reward like CodeRL [Le et al., 2022] utilizes discrete, rulebased heuristics to evaluate semantic correctness, primarily relying on unit tests. However, test cases are often scarce or even unavailable in function- or program-level translation training data, resulting in unstable execution-based reward signals. Furthermore, such reward is inherently binary and sparse, making it difficult to quantify the proximity of an incorrect translation to the correct logic. Reference-based reward like PPOCoder [Shojaee et al., 2023] employs reward signals derived from static program representations, such as abstract syntax trees and dataflow graphs, which often depend on comparisons with ground-truth reference implementations. The reliance on reference translations restricts their applicability in real-world scenarios, where such references are often incomplete or entirely absent. Moreover, minor code variations, such as variable renaming, can lead to significant runtime or functional errors, which are often not adequately penalized by textual metrics.

Semantic RewardModel. To address these issues, we propose to assess semantic correctness by measuring semantic distance within a high-dimensional latent space. Formally, our objective is to learn a mapping function f : X ∪ Y → Z such that the geometric proximity in Z reflects the functional equivalence between the source code x and the target code y. To shape this latent space, we construct a source-anchored triplet T = (x, y+, Y -) for each training instance. We designate the source code x as the fixed anchor point in the latent space. It serves as the semantic oracle which all translation candidates are measured. The reference translation y+ is defined as the positive sample and Y - = {y1 , . . . , y-K } is semantically divergent negatives. The optimization goal is to pull the embedding f(y+) into the immediate ϵ-neighborhood of the anchor f(x), while explicitly repelling the hard negatives in Y - beyond this neighborhood boundary, thereby establishing a robust semantic margin against subtle deviations.

Training Objective. Our semantic reward model adopts a dual-encoder architecture, which contains a source code encoder and a target code encoder, sharing the same parameters. To effectively learn semantic similarity for cross-lingual code representation, we use the InfoNCE loss [van den Oord et al.,

2018] as the training objective:

where x denotes the source code, y+ and yi denotes reference translation and negative translations respectively. f(·) is the encoder that maps code snippets to semantic representations, cos(·, ·) denotes cosine similarity, and τ is the temperature parameter.

Dataset for Training. The positive samples come from the reference code in the supervised dataset. To construct negative samples for training the semantic reward model, we employ an LLM as a perturbation generator. Specifically, the LLM rewrites the reference code by introducing subtle perturbations that alter the intended semantics. These perturbed variants, which often remain syntactically valid but semantically flawed, making them ideal negative examples. Together with the original reference translations, they form the training dataset for learning cross-lingual semantic alignment.

Reward Score Function. Based on the semanticmodel, we encode both the source and candidate target code into embeddings within a shared vector space. For a set of candidate translations y = {y1, . . . , yn} in the target language, we compute the cosine similarity between each candidate yi and the source code x. The similarity scores are transformed into logit values and standardized via z-score normalization to produce the final semantic reward:

where µ and σ represent the mean and standard deviation of the scores s1, . . . , sn, respectively. This function enables listwise scoring by capturing the semantic similarity between the source code and each candidate translation.

3.2 Preference Optimization

Problem Formulation. We formulate code translation as a multi-objective optimization problem over syntax and semantics, and apply a linear scalarization strategy to integrate the multiple objectives into a unified optimization goal. We denote the syntax objective as g, the semantics objective as s, and the preference dataset for code translation as D = {Dg,Ds}, where Dg and Ds denote the syntactic preference dataset and semantic preference dataset, respectively. Given the dataset D, we define the oracle reward model as a weighted combination of syntax and semantic rewards:

where r∗ g (x, y) and r∗ s (x, y) evaluate the syntax and semantic rewards of the translated target code, respectively. w ∈ [0, 1] is a weight parameter to balance syntactic and semantic preferences. In contrast to the classical Pareto optimization, which seeks to approximate the Pareto front of multiple objectives, we focus on optimizing the code translation model under a specific preference weighting w = 0.5. This choice represents an equal prioritization of syntactic and semantic objectives.

This leads to the following training objective for the translation model πθ:

where πsft is the supervised finetuned model serving as the reference policy, β is a regularization coefficient controlling deviation from the reference model.

While previous works [Shojaee et al., 2023] have leveraged RL for reward-based finetuning, such approaches tend to be unstable and resource-intensive. We propose an RL-free variant of DPO, designed to integrate syntax and semantics into a unified training process efficiently.

Derivation of CTO. The optimal policy under the RLHF objective (Eq. (4)) takes the form:

where the partition function is defined as: Z(x) = ∑y πsft(y | x) exp ( 1β (w · r∗ g (x, y) + (1 - w) · r∗ s (x, y))).

Let Dk = {(x, yw, yl)} denote the pairwise preference dataset under objective k ∈ {g, s}, where yw ≻ yl indicates that yw is preferred over yl for input x. The pairwise preference likelihood under the oracle reward model is given by:

where σ(·) denotes the sigmoid function.

Substituting Eq. (5) into Eq. (6) and approximating the ground-truth reward r∗-k (x, y) from the complementary objective -k using a learned reward model r-k(x, y), we derive the final training objective of CTO:

where -k denotes the objective complementary to k (i.e., if k = g, then -k = s, and vice versa).

This formulation reveals the core mechanism of CTO: by leveraging preference data from one objective (captured in Dk) and integrating the explicit reward difference from the other (r-k) as a rectification term, we realize the simultaneous modeling of both syntactic and semantic objectives within a unified optimization process.

Preference Dataset Construction. Constructing a highquality preference dataset is essential for effective model training. Syntactic feedback is generally more stable and verifiable, as grammar correctness can be deterministically checked via compilation. Such feedback serves as an oraclelike signal that enables the reliable classification of candidate translations into preferred and less preferred sets. In contrast, semantic feedback is inherently more ambiguous. Although extended unit tests and semantic reward models offer approximate semantic judgments, determining the absolute correctness of a translation remains an undecidable problem. Given this discrepancy in reliability, we build our preference dataset primarily based on syntactic correctness, where preferences between candidate translations are derived from deterministic compilation results.

As shown in Figure 2, we begin with a supervised dataset comprising aligned source and target code pairs. A base code LLM is finetuned on this dataset to improve its code translation capabilities. For a source code snippet x, we generate a set of candidate translations {y1, y2, . . . , yn} using the finetuned model. Each candidate yi is passed through a compiler to obtain a binary signal indicating whether the code compiles successfully. Based on the compilation results, the candidates are partitioned into a pass set and a fail set, from which we construct a preference dataset. Instead of selecting random pairs, we employ git-diff 1 to identify the closest matching pair (yw, yl), ensuring that low-gap pairs capture fine-grained preferences, where yw comes from the pass set and yl comes from the failed set. We then incorporate scores from the semantic reward model as soft guidance signals. Specifically, the semantic scores of yw and yl are used during optimization to enrich the syntactic preference with semantic discrimination.

This strategy leverages the determinism of syntactic preference labels while still benefiting from the fine-grained semantic discrimination offered by the semantic reward model. Such integration facilitates the simultaneous optimization of syntactic accuracy and semantic alignment.

4 Experiments and Results

4.1 Experiment Settings

Dataset Construction. Our supervised finetuning training set and preference dataset construction are based on the XLCoST dataset [Zhu et al., 2022a], which contains parallel snippet-level and program-level code for commonly used programming languages, with each sample accompanied by an example test case to verify functionality. We collect the parallel program-level data from the training and validation splits. The supervised finetuning training set consists of 6,884 Java↔C++, 6,419 C++↔Python, and 7,278 Java↔Python samples, with a validation set of 346, 323, and 376 samples for each pair, respectively. After finetuning, we generate 10 candidate responses for each input using a sampling temperature of 0.9.

For the semantic model training dataset, we employ the Qwen3-8B model to rewrite the reference code of XLCoST, thereby generating corresponding negative samples. Because the inference is conducted on candidates sampled from the supervised finetuning model at the semantic model's test time, rather than on the reference data itself, this avoids data leakage and ensures benchmarks remain entirely unseen by the semantic model.

1 https://git-scm.com/docs/git-diff

Benchmarks. To assess the performance of our CTO, we employ two benchmark datasets:

• TransCoder-Test [Roziere et al., 2020] is a standard benchmark for unsupervised code translation, with tasks in Java, C++, and Python (482, 467, and 464 tasks, respectively) and an average of 10 test cases per task.
• HumanEval-X [Zheng et al., 2023] is extended from the HumanEval benchmark [Chen et al., 2021] and designed for cross-lingual code correctness evaluation using automated test cases. It includes 164 tasks, with 6.9 test cases on average.

Evaluation Metrics. Computational Accuracy at top-K (CA@K) measures the proportion of tasks where at least one of the top-K translations generated by the model passes all test cases. We use CA@1, showing the model's effectiveness in generating functionally equivalent code to the reference.

Baseline Methods. We compare CTO to two categories of code translation methods:

• Unsupervised translation does not require parallel corpora for training. Instead, it leverages self-supervised learning techniques: (1) TransCoder [Roziere et al., 2020] is an unsupervised model with approximately 100M parameters that leverages masked language modeling, denoising autoencoding, and back translation as its core pre-training tasks. (2) TransCoder-ST [Roziere et al., 2022] extends TransCoder by incorporating automated unit tests to reduce noise in back translation, thereby improving model performance.
• Supervised translation relies on parallel corpora with aligned source and target code pairs to learn a transformation between different programming languages: (1) CodeT5-SFT is a supervised finetuned variant of CodeT5 [Wang et al., 2021]. We choose CodeT5 as the base model due to its comparable scale (770M) and architecture to TransCoder and TransCoder-ST, ensuring a balanced comparison. (2) PPOCoder [Shojaee et al., 2023] is built upon CodeT5-SFT with proximal policy optimization (PPO). It incorporates compiler feedback and both syntactic and semantic match scores as reward signals to improve translation quality. (3) CodeLlama-

7B-SFT is a supervised finetuned variant of CodeLlama7B [Rozi`ere et al., 2023], which serves as a representative foundational model for code-related tasks. (4) Qwen2.5-Coder-7B-SFT is a supervised finetuned variant of Qwen2.5-Coder-7B [Hui et al., 2024].

Implementation. For supervised finetuning, CodeT5 uses a learning rate of 5e-5 and a batch size of 128, while CodeLlama-7B and Qwen2.5-Coder-7B use 3e-5 and 32, respectively, with gradient accumulation for memory constraints. For the semantic reward model, we employ Qwen3Embedding 0.6B as the base model. The training is conducted with a learning rate of 6e-6 and a batch size of 32 for 5 epochs. For preference optimization, we set the learning rate to 5e-7 for CodeT5-SFT, and 5e-5 for both CodeLlama-SFT and Qwen2.5-Coder-7B. For CodeLlama-7B and Qwen2.5Coder-7B, we apply LoRA [Hu et al., 2022] for all finetuning processes, including both supervised finetuning and preference optimization phases.

All experiments are conducted on one NVIDIA RTX A800 GPU and Ubuntu 20.04 LTS. See our source code 2 for other implementation details.

4.2 Code Translation Results

Table 1 shows the experimental results on the TransCoderTest and HumanEval-X datasets. Our CTO consistently outperforms existing methods across six translation tasks.

Compared with the unsupervised code translation methods, CTO (CodeT5) surpasses both TransCoder and TransCoderST across all evaluated scenarios. This performance gap underscores the efficacy of supervised finetuning on parallel corpora, which facilitates the direct acquisition of crosslingual mappings and ensures higher translation reliability. In contrast, unsupervised methods rely on back translation, which can introduce noise and misalignment. For PPOCoder, we follow its reward function and parameter settings in [Shojaee et al., 2023] and apply PPO on our finetuned CodeT5 model. Contrary to our expectations, PPOCoder exhibits performance degradation in most cases. This implies that PPOCoder does not always benefit from its reward function and may even introduce instability to the finetuned model.

Since the preference dataset is derived from supervised finetuned models, its quality depends on the model's ability to generate valid compilable samples. However, CodeT5SFT generates fewer valid samples in translation tasks from Python, making preference data collection more difficult. This observation is consistent with previous studies [Feng et al., 2024], which have noted a strong correlation between the performance of supervised finetuned models and the effectiveness of preference optimization.

2 https://github.com/nju-websoft/CTO

To assess the scalability of CTO on larger code models, we conduct experiments using decoder-only LLMs, CodeLlama7B and Qwen2.5-Coder-7B. Across six language pairs and two benchmarks, CTO consistently outperforms their supervised finetuned counterparts (CodeLlama-7B-SFT and Qwen2.5-Coder-7B-SFT). These improvements validate the generalization of CTO across different model architectures and parameter sizes.

4.3 Ablation Study and Alternative Methods

To investigate the effectiveness of multi-objective preference optimization in our proposed method, we conduct an ablation study by systematically removing key components and analyzing their impact. To measure the contribution of multiobjective preference optimization, we design two variants:

• CTO w/o semantic removes the semantic preference from the preference optimization stage, meaning the model is optimized solely based on the syntax preference data. Consequently, CTO degenerates into DPO, which helps isolate the impact of semantic reward.
• CTO w/o syntax removes the syntax preference from the preference optimization stage, i.e., the model is optimized solely based on the semantic preference data. In this setting, CTO degenerates to vanilla DPO, where the reference translation is treated as the chosen sample and the rejected samples are standard outputs that do not match the reference.

As shown in Table 2, the results clearly indicate that each preference contributes significantly to the final performance. Notably, in the w/o syntax variant, we rely on the example unit test shipped with the original dataset as auxiliary signals to guide optimization. But the performance still falls short compared to CTO. These results confirm that multi-objective preference plays a crucial role in maximizing performance.

Additionally, to further validate the effectiveness of our preference optimization strategy, we compare our method Table 3: Comparison of CA@1 scores between reference-based metrics and the semantic reward model.

Incorrect function calls

Data manipulation errors

Algorithmic flaws

Incorrect var/op usage

Logical errors

with two typical reward-free preference optimization techniques: identity preference optimization (IPO) [Azar et al., 2024] and simple preference optimization (SimPO) [Meng et al., 2024]. These baselines perform preference optimization on supervised finetuned models, enabling us to assess whether CTO provides a tangible advantage over existing techniques. As also presented in Table 2, CTO achieves

higher performance across most translation tasks on both TransCoder-Test and HumanEval-X benchmarks. While IPO and SimPO fail to match the performance of our method in certain scenarios, they even cause performance degradation when applied to supervised finetuned models.

Overall, these results demonstrate that our CTO effectively integrates syntactic and semantic modeling within a preference optimization framework, leading to superior performance in code translation tasks. These findings reinforce the importance of multi-objective preference optimization as a robust strategy for enhancing code translation models.

4.4 RewardModel Evaluation

Negative Sample Types. To evaluate the quality and diversity of our semantic reward model's training data, we analyze the distribution of various negative sample types, following the semantic error classification outlined in [Pan et al., 2024]. As shown in Figure 3, the categorization reveals that logical errors constitute the largest portion of the dataset. The extensive coverage of error categories, coupled with their relatively balanced representation, prevents the model from being biased toward specific patterns and ensures robust detection of various semantic discrepancies.

Latent Space Visualization. To assess the effectiveness of our trained reward model, we conduct an evaluation of our semantic reward model and visualize the source and target code embeddings derived from the reward model in a shared vector space. Figure 4 shows that embeddings from different programming languages form coherent clusters instead of being strictly segregated by language. This verifies the model's ability to fuse cross-lingual code semantics into a unified embedding space, suggesting that it captures meaningful alignment between source and target language structures.

Effectiveness of the Semantic RewardModel. Lastly, we compare the performance of our semantic reward model and previous reference-based metrics such as CodeBLEU [Ren et al., 2020] and CodeBERTScore [Zhou et al., 2023] when applied to CTO. As presented in Table 3, our semantic reward model consistently outperforms both CodeBLEU and CodeBERTScore across most scenarios. This performance gap highlights the advantage of leveraging our learned semantic model over relying solely on reference-based metrics as the semantic reward.

Overall, these results confirm the cross-lingual semantic coherence captured by our semantic reward model and its superior effectiveness over prior reference-based metrics.

5 Conclusion

In this paper, we present CTO to improve code translation by syntax-guided and semantic-aware preference optimization. We formulate code translation as a multi-objective preference optimization problem. By leveraging compiler feedback and semantic reward modeling, CTO effectively refines translation quality, ensuring both syntactic correctness and semantic alignment. Experimental results across multiple benchmarks demonstrate that CTO significantly outperforms state-of-theart methods in code translation accuracy and robustness. The proposed preference optimization approach is also generalizable, making it adaptable to various programming languages and model architectures. In future work, we will extend CTO to support a broader range of programming languages and explore its effectiveness in other software migration scenarios.

Acknowledgments

This work was supported by the "111 Center" (No. B26023), the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (No. JYB2025XDXM118), and the Cooperation Fund of Huawei Cooperation Project (No. TC20230202021-2024-12).