SynthRL: Cross-domain Synthesizer Sound Matching via Reinforcement Learning

1 Introduction

In modern music production, sound design plays a critical role in shaping the auditory identity of a track, enabling producers to express their creativity and evoke specific emotions in listeners. A key tool in sound design is the synthesizer, which allows producers to generate and manipulate a vast variety of sounds by combining various waveforms, filters, and modulation techniques. However, crafting the desired sound with a synthesizer is a complex process, requiring significant study, effort, and time to master the manipulation of its parameters. Furthermore, synthesizers differ in the methods they use to produce sounds, such as additive, subtractive, and frequency modulation (FM) synthesis, meaning that even experienced users may find it challenging to use different types of synthesizers proficiently [Powell, 1987]. Consequently, there has been growing research focused on developing methods to automatically search for synthesizer parameter settings that can replicate the timbre of sounds from other songs or sound samples [Sebastian et al., 2009;

Roth and Yee-King, 2011; Tatar et al., 2016]. Using a model that has learned this task called synthesizer sound matching, even beginners can easily create the desired sound through a synthesizer and further adjust parameters to refine the sounds.

With advancements in deep learning, recent studies have proposed methods utilizing deep neural networks to estimate synthesizer parameters for desired sound inputs [YeeKing et al., 2018; Barkan et al., 2019; Vaillant et al., 2021; Chen et al., 2022]. One of the primary challenges in these studies is that the sound generation process in synthesizers is non-differentiable. In other words, it is infeasible to incorporate the sound generated by the synthesizer based on the estimated parameters into the differentiable loss function for backpropagation, such as the spectrogram difference from the ground truth sound. Therefore, previous studies have focused on training the model using only a parameter loss that minimizes the error between the ground truth synthesis parameters and the estimated parameters. However, since the goal is to generate a sound that closely resembles the input sound, the inability to directly use the perceptual difference between the ground truth sound and the generated sound in the learning process represents a significant limitation. In addition, using only the parameter loss requires the availability of ground truth parameters, meaning that only sounds generated from known synthesizer parameters (in-domain) can be used as training data. For example, users often aim to replicate sounds sourced from a different type of synthesizer or instrument (out-of-domain) using their own or more familiar synthesizer. However, since out-of-domain sounds cannot be used to calculate parameter loss, models trained with only indomain sounds may perform poorly when encountering these unfamiliar sounds.

A prominent line of research to address these limitations involves the implementation of synthesizers using differentiable digital signal processing (DDSP) modules [Masuda and Saito, 2021; Caspe et al., 2022; Uzrad et al., 2024]. The sound synthesis process of a DDSP synthesizer is fully differentiable, allowing the generated sound to be used for training through backpropagation. Nonetheless, this approach confines the application of the sound matching model to specific DDSP synthesizers, hindering users from applying the model to their preferred synthesizers. Moreover, implementing a wide variety of conventional synthesizers in a differentiable manner requires considerable effort for each synthesizer, due to the distinct and complex sound synthesis techniques they employ.

To address the aforementioned issues in synthesizer sound matching, we propose a novel approach using reinforcement learning (RL), named SynthRL. SynthRL employs the generated sound in the reward function of RL, enabling the model to optimize for greater similarity to the ground truth sound. Consequently, our approach facilitates the incorporation of perceptual loss into the training of non-differentiable synthesizer, as well as the use of out-of-domain sounds. We also demonstrate that reward-based prioritized experience replay (PER) [Schaul, 2015], tailored to the unique characteristics of the synthesizer sound matching task, is crucial for the effective application of RL. Moreover, while our primary focus lies in the application of RL, we also introduce an effective synthesizer sound matching model architecture based on transformer encoder-decoder framework. Through extensive experiments, we demonstrate that our method outperforms existing baselines on both in-domain and out-of-domain using a widely used conventional synthesizer.

2 RelatedWorks

2.1 Synthesizer SoundMatching

The complexity and difficulty of operating synthesizers have motivated considerable efforts to automatically estimate the parameter sets required to create desired sounds. Early studies employed search-based optimization methods such as genetic algorithms [Horner et al., 1993; Garcia, 2002; Mitchell and Sullivan, 2005; Yee-King and Roth, 2008; Tatar et al., 2016] or Particle Swarm Optimization [Sebastian et al., 2009].

More recent advancements have leveraged neural networks, where models are trained to minimize the error between the ground truth parameters and the estimated parameters. For instance, FlowSynthesizer [Esling et al., 2019] integrates an autoencoder with normalizing flows, followed by PresetGenVAE [Vaillant et al., 2021], which further advances similar methods. Sound2Synth [Chen et al., 2022], a previous state-of-the-art (SOTA) method, extracts a diverse set of acoustic features and integrates them as model inputs. These approaches show promising results for in-domain sounds.

The non-differentiability of synthesizers has spurred extensive research into the development of differentiable neural synthesizers [Engel et al., 2020; Shan et al., 2022; Renault et al., 2022; Wiggins and Kim, 2023]. In line with these studies, previous research addressing cross-domain synthesizer sound matching has predominantly employed approaches that develop differentiable synthesizers [Masuda and Saito, 2021; Caspe et al., 2022; Uzrad et al., 2024]. Additionally, InverSynth II [Barkan et al., 2023] proposed a differentiable proxy model of the synthesizer and a inference-time finetuning strategy for improved in-domain sound matching.

To the best of our knowledge, there has been no research focused on cross-domain sound matching for conventional non-differentiable synthesizers so far. In this work, we introduce RL to synthesizer sound matching for the first time, providing a framework that facilitates performance improvements across both in-domain and out-of-domain sounds.

2.2 Reinforcement Learning for Fine-tuning

Reinforcement learning (RL) has emerged as a powerful paradigm for fine-tuning models, particularly in scenarios where explicit gradients for feedback are unavailable. For example, early applications included tasks such as machine translation [Ranzato et al., 2016; Bahdanau et al., 2017] and summarization [Wu and Hu, 2018; Gao et al., 2019], where non-differentiable evaluation metrics were used as rewards to guide model optimization. In recent years, reinforcement learning from human feedback (RLHF) has gained widespread adoption across diverse domains, including large language models [Ziegler et al., 2019; Ouyang et al., 2022; Bai et al., 2022], image generation [Lee et al., 2023; Fan et al., 2023; Black et al., 2024], and music generation [Cideron et al., 2024]. These studies have demonstrated that RLHF successfully aligns model outputs with human preferences, enhancing the relevance and quality of generated content. Building upon insights from these prior studies, we adopt RL to tackle the challenges posed by the non-differentiability of synthesizers, aiming to optimize performance in crossdomain sound matching.

3 Method

3.1 Overview

The problem of synthesizer sound matching can be framed as predicting the optimal synthesis parameters x = {xi}ni=1 , so that when passed to a synthesizer S, the generated audio signal ˆ y = S(ˆx) closely matches the target sound y. The synthesis parameters x include n attributes that define the behavior of the synthesizer, such as the waveform type of oscillators, amplitude envelope, filter configurations, and modulation depth. We employ a neural network model πθ, which takes the target sound as input and estimates the corresponding synthesis parameters, ˆx = πθ(y).

Our model training process consists of three stages, as illustrated in Figure 1. First, we train the model by minimizing the parameter loss using only in-domain sounds. In the subsequent stage, we gradually introduce RL loss to further enhance performance on in-domain sounds. Finally, RL is applied exclusively to fine-tune the model on out-of-domain sounds, improving the cross-domain generalization of sound matching. Through the following sections, we provide a detailed breakdown of our proposed approach including the model architecture and the specifics of the learning process.

3.2 Model Architecture

Our proposed SynthRL model is based on a transformer encoder-decoder architecture (Figure 2). First, the target waveform y is converted into a single-channel melspectrogram of size F ×T ×1. The encoder begins with a stack of ℓ 2D convolutional layers, each with a stride of 2, reducing the input melspectrogram from size F ×T ×1 to F/2ℓ ×T /2ℓ ×d, where d is the output channel dimension. The output of the 2D CNN is flattened into a vector of size (FT/22ℓ ) × d and passed into the transformer encoder with 2D positional encoding added. The final output of the encoder is a feature map z ∈ R(FT/22ℓ )×d of the target sound.

The decoder adopts a non-autoregressive transformer decoder architecture [Carion et al., 2020]. It takes as input a set of n learnable query vectors in parallel, each of dimension d, where n corresponds to the number of synthesis parameters. Each query vector represents a different synthesis parameter. These queries first undergo self-attention to capture dependencies among the synthesis parameters, followed by crossattention with the feature map z. In this step, the feature map serves as the key and value, allowing the decoder to incorporate relevant information from the target sound representation while attending to each query. The resulting decoder output vectors are then passed through n projection heads, which generate logits to estimate ˆxi for each synthesis parameter.

Each projection head is designed to handle a specific synthesis parameter. In most synthesizers, each synthesis parameter xi ∈ x belongs to either a categorical or numerical type. For example, x1 could denote the wave type of an oscillator, taking one of four categorical values: sine, triangle, square or sawtooth. In contrast, x2 might correspond to the filter cutoff frequency, which is a continuous numerical value ranging between 0 and 1. To unify the estimation process, we discretize all numerical parameters into 25 equally spaced classes, converting their estimation into a classification problem. Consequently, each projection head maps the d-dimensional output of its corresponding decoder query to a vector with a dimensionality equal to the number of classes for the associated parameter.

3.3 Parameter Loss

Building upon the classification framework for synthesis parameters introduced earlier, we define the parameter loss as a cross-entropy loss between the ground truth parameters x and their corresponding estimates ˆx. For discretized numerical parameters, however, a direct application of crossentropy can overlook the inherent structure of these parameters, where errors involving adjacent classes are less significant than those involving distant ones.

To address this issue, we smooth the ground truth label distribution by redistributing a portion of the probability mass from the true class to its neighboring classes. Specifically, the ground truth label is convolved with a Gaussian kernel and the smoothed distribution is normalized to maintain the total probability, following the prior work [Chen et al., 2022]. This approach ensures that predictions closer to the ground truth are penalized less severely, which better aligns with the numerical relationships between the classes.

3.4 Reinforcement Learning Procedure

In this section, we provide a detailed description of the RL procedure applied to synthesizer sound matching. First, we present the RL formulation, followed by the design of the reward function. Finally, we elaborate on the use of PER with our modification to enhance the learning efficiency.

RL Formulation

A typical RL formulation considers problems as Markov Decision Process (MDP) defined by a tuple (S, A, R, P ), where S denotes a state space, A is an action space, R is a reward function, and P is a transition probability. The objective of RL is to learn an optimal policy that maximizes the expected cumulative reward. We formulate the problem of synthesizer sound matching within an RL framework, defining the state as the target sound y ∈ S, the action as the estimated synthesis parameter ˆx ∈ A, the reward as the similarity between the target and generated sound. The policy corresponds to the SynthRL model πθ, which takes y as input and outputs ˆx.

In contrast to the majority of RL tasks that revolve around multi-step decision processes, our problem formulation is centered on a single-step decision process which still fits within the MDP framework. This formulation can also be viewed as a contextual bandit problem [Lu et al., 2010]. Therefore, the cumulative reward is equivalent to the single reward for the current state-action pair (y, ˆx), and our RL objective function to maximize is given by J(θ) = Eˆx∼πθ [R(y, ˆ y)], where ˆ y = S(ˆx) is the generated sound by the synthesizer S. We employ REINFORCE [Williams, 1992], which is a simple yet effective algorithm particularly for single-step processes. REINFORCE updates the policy by gradient ascent, where the gradient of the objective function is computed with respect to the policy parameters. The gradient of the objective function is defined as follows:

From this equation, the reward R(y, ˆ y) is a scalar value computed from the target and generated sounds, and the gradient is required only for the log-probability of the policy πθ. Therefore, our RL objective enables policy optimization for a non-differentiable synthesizer S, utilizing a reward based on sound similarity.

Reward Function

To optimize the policy for estimating synthesis parameters that produce sounds ˆ y = S(ˆx) closely resembling the target sound, we design the reward function based on perceptual similarity metrics. We integrate the following three metrics which are commonly used in sound similarity measurement:

• Spectral convergence [Arık et al., 2019]:

• MFCCMAE [Horner et al., 2011]:

where STFT(·) is the short-time Fourier transform and MFCC(·) is the 13-band mel-frequency cepstral coefficients. These metrics all exhibit lower values as sound similarity increases. The reward function is defined as the reciprocal of a weighted sum of the metrics as follows:

Algorithm 1 Reward-based PER

Require: Synthesizer S, policy πθ, learning rate α, buffer capacity m, dataset D

Initialize: Prioritized replay buffer B

• 1: for epoch = 1,N do
• 2: for each iteration do
• 3: Sample mini-batch of target sounds: yi ∼ D // Prioritized replay buffer update
• 4: Sample actions from policy: ˆxi ∼ πθ(·|yi)
• 5: Generate sounds ˆ yi = S(ˆxi)
• 6: Calculate rewards Ri = R(yi, ˆ yi) using Eq. 5
• 7: if epoch ≤ m then
• 8: Store experiences (yi, ˆxi, Ri) into B
• 9: else if Ri > min(R(yi, ˆ y) ∈ B) then
• 10: Replace argmin (yi ,ˆx,R)∈B R(yi, ˆ y) with (yi, ˆxi, Ri)
• 11: end if

// Policy training

• 12: Sample experiences: (y, ˆx,R(y, ˆ y)) ∼ B
• 13: Update πθ using Eq. 6:
• 14: θ ← θ + α∇θJ(θ)
• 15: end for
• 16: end for

The weights w1, w2, and w3 are hyperparameters, and we empirically set their default values to w1 = 0.27, w2 = 0.7, and w3 = 0.03 considering the scale of each metric. In the experimental section, we will provide a detailed analysis of the reward function including alignment between the computed reward and subjective evaluations of similarity, as well as limitations of using a single metric as the reward.

Reward-based PER

As mentioned earlier, the REINFORCE algorithm updates the policy πθ using rewards R(y, ˆ y) obtained from stochastically sampled actions ˆx ∼ πθ(y) by gradient ascent using Equation 1. This learning process enables the policy to iteratively improve by assigning higher probabilities to actions that yield greater rewards. However, we found that directly applying this approach to synthesizer sound matching did not lead to effective learning from our preliminary experiments.

From the perspective of RL, the synthesizer sound matching task involves a complex action space comprising hundreds of synthesis parameters. Moreover, even minor changes to certain parameters can drastically alter the resulting sound, particularly for categorical parameters where discrete changes can result in significant timbre shifts. Consequently, stochastically sampling actions from the policy can give rise to sparse occurrences of high-reward actions, impairing the learning efficiency.

To mitigate this challenge, we leverage PER, a technique originally proposed to enhance learning efficiency in offpolicy RL algorithms by prioritizing experiences that would most likely lead to an improvement in the policy or value function. In our approach, we selectively reuse experiences with high rewards which are directly associated with the learning objective. Concretely, for each target sound y, we maintain a prioritized replay buffer that stores m actions and their corresponding rewards. When the buffer is full, a new action ˆx sampled from the policy πθ(y) replaces the leastrewarded experience in the buffer if its reward exceeds the current minimum. These experiences stored in the prioritized replay buffer are uniformly sampled to update the policy. In this case, a discrepancy arises between the distribution µ(ˆx|y) used to sample actions induced by the prioritized replay buffer and the policy πθ(ˆx|y) being optimized, since REINFORCE is inherently an on-policy algorithm which precludes the reuse of old experiences. This mismatch is corrected by importance sampling, where the gradient is reweighted based on the ratio of probabilities under these two different distributions [Jie and Abbeel, 2010]. As a result, the gradient of the objective becomes as follows:

For practical implementation, we approximate the sampling distribution as uniform over the buffer, µ(ˆx|y) = 1/m. The reinforcement learning process using reward-based PER is summarized in Algorithm 1.

4 Experiment

4.1 Dataset

While our proposed method is applicable to various types of synthesizers, we conducted our experiments using the Dexed 1 synthesizer, which has been widely adopted in previous synthesizer sound matching research. Dexed is an open-source synthesizer modeled on the Yamaha DX7 which is one of the most iconic FM synthesizers. It includes 144 synthesis parameters that collectively shape the timbre, comprising 84 numerical and 60 categorical types. We use target sound datasets consisting of single MIDI notes played at pitch 60 and velocity 85. Each note has a duration of 4 seconds, with 3 seconds of sustain followed by 1 second of release, providing a well-defined envelope for the synthesized sounds.

For the in-domain dataset, we use sounds generated from approximately 30,000 publicly available Dexed sound presets collected in prior research [Vaillant et al., 2021]. This dataset excludes redundant or silent sounds, processed by previous researchers. For the out-of-domain dataset, we use sounds generated from a different type of open-source synthesizer Surge XT 2 that employs a subtractive hybrid synthesis method. Due to the fundamental difference in synthesis methods, Surge XT produces a distinct set of sounds compared to those of Dexed. We collected approximately 3,500 non-redundant sounds from publicly available Surge presets. These datasets were divided into training, validation, and test sets using a ratio of 64%, 16%, and 20%, respectively.

1 https://github.com/asb2m10/dexed

4.2 Training Details

Through the following experiments, we compare SynthRL with two baseline methods and investigate the effectiveness of the proposed components in our method. The first baseline is Sound2Synth, a SOTA method demonstrated on the Dexed synthesizer for sound matching. The second is PresetGenVAE, which exhibits relatively smaller performance degradation on the out-of-domain dataset.

SynthRL is first trained on the in-domain dataset for 400 epochs. Over the initial 200 epochs, only the parameter loss described in Section 3.3 is applied. During the next 100 epochs, both the RL loss from Equation 6 and the parameter loss are used, with the ratio of the RL loss linearly increasing from 0 to 1. The final 100 epochs of in-domain training are performed using only the RL loss. After the in-domain training is completed, the model is fine-tuned on the out-ofdomain dataset for an additional 400 epochs using only the RL loss.

We assign distinct names to the trained models based on the three training stages, for the ablation study on the RL training which is a key component of our proposed method as follows:

• SynthRL-p is the model trained for initial 200 epochs using only the parameter loss (stage 1).
• SynthRL-i is further trained model on the in-domain dataset with the addition of RL loss from 200 to 400 epochs (stage 2).
• SynthRL-o is the fine-tuned model on the out-ofdomain dataset (stage 3).

The baseline models are trained on the in-domain dataset for 400 epochs using the official source code released by the authors. These in-domain-only trained models are also used for evaluation on the out-of-domain dataset, since they are inherently incapable of out-of-domain training, which provided significant motivation for our research as mentioned earlier. Our source code and audio samples are available on the accompanying webpage 3 .

2 https://github.com/surge-synthesizer/surge

5 Result and Analysis

5.1 Quantitative Evaluation

Following previous studies [Vaillant et al., 2021; Barkan et al., 2023], we evaluate sound similarity quantitatively using spectrogram MAE, spectral convergence, and MFCC MAE. Table 1 presents the results for all evaluation metrics on both the in-domain and out-of-domain datasets. Overall, our method outperforms the baselines across all metrics on both domains.

For the in-domain results, SynthRL-p outperforms the stronger baseline Sound2Synth by an average 4.1% across all metrics. The performance gain of SynthRL-p over the baselines is attributed to the model architecture of SynthRL, as it is trained with only the parameter loss similar to other methods. It indicates that our proposed transformer encoder-decoder based architecture is effective for synthesizer sound matching. A comparison of SynthRL-p and SynthRL-i demonstrates that the addition of RL leads to further performance improvement on average 5.8%. Our RL approach successfully compensates for limitations of the parameter loss, such as the difficulty in learning subtle parameter differences that result in noticeable timbre changes.

On the out-of-domain dataset, the baselines and SynthRLi all exhibit noticeable performance drops compared to the in-domain case. This degradation inherently arises from the limited in-domain sound distribution of the synthesizer. Owing to the cross-domain applicability of our RL method, SynthRL-o shows a substantial improvement over SynthRLi by an average 17.9% across all metrics. Compared to the baselines, SynthRL-o outperforms PresetGenVAE by an average 17.2% and Sound2Synth by 37.7%.

5.2 Subjective Evaluation

We conducted Mean Opinion Score (MOS) and ABX similarity tests through Amazon Mechanical Turk (MTurk). In the MOS test, 64 evaluators rated the similarity of the samples to the ground truth on a scale from 1 (completely different) to 5 (identical). In the ABX similarity test, 96 evaluators listened to two samples generated by different models and chose the sample more similar to the ground truth. For each test, 64 randomly selected target sounds were evaluated and all evaluators were instructed to wear headphones in a quiet environment. We intentionally included pairs of completely different and identical sounds in the evaluation samples, and responses from evaluators who failed to distinguish them were considered unreliable and excluded. The results of the MOS and ABX tests are presented in Table 2 and Figure 3, respectively.

3 https://argaaw.github.io/synthrl-demo/

For the in-domain dataset, SynthRL-i achieves the highest MOS of 4.29. However, considering the 95% confidence interval (CI), the differences from SynthRL-p (4.16) and Sound2Synth (3.91) are not statistically significant. This result indicates that all three models exhibit strong in-domain performance, rather than highlighting the numerical differences since it is difficult for humans to distinguish subtle differences using 5-point scale. The pairwise performance differences between individual models are more clearly demonstrated in the ABX test results. In this test, SynthRL-i is preferred over all other models, with a minimum of 37.9% win and 21.9% loss compared to SynthRL-p.

In the out-of-domain MOS test, SynthRL-o scores 3.01 with a 95% CI of 0.26, considerably outperforming SynthRLi (2.28) and the other baselines. This remarkable improvement is further validated by the ABX test, where SynthRL-o achieves significantly higher preference rates compared to all other models, with a minimum of 70.5% over PresetGenVAE. Overall, these results demonstrate that our RL method leads to consistent improvements across both domains from a human perceptual perspective, as well as in quantitative performance.

5.3 Analysis of Reward Design

Our reward design is a combination of three sound similarity metrics, motivated by the experimentally observed limitations of using a single metric. For instance, spectrogram MAE is a widely used metric for sound loss functions, along with various variants such as multi-scale approaches [Engel

et al., 2020]. However, using a reward solely based on spectrogram MAE in RL can lead to undesired learning outcomes, as illustrated in Figure 4. The top row depicts spectrograms of a target sound with few harmonics (left) and the corresponding output of a SynthRL model trained with the single spectrogramMAE-based reward (right). In this case, the two sounds differ significantly as the target sound is sustained for a long duration, whereas the estimated one is transient. Nevertheless, the spectrogram MAE between the two sounds is 0.182, which is notably low compared to 0.292 between spectrograms that are barely distinguishable as shown in the bottom row. This low value stems from the widely distributed low energy regions represented in black on the spectrograms. Consequently, using only spectrogram MAE as the reward leads the model to consistently output transient sounds for diverse target sounds with overall low energy. In contrast, the other two metrics exhibit higher values compared to the bottom row, particularly spectral convergence with a significant high value of 1.789, effectively reflecting the difference of the two sounds. The complementary relationship among these reward components plays a crucial role in successful training of SynthRL across diverse sounds.

We further investigate the alignment between our designed reward function and human perception of sound similarity. A total of 120 evaluators recruited via MTurk rated the similarity of randomly sampled pairs of sounds on a 5-point MOS scale, with each pair being rated by 30 evaluators. Figure 5 shows the scatter plot of the rewards calculated from Equation 5 and the MOS for the sample pairs. It demonstrates a clear positive relationship between the rewards and MOS, with a Pearson correlation of 0.85 (p < 0.001). While the deviations are relatively high in the mid-range MOS (2-4), considering the subjective nature of sound similarity judgments when the sounds are neither completely different nor identical, the results suggest that our reward function is welldesigned to align with the intended goal.

5.4 Effect of Reward-based PER

We conduct an ablation study to analyze the effectiveness of reward-based PER. Figure 6 illustrates average reward curves on validation sets over training. Step 0 in the figure represents the point at which RL training begins for each domain, corresponding to fully trained SynthRL-p for in-domain and fully trained SynthRL-i for out-of-domain. The result demonstrates that training with reward-based PER is more effective in both cases. Interestingly, the performance on indomain decreases during training when reward-based PER is not used. We conjecture that the degradation occurs because the model already achieves sufficiently high performance through parameter loss training before the introduction of RL loss. In other words, applying RL without prioritized sampling in the complex state and action space of the synthesizer sound matching task can lead to inferior performance compared to training with only parameter loss.

6 Conclusion

We introduce SynthRL, the first application of RL in synthesizer sound matching, addressing the limitation of reliance on parameter loss and the challenges of cross-domain generalization. SynthRL effectively incorporates perceptual loss through RL, enabling fine-tuning on out-of-domain sounds with non-differentiable synthesizers, as well as further improving in-domain performance. Our extensive experiments shed light on the essential factors for successfully applying RL to synthesizer sound matching, including the effective reward design and the importance of prioritized sampling.

Acknowledgements

This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) [No. RS-2024-00461617, 50%], Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No. RS-2022-II220320, 2022-0-00320, 40%], [No. RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University), 5%], and [No.RS-2021 II212068, Artificial Intelligence Innovation Hub (Artificial Intelligence Institute, Seoul National University), 5%].