Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation

1 Introduction

In decision-making scenarios, each expert might have her own preference on multiple, possibly conflicting objectives (multi-objectives). Accordingly, learning Pareto-optimal policies in multi-objective environments has been considered essential and practical to provide users with a selection of diverse expert-level policies, which can cater their specific preferences (e.g., [Xu et al., 2020; Kyriakis et al., 2022]). However, in the area of imitation learning, such multi-objective problem has not been fully explored due to the requirement for comprehensive expert datasets encompassing the full range of multi-objective preferences (e.g., [Zhu et al., 2023]), which might be unattainable in real-world scenarios.

∗Honguk Woo is the corresponding author.

In the ideal scenario depicted on the left side of Figure 1, having comprehensive expert datasets encompassing diverse multi-objective preferences enables the straightforward derivation of a Pareto policy set by reconstructing policies from each dataset. However, this is often not feasible in real-world situations where datasets might not represent all preferences. This common limitation is illustrated on the right side of Figure 1. Here, one typically has access to only two distinct datasets, each reflecting different multiobjective preferences. In such limited dataset cases, a viable approach involves merging these datasets in varying proportions, followed by the application of imitation learning on each blended dataset. However, this approach often leads to a collection of non-Pareto-optimal policies, as demonstrated in Section 4.

In this paper, we address the challenges of multi-objective imitation learning in situations with strictly limited datasets, specifically focusing on Pareto policy set generation. Our goal is to derive optimal policies that conform with diverse multi-objective preferences, even in the face of limited datasets regarding these preferences. To do so, we investigate inverse reinforcement learning (IRL) and present a Pareto IRL (ParIRL) framework in which a Pareto policy set corresponding to the best compromise solutions over multiobjectives can be induced. This framework is set in a similar context to conventional IRL where reward signals are not from the environment, but it is intended to obtain a dense set of Pareto policies rather than an individually imitated policy.

In ParIRL, we exploit a recursive IRL structure to find a Pareto policy set progressively in a way that at each step, nearby policies can be derived between the policies of the previous step. Specifically, we adapt IRL using reward distance regularization; new policies are regularized based on reward distance estimates to be balanced well between distinct datasets, while ensuring the regret bounds of each policy. This recursive IRL is instrumental in achieving the dense approximation of a Pareto policy set. Through distillation of the approximated Pareto policy set to a single policy network, we build a diffusion-based model, which is conditioned on multiobjective preferences. This distillation not only enhances the Pareto policy set but also integrates it into a single unified model, thereby facilitating the zero-shot adaptation to varying and unseen preferences.

The contributions of our work are summarized as follows.

• We introduce the ParIRL framework to address a novel challenge of imitation learning, Pareto policy set generation from strictly limited datasets.
• We devise a recursive IRL scheme with reward distance regularization to generate policies that extend beyond the datasets, and we provide a theoretical analysis on their regret bounds.
• We present a preference-conditioned diffusion model to further enhance the approximated policy set on unseen preferences. This allows users to dynamically adjust their multi-objective preferences at runtime.
• We verify ParIRL with several multi-objective environments and autonomous driving scenarios, demonstrating its superiority for Pareto policy set generation.
• ParIRL is the first to tackle the data limitation problem for Pareto policy set generation within the IRL context.

2 Preliminaries and ProblemFormulation

2.1 Background

Multi-Objective RL (MORL). A multi-objective Markov decision process (MOMDP) is formulated with multiple reward functions, each associated with an individual objective.

Here, s ∈ S is a state space, a ∈ A is an action space, P : S × A × S → [0, 1] is a transition probability, and γ ∈ [0, 1] is a discount factor. MOMDP incorporates a vector of m reward functions r = [r1, ..., rm] for r : S×A×S → R, a set of preference vectors Ω ⊂ Rm, and a linear preference function f(r,ω) = ωTr where ω ∈ Ω. The goal of MORL is to find a set of Pareto polices π∗ ∈ Π∗ for an MOMDP environment, where π∗ maximizes scalarized returns, i.e., maxπ Ea∼π(·|s)[∑Ht=1 γ t f(r, ω)].

Inverse RL (IRL). Given an expert dataset T ∗ = {τi}ni=1 , where each trajectory τi is represented as a sequence of state and action pairs {(st, at)}Tt=1 , IRL aims to infer the reward function of the expert policy, thus enabling the rationalization of its behaviors. Among many, the adversarial IRL algorithm (AIRL) casts IRL into a generative adversarial problem [Fu et al., 2018; Wang et al., 2024] with such discriminator as

where s′ ∼ P(s, a, ·) and ˜ r is a inferring reward function. The discriminator is trained to maximize the cross entropy between expert dataset and dataset induced by the policy via

where Tπ is the dataset induced by learning policy π. The generator of AIRL corresponds to π, which is trained to maximize the entropy-regularized reward function such as

2.2 Formulation of Pareto IRL

We specify the Pareto IRL problem which derives a Pareto policy set from strictly limited datasets. Consider M distinct expert datasets T∗ = {T ∗ i }Mi=1 where each expert dataset T ∗ i is collected from the optimal policy on some reward function rmo = ωT i r with a fixed preference ωi ∈ Ω. Furthermore, we assume that each dataset T ∗ i distinctly exhibits dominance on a particular reward function ri. In the following, we consider scenarios with two objectives (M =2), and later discuss the generalization for three or more objectives in Appendix A.4.

Given two distinct datasets, in the context of IRL, we refer to Pareto policy set derivation via IRL as Pareto IRL. Specifically, it aims at inferring a reward function ˜ r and learning a policy π for any preference ω from the strictly limited datasets T∗ . That is, when exploiting limited expert datasets in a multi-objective environment, we focus on establishing the Pareto policy set effectively upon unknown reward functions and preferences.

Figure 2 briefly illustrates the concept of Pareto IRL, where a self-driving task involves different preferences on two objectives, possibly conflicting, such as driving speed and energy efficiency. Consider two distinct expert datasets, where each expert has her own preference settings for the driving speed and energy efficiency objectives (e.g., T ∗ 1 and T ∗ 2 involve one dominant objective differently). While it is doable to restore a single useful policy individually from one given expert dataset, our work addresses the issue to generate a set of policies Π which can cover a wider range of preferences beyond given datasets. The policies are capable of rendering optimal compromise returns, denoted by dotted circles in the figure, and they allow users to immediately select the optimal solution according to their preference and situation.

For an MOMDP with a set of preference vectors ω ∈ Ω, a vector of reward functions r, and a preference function f in (1), Pareto policy set generation is to find a set of multiobjective policies such as

for M expert preference datasets {T ∗ i }Mi=1 . Rr(π) represents returns induced by policy π on reward function r. Neither a vector of true reward functions r is explicitly revealed, nor the rewards signals are annotated in the expert datasets, similar to conventional IRL scenarios.

3 Our Framework

To obtain a Pareto policy set from strictly limited datasets, we propose the ParIRL framework involving two learning phases: (i) recursive reward distance regularized IRL, (ii) distillation to a preference-conditioned model.

In the first phase, our approach begins with direct imitation of the given expert datasets, and then recursively finds neighboring policies that lie on the Pareto front. Specifically, we employ the reward distance regularized IRL method that incorporates reward distance regularization into the discriminator's objective to learn a robust multi-objective reward function. This regularized IRL ensures that the performance of the policy learned by the inferred multi-objective reward function remains within the bounds of the policy learned by the true reward function. By performing this iteratively, we achieve new useful policies that are not presented in the expert datasets, thus establishing a high-quality Pareto policy set.

In the second phase, we distill the Pareto policy set into a preference-conditioned diffusion model. The diffusion model encapsulates both preference-conditioned and unconditioned policies, each of which is associated with the preferencespecific knowledge (within a task) and the task-specific knowledge (across all preferences), respectively. Consequently, the unified policy model further enhances the Pareto policy set, rendering robust performance for arbitrary unseen preferences in a zero-shot manner. It also allows for efficient resource utilization with a single policy network.

3.1 Recursive Reward Distance Regularized IRL

Notation. We use superscripts g ∈ {1, ...,G} to denote recursive step and subscripts i ∈ {1, 2} to denote i-th multiobjective policies derived at each recursive step g. We consider two objectives cases in the following.

Individual IRL. As shown in the Figure 3 (i-1), the framework initiates with two separate IRL procedures, each dedicated to directly imitating one of the expert datasets. For this, we adopt AIRL [Fu et al., 2018] which uses the objectives (3) and (4) to infer reward functions {˜ r1 i }2i=1 and policies {π1 i }2i=1 from the individual expert dataset T ∗ i ∈ T∗ .

Reward distance regularized IRL. Subsequently, as shown in the Figure 3 (i-2), at each recursive step g ≥ 2, we derive new multi-objective reward functions {˜ rg i }2i=1 and respective policies {πg i }2i=1 that render beyond the given datasets. To do so, a straightforward approach might involve conducting IRL iteratively by blending the expert datasets at different ratios. However, as illustrated in Figure 4(a), the resulting policies tend to converge towards some weighted mean of datasets, rather than fully exploring non-dominant optimal actions beyond simple interpolation of given expert actions.

To address the problem, we present a reward distance regularized IRL on datasets Tg-1 = {T g-1 i }2i=1 collected from the policies derived at the previous step. Given a reward distance metric d(r, r′ ), we compute the distance between the newly derived reward function ˜ rg i and previously derived reward functions ˜rg-1 = [˜ rg-1 1 , ˜ rg-1 2 ]. Further, we define target distances as a vector ϵgi = [ϵgi,1 , ϵgi,2 ] to constrain each of the corresponding measured reward distances. Then, we define a reward distance regularization term as

where the subscripts i and j denote the newly derived reward function and the previously derived one, respectively. Finally, we incorporate (6) into the discriminator objective (3) as

where β is a hyperparameter. This allows the discriminator to optimize a multi-objective reward function for a specific target distance across datasets. The reward distance regularized IRL procedure is performed twice with different target distances, to derive policies adjacent to each of the previously derived policies. Furthermore, we fork the new regularized IRL procedure with the previously one (that is adjacent) to enhance the efficiency and robustness in learning.

The choice of the target distance is crucial, as the regret of a multi-objective policy is bounded under the reward distance (11). Thus, we set the sum of the target distances as small as possible. As the reward distance metrics satisfy the triangle inequality d(˜ rg-1 1 , ˜ rg-1 2 ) ≤ d(˜ rg i , ˜ rg-1 1 ) +

Algorithm 1 Recursive reward distance regularized IRL

Input: Expert preference datasets T∗ = {T ∗ 1 , T ∗ 2 },Π = ∅ 1: /* 1-st step: individual IRL procedures */ 2: for i ← 1, ..., 2 do 3: Obtain ˜ r1 i and π1 i using T ∗ i through (3) and (4) 4: Collect dataset T 1 i by executing π1 i 5: Execute T1 ←T1 ∪ T 1 i and Π←Π∪ {π1 i } 6: /* g-th step: regularized IRL procedure */ 7: for g ← 2, ...,G do 8: for i ← 1, ..., 2 do 9: Initialize ˜ rg i ← ˜ rg-1 i and πg i ← πg-1 i 10: Set ϵgi = [ϵgi,1 , ϵgi,2 ] based on (8) 11: while not converge do 12: Update ˜ rg i using ϵgi and regularized loss in (7) 13: Update πg i w.r.t. ˜ rg i using (4) 14: Collect dataset T g i by executing πg i 15: Execute Tg ← Tg ∪ T g i and Π←Π∪ πg i 16: return Π

d(˜ rg i , ˜ rg-1 2 ), we limit the sum of target distances to

In practice, we assign a small constant value for one of the target distances ϵgi,i , while the other is determined as ˆ ϵgi -ϵgi,i . By doing so, we are able to effectively derive a new policy that is adjacent to one of the previous policies.

Any reward distance metric that guarantees the regret bounds of policy can be used for ParIRL. In our implementation, we adopt EPIC, also known as equivalent policy invariant comparison pseudometric [Gleave et al., 2020], which quantitatively measures the distance between two reward functions. The learning procedure of recursive reward distance regularized IRL is summarized in Algorithm 1. In Appendix A.4, we discuss the generalization of reward distance regularization for more than two objectives (M ≥ 3).

3.2 Regret Bounds of Reward Distance Regularized Policy

We provide an analysis of the regret bounds of a reward distance regularized policy. Let ˜ r be our learned reward function and π∗ r be the optimal policy with respect to reward function r. Suppose that there exists a (ground truth) multi-objective reward function rmo = ωT r with preference ω = [ω1, ω2]. With the linearity of rmo, we obtain

Let D be the distribution over transitions S × A × S used to compute EPIC distance dϵ, and Dπ,t be the distribution over transitions on timestep t induced by policy π. Using Theorem A.16 in [Gleave et al., 2020], we derive that for α ≥ 2, (9) is bounded by the sum of individual regret bounds, i.e.,

where L is a constant, K = α/(1 - γ), ∆α(˜ r) = ∑Tt=0 γ tWα(Dπ∗˜ r ,t,D), and Wα is the relaxed Wasserstein distance [Villani, 2003]. Consequently, we obtain

As such, the regret bounds of our learned policy π on reward function ˜ r are represented by the regularization term based on EPIC along with the differences between the respective distributions of transitions generated by π∗˜ r and the distribution D used to compute EPIC distance. This ensures that the regret bounds of π can be directly optimized by using (7). In our implementation, instead of directly multiplying the preference ω to the loss function, we reformulate the preference into the target distance to balance the distance better. The details with proof can be found in Appendix A.2.

3.3 Preference-conditioned DiffusionModel

To further enhance the Pareto policy set Π obtained in the previous section, we leverage diffusion models [HO et al., 2020; Ho and Salimans, 2022], interpolating and extrapolating policies via distillation. We first systematically annotate Π with preferences ω ∈ Ω in an ascending order. We then train a diffusion-based policy model, which is conditioned on these preferences; i.e.,

where superscripts k ∼ [1, K] denote the denoising timestep, a0 (= a) is the original action, and ak-1 is a marginally denoised version of ak . The diffusion model is designed to predict the noise from a noisy input ak = √ ¯αk a + √1 - ¯αk η with a variance schedule parameter ¯αk and η ∼ N(0, I), i.e.

where {Tg }Gg=1 is the entire datasets collected by the policies in Π. Furthermore, we represent the model as a combination of preference-conditioned and unconditioned policies,

where δ is a guidance weight. The unconditioned policy encompasses general knowledge across the approximated Pareto policies, while the conditioned one guides the action according to the specific preference.

During sampling, the policy starts from a random noise and iteratively denoises it to obtain the executable action,

where αk and σk are variance schedule parameters. The diffusion model ˆπu allows for efficient resource utilization at runtime with a single policy network, and is capable of rendering robust performance for unseen preferences in a zeroshot manner. Consequently, it enhances the Pareto policy set in terms of Pareto front density , as illustrated in Figure 3 (ii).

4 Evaluation

4.1 Experiment Settings

Environments. For evaluation, we use (i) a multi-objective car environment (MO-Car), and several multi-objective variants of MuJoCo environments used in the MORL literature [Xu et al., 2020; Kyriakis et al., 2022] including (ii) MO-Swimmer, (iii) MO-Cheetah, (iv) MO-Ant, and (v) MOAntXY. For tradeoff objectives, the forward speed and the energy efficiency are used in (ii)-(iv), and the x-axis speed and the y-axis speed are used in (v). In these environments, similar to conventional IRL settings, reward signals are not used for training; they are used solely for evaluation.

Baselines. For comparison, we implement following imitation learning algorithms: 1) DiffBC [Pearce et al., 2023], an imitation learning method that uses a diffusion model for the policy, 2) BeT [Shafiullah et al., 2022], an imitation learning method that integrates action discretization into the transformer architecture, 3) GAIL [Ho and Ermon, 2016], an imitation learning method that imitates expert dataset via the generative adversarial framework, 4) AIRL [Fu et al., 2018], an IRL method that induces both the reward function and policy, 5) IQ-Learn [Garg et al., 2021], an IRL method that learns a q-function to represent both the reward function and policy, 6) DiffAIL [Wang et al., 2024], an IRL method that incorporates the diffusion loss to the discriminator's objective. To cover a wide range of different preferences, these baselines are conducted multiple times on differently augmented datasets, where each is a mixed dataset that integrates given datasets in the same ratio to a specific preference. We also include MORL [Xu et al., 2020] that uses explicit rewards from the environment, unlike IRL settings. It serves as Oracle (the upper bound of performance) in the comparison.

Metrics. For evaluation, we use several multi-objective metrics [Yang et al., 2019; Xu et al., 2020].

• Hypervolume metric (HV) represents the quality in the cumulative returns of a Pareto policy set. Let F be the Pareto frontier obtained from an approximated Pareto policy set for m objectives and R0 ∈ Rm be a reference point for each objective. Then, HV = ∫ 1H(F)(z)dz where H(F) = {z ∈ Rm | ∃R ∈ F : R0 ≤ z ≤ R}.
• Sparsity metric (SP) represents the density in the average return distance of the Pareto frontier. Let Fj(i) be the i-th value in a sorted list for the j-th objective. Then, SP = 1 |F|-1 ∑mj=1 ∑|F| i=1 (Fj (i) - Fj (i + 1))2 .

We also use a new metric designed for Pareto IRL.

• Coherence metric (CR) represents the monotonic improvement property of approximated policy set Π = {πi}i≤N generated by two expert datasets. Let policy list (π1, ..., πN) be sorted in ascending order by the expected return of the policies with respect to reward function r1, Then, CR = 2 N(N-1) ∑Ni=1 ∑Nj=i 1h(i,j) where h(i, j)=Rr1 (πi)≤Rr1 (πj ) and Rr2 (πi)≥Rr2 (πj ).

For HV and CR, higher is better, but for SP, lower is better.

4.2 Performance of Pareto Set Generation

Table 1 compares the performance in the evaluation metrics (HV, SP, CR) achieved by our framework (ParIRL, ParIRL+DU) and other baselines (DiffBC, BeT, GAIL, AIRL, IQ-Learn, DiffAIL). ParIRL is trained with the recursive reward distance regularized IRL, and ParIRL+DU is enhanced through the distillation. For the baselines, the size of a preference set (with different weights) is given equally to the number of policies derived via ParIRL. When calculating HV and SP, we exclude the out-of-order policies obtained from an algorithm with respect to preferences. As shown, our ParIRL and ParIRL+DU consistently yield the best performance for all environments, outperforming the most competitive baseline AIRL by 15.6% ∼ 23.7% higher HV, 80.4% ∼ 98.2% lower SP, and 21.7% ∼ 22.2% higher CR on average. Furthermore, we observe an average HV gap of 9.8% between ParIRL+DU and Oracle that uses the ground truth reward signals. This gap is expected, as existing IRL algorithms are also known to experience a performance drop compared to RL algorithms that directly use reward signals [Fu et al., 2018]. For the baselines, such performance degradation is more significant, showing an average drop of 26.9% in HV between AIRL and Oracle. ParIRL+DU improves the performance in HV over ParIRL by 7.0% on average, showing the distilled diffusion model achieves robustness on unseen preferences.

To verify the performance of ParIRL for three objectives case, we extend MO-Car to MO-Car* where the tradeoff objectives are the velocities on three different directions. Our ParIRL and ParIRL+DU show superiority in terms of HV, but sometimes show slightly lower performance in SP. I t is because the baselines tend to shrinks towards the lowperformance region, thus yielding lower SP. As CR is defined only for two objectives cases, CR for MO-Car* is not reported. The generalization of reward distance regularization for three or more objectives is discussed in Appendix A.4.

In this experiment, the baselines exhibit relatively low performance due to their primarily concentration on imitating the datasets, posing a challenge in generating policies that go beyond the limited datasets. Specifically, as DiffBC and BeT are designed to handle datasets with multiple modalities, they do not necessarily lead to the generation of novel actions. Meanwhile, the IRL baselines demonstrate relatively better performance, as they involve environment interactions. However, imitating from a merged dataset with specific ratio tends to converge towards the mean of existing actions, thus leading to sub-optimal performance.

4.3 Analysis

Pareto Visualization. Figure 4(a) depicts the Pareto policy set by our ParIRL and ParIRL+DU as well as the baselines (DiffBC, AIRL) for MO-AntXY. The baselines often produce the non-optimal solutions, specified by the dots in the lowperformance region. ParIRL+DU produces the most densely spread policies, which lie on the high-performance region.

Learning Efficiency. Figure 4(b) depicts the learning curves in HV for MO-AntXY over recursive steps. For baselines, we intentionally set the number of policies of the baselines equal to the number of policies derived through ParIRL for each step. The curves show the superiority of our recursive reward distance regularized IRL in generating the higher quality (HV) Pareto frontier. Furthermore, the recursive learning scheme significantly reduces the training time, requiring only 13% ∼ 25% of training timesteps compared to the IRL baselines. This is because ParIRL explores adjacent policies progressively by making explicit use of the previously derived policies to fork another regularized IRL procedure.

Ablation Studies. Table 2 provides an ablation study of ParIRL with respect to the reward distance metrics and recursive learning scheme. For this, we implement ParIRL/MSE and ParIRL/PSD, which use mean squared error (MSE) and Pearson distance (PSD) for reward distance measures, respectively; we also implement ParIRL/RC which represents ParIRL without recursive learning scheme. While MSE tends to compute the exact reward distance and PSD estimates the linear correlation between rewards, EPIC accounts for the reward function distance that is invariant to potential shaping [Gleave et al., 2020], thus making ParIRL optimize the regret bounds of a policy learned on an inferred reward function. Moreover, ParIRL/RC degrades compared to ParIRL, clarifying the benefit of our recursive learning scheme.

Table 3 shows the effect of our preference-conditioned diffusion model. ParIRL+BC denotes distillation using the naive BC algorithm. We test ParIRL+DU with varying guidance weights δ in (14), ranging from 0.0 to 1.8. The results indicate that ParIRL+DU improves by 6.42% at average over ParIRL+BC. Employing both unconditioned and conditioned policies (δ > 0) contributes to improved performance.

4.4 Case Study on Autonomous Driving

To verify the applicability of our framework, we conduct a case study with autonomous driving scenarios in the CARLA simulator [Dosovitskiy et al., 2017]. In Figure 5, the comfort mode agent drives slowly without switching lanes, while the sport mode agent accelerates and frequently switches lanes (indicated by dotted arrow) to overtake front vehicles (highlighted by dotted circle) ahead. Using the distinct datasets collected from these two different driving modes, ParIRL generates a set of diverse custom driving policies. Specifically, as depicted in the bottom of Figure 5, the closer the custom agent's behavior is to the sport mode, the more it tends to switch lanes (increasing from 0 to 2) and to drive at higher speeds with lower energy efficiency. The agent in custom mode-2 balances between the comfort and sport modes well, maintaining the moderate speed and changing lanes once.

5 RelatedWork

Multi-objective RL. In the RL literature, several multiobjective optimization methods were introduced, aiming at providing robust approximation of a Pareto policy set. [Yang et al., 2019; Xu et al., 2020] explored Pareto policy set approximation through reward scalarization in online settings, where reward signals are provided. Recently, [Zhu et al., 2023] proposed the Pareto decision transformer in offline settings, requiring a comprehensive dataset that covers all preferences. These prior works and ours share a similar goal to achieve a tradeoff-aware agent based on Pareto policy set approximation. However, different from the prior works, our work concentrates on practical situations with the strictly limited datasets and without any rewards from the environment. Inverse RL. To infer a reward function from datasets, IRL has been investigated along with adversarial schemes. [Fu et al., 2018] established the practical implementation of IRL based on the generative adversarial framework; which was further investigated by [Zeng et al., 2022; Garg et al., 2021; Wang et al., 2024]. Recently, [Kishikawa and Arai, 2021] introduced a multi-objective reward function recovery method, using a simple discrete grid-world environment. Contrarily, our ParIRL targets the approximation of a Pareto policy set. Instead of exploring the linear combinations of rewards, ParIRL employs the reward distance metric, and further, optimizes the performance lower bound of learned policies.

Reward Function Evaluation. Reward function evaluation is considered important in the RL literature, but was not fully investigated. [Gleave et al., 2020] first proposed the EPIC by which two reward functions are directly compared without policy optimization, and verified that the policy regret is bounded. This was extended by [Wulfe et al., 2022] for mitigating erroneous reward evaluation. However, those rarely investigated how to use such metrics for multi-objective learning. Our work is the first to conjugate reward function evaluation for Pareto policy set approximation in IRL settings.

6 Conclusion

We presented the ParIRL framework to induce a Pareto policy set from strictly limited datasets in terms of preference diversity. In ParIRL, the recursive IRL with the reward distance regularization is employed to achieve the Pareto policy set. The set is then distilled to the preference-conditioned diffusion policy, enabling robust policy adaptation to unseen preferences and resource efficient deployment. Our framework is different from the existing IRL approaches in that they only allow for imitating an individual policy from given datasets.

Acknowledgements

We would like to thank anonymous reviewers for their valuable comments. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-01045, 2022-0-00043, 2021-0-00875, 2020-001821, 2019-0-00421) and by the National Research Foundation of Korea (NRF) grant funded by the MSIT (No. RS2023-00213118) and by Samsung electronics.