Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Information

1 Introduction

How can we accelerate large language models (LLMs) without sacrificing accuracy? Recent LLMs have shown impressive performance across various tasks such as translation, code completion, and personal assistant [Brown et al., 2020; Zhang et al., 2022; Team et al., 2023; Chowdhery et al., 2023; Touvron et al., 2023a; Touvron et al., 2023b; Research et al., 2024]. The vast number of parameters in LLMs enables the remarkable capabilities, but slows down the inference speed of LLMs, limiting their practical deployments. Hence, accelerating LLMs is essential to fully leverage their benefits.

Pruning compresses and expedites neural network models via removing unnecessary parameters [Park et al., 2024b; Lee et al., 2021]. LLMs consist of multi-head attention (MHA) and multi-layer perceptron (MLP) sublayers stacked alternatingly. Sublayer pruning identifies unimportant sublayers, and removes them [Song et al., 2024; Men et al., 2024;

Zhong et al., 2024]. Note that sublayers are sequentially calculated unlike smaller units such as attention heads or neurons which are processed in parallel [Ma et al., 2023; Ashkboos et al., 2024]. Thus, sublayer pruning algorithms display better accuracy-speedup trade-off than finer-grained ones since removing parallelizable computations prevents GPUs from fully utilizing their computational capability.

The core objective of sublayer pruning is to accurately identify the sublayers to prune. However, existing sublayer pruning algorithms face challenges in preserving the accuracy of a pruned LLM since they fail to consider the characteristics of each sublayer. Figure 1 illustrates the results of different schemes to prune sublayers of an LLM. All the schemes prune less important sublayers to reduce the latency of the model by 300ms. Figure 1(a) summarizes the attributes of each sublayer, and (b) shows how each scheme evaluates the importance of each sublayer. Pairwise selection algorithms [Song et al., 2024; Men et al., 2024] prune MHA and MLP sublayers in pairs, to reduce the number of importance evaluations for identifying the targets. Those algorithms face challenges in preserving the accuracy of the pruned LLM since MHA sublayers induce less accuracy drop than MLP sublayers do when pruned. Individual selection algorithms [Zhong et al., 2024], which evaluate and prune each sublayer separately, still fail to achieve the best accuracy for the following two reasons. First, they overlook the latency difference between MHA and MLP sublayers. They prioritize MHA sublayers for removal due to their seemingly lower impact on accuracy. However, removing a single MLP sublayer provides equivalent latency reduction to removing two MHA sublayers, and they do not consider this effect for accelerating LLMs. Second, they disregard the fact that the accuracy drop caused by pruning is changed after tuning. They select sublayers solely based on the damage before tuning, failing to find sublayers that cause lower damage after tuning. To overcome these limitations, a pruning algorithm needs to consider both the latency and tunability of each sublayer.

We propose SPRINT (Sublayer PRuning wIth LateNcy and Tunability Information), an accurate sublayer pruning method for LLMs. SPRINT preserves the accuracy of the pruned models via accurately selecting a sublayer to remove. As shown in Figure 1(b), SPRINT considers the amount of latency reduction after pruning and accuracy drop after tuning. Moreover, we minimize the cost of SPRINT by 1) activation checkpointing to mitigate the repetitive computations, and 2) fast candidate selection to reduce the number of time-intensive tuning. We verify the effectiveness of SPRINT with extensive experiments. SPRINT accelerates Llama-2 and Llama-3 models, achieving up to 23.88%p higher accuracy on zero-shot commonsense reasoning benchmarks than baselines, and shows the best accuracy-speedup trade-off.

We summarize our main contributions as follows:

• Algorithm. We propose SPRINT, an accurate sublayer pruning method for LLMs. SPRINT accurately identifies less important sublayers in a model by the following four effective techniques: 1) latency-aware importance scoring, 2) tunability-aware sensitivity evaluation, 3) activation checkpointing, and 4) fast candidate selection.
• Experiments. We demonstrate that SPRINT achieves the state-of-the-art performance on commonsense reasoning benchmarks. SPRINT accelerates Llama-2 and Llama-3 models, achieving up to 23.88%p higher accuracy on zero-shot commonsense reasoning benchmarks, showing the most favorable accuracy-speedup trade-off.
• Analysis. We analyze the pruning patterns of the models pruned by SPRINT. We derive the findings that MLP and lower sublayers in an LLM serve as a critical component of LLM's capabilities, while MHA and upper sublayers contribute less to accuracy.

The rest of this paper is organized as follows. We first define LLM acceleration problem and provide backgrounds. We then propose SPRINT, our pruning method. After presenting experimental results, we conclude. Our source code is available at https://github.com/snudm-starlab/SPRINT.

2 Preliminary

2.1 ProblemDefinition

Problem 1 (LLM Acceleration Problem). Given a pretrained large language model F, a sample dataset D, and a latency upper bound τ, the problem is to find an accurate model ̂ F whose latency does not exceed τ . □

2.2 Transformer Architecture

Recent LLMs [Touvron et al., 2023b; Dubey et al., 2024] have a Transformer-based architecture [Vaswani et al., 2017] which consists of multi-head attention (MHA) and multilayer perceptron (MLP) sublayers stacked alternatingly. A Transformer model F with S sublayers refines an input sequence vector x as in Equation (1).

G is a generator module, E is an embedding look-up table, I represents a residual connection, and ◦ is the composition of functions. f (s) denotes the sth sublayer function which is either a self-attention network in an MHA sublayer when s is an odd number, or a feed-forward network in an MLP sublayer when s is an even number. We decompose f (s) into output projectionW (s) and remainder h(s) (·) as in Equation (2).

X(s) is an input matrix for the sth sublayer, where X(1) = E(x). Given an intermediate activation Z(s) = h(s) (X(s) ), f (s) is a linear transformation function with regard to Z(s) .

2.3 Sublayer Pruning

Sublayer pruning [Song et al., 2024; Men et al., 2024; Zhong et al., 2024] accelerates LLMs via pruning unnecessary sublayers in them. Sublayer pruning algorithms measure an importance score η for each sublayer, and eliminate those with the lowest scores. The importance scoring leverages sensitivity ζ which represents the performance difference of a model before and after the pruning. It is crucial to accurately score the importance of sublayers since the accuracy of the model heavily depends on which sublayers are pruned.

2.4 Fast In-compression Tuning

Pruning causes the accuracy loss by repeatedly removing parameters from a model. To mitigate the accuracy degradation, tuning the pruned model is essential so that the inference results are similar to those of the uncompressed model [Park et al., 2024a; Park et al., 2024b]. The objective of the tuning

Algorithm 1 Overall process of SPRINT

Input: An LLM F, a calibration dataset D, a latency constraint τ , number α of checkpoints, and number β of candidates

Output: A pruned LLM ̂ F

• 1: Initialize ̂ F as F
• 2: Initialize a dictionary Q of α checkpoints
• 3: Measure latencies T = {t(MHA) , t(MLP) }
• 4: while latency( ̂ F) > τ do
• 5: C,Q← fast candidate selection( ̂ F , D, T , β, Q) ▷ Select a set C of candidates (Section 3.4)
• 6: f ∗ ← tunability aware target selection(C, ̂ F, D, T ,Q) ▷ Find the least important sublayer f ∗ (Section 3.3)
• 7: Remove f ∗ from ̂ F and apply tuning
• 8: end while
• 9: return ̂ F

We propose SPRINT to address these challenges. The main ideas of SPRINT are as follows.

• I1. Latency-aware Importance Scoring. We consider the amount of latency reduction after pruning each sublayer to precisely identify unimportant sublayers.
• I2. Tunability-aware Sensitivity Evaluation. We accurately select sublayers to prune by measuring their sensitivity after tuning.
• I3. Avoiding Unnecessary Computations. We propose activation checkpointing and fast candidate selection to avoid the unnecessary computations.

Algorithm 1 and Figure 2 show the overall process of SPRINT. Given a pretrained LLM F and a latency constraint τ , SPRINT returns the pruned LLM ̂ F satisfying the latency constraint. SPRINT initializes α activation checkpoints Q to store reusable activations (line 2, details in Section 3.4). Then, SPRINT measures the amounts T of latency reduction resulting from the removal of sublayers (line 3) for latency-aware

is to align the output of the sth sublayer in the pruned model with that in the unpruned model, as described in Equation (3).

where ̂ X(s) , ̂ W (s) , and ̂Z(s) are the input, output projection, and intermediate representation of the sth sublayer in the pruned model, respectively. ̂ X(s+1) = ̂ X(s) + ̂ W (s) ̂Z(s) is the output of sth sublayer in the pruned model, while X(s+1) is the output of the sth sublayer in the unpruned model.

Note that solving Equation (3) does not require timeconsuming stochastic gradient descents as in previous works [Hu et al., 2022; Xu et al., 2024]. Instead, the equation is efficiently solved incorporating PyTorch's solver (torch.linalg.lstsq) once ̂ X(s) , ̂Z(s) , and X(s+1) are computed. Thus, fast in-compression tuning is computationally affordable for iterative pruning algorithms.

3 ProposedMethod

3.1 Overview

We address the following challenges to prune sublayers in LLMs minimizing the loss of accuracy.

• C1. Latency Difference of Sublayers. Existing works ignore the latency difference of MHA and MLP sublayers. How can we compare the importance of sublayers with different latencies to effectively accelerate LLMs?
• C2. Ignoring the Impact of Tuning. Existing works ignore the impact of tuning when selecting sublayers to prune, resulting in misselection. How can we incorporate the impact of tuning to accurately select sublayers to prune?
• C3. Expensive Computational Cost. Sublayer pruning is computationally expensive since it repeatedly measures the importance scores of all sublayers at each iteration. How can we enhance the efficiency of sublayer pruning?

importance scoring (details in Section 3.2). SPRINT repeats the iterative process of 1) scoring the importance of sublayers and 2) removing the least important one (lines 4-8) until the latency constraint is met. For each iteration, SPRINT first selects β candidate sublayers C to prune by scoring the pseudoimportance of each sublayer (line 5, see Section 3.4), and updates the checkpoints Q. SPRINT then scores the importance of each candidate in C with tunability-aware sensitivity evaluation (line 6, details in Section 3.3). Based on the importance scores, SPRINT prunes the least important sublayer f ∗ and tunes the remaining model (line 7). SPRINT returns the pruned model ̂ F (line 9) which satisfies the constraint.

3.2 Latency-aware Importance Scoring

Observation. How can we identify the most appropriate sublayer to prune for accelerating LLMs with minimal accuracy loss? Existing works [Song et al., 2024; Zhong et al., 2024; Men et al., 2024] measure the importance of sublayers without considering the latency difference of sublayers. Figure 3(a) shows the latencies of Llama-3 8B models after pruning different numbers of MHA and MLP sublayers. As shown in the figure, pruning an MHA sublayer yields over three times greater latency reduction than pruning an MLP sublayer; thus, it is beneficial to prune an MHA sublayer instead of an MLP sublayer if they cause the same damage. Therefore, it is essential to consider latency when selecting sublayers to prune.

Our solution. We incorporate the latencies of sublayers into our importance scoring process and assign lower importance scores for the sublayers with higher latencies to promote pruning the high-latency sublayers. The importance η(s) of the sth sublayer is defined as follows:

where ζ (s) is the sensitivity of the sth sublayer which approximates the amount of accuracy degradation after pruning it. t(s) is the amount of reduced latency through pruning the sth sublayer. Hence, η(s) reflects the cost-effectiveness of the sth sublayer in contributing to accuracy. As shown in Figure 3(a), sublayers of the same type offer almost the same degree of latency reduction; t(s) is either t(MHA) or t(MLP) depending on the sublayer's type. SPRINT measures t(MHA) and t(MLP) by comparing the latencies of unpruned and partially pruned models before starting its iterative pruning process.

3.3 Tunability-aware Sensitivity Evaluation

Observation. How can we accurately estimate the sensitivities of sublayers? Sublayer pruning algorithms measure sensitivities to approximately estimate the accuracy loss after pruning each sublayer. The lost accuracy is recovered via tuning, and each sublayer has a different capability for recovering. However, existing works [Men et al., 2024; Song et al., 2024; Zhong et al., 2024] ignore the effect of tuning when estimating the sensitivities. Figure 3(b) compares the ranking in accuracy degradation after pruning each sublayer before and after tuning. As shown in the figure, the index of the peak sublayer that evokes the lowest damage is changed after tuning. This indicates that pruning sublayers with the lowest sensitivity without considering the effect of tuning removes useful sublayers that exhibit low accuracy degradation after tuning. Therefore, sublayer pruning algorithms must account for the effect of tuning by prioritizing the removal of sublayers that result in minimal accuracy drop after tuning.

Our solution. SPRINT compares the activations of the original model and the pruned model after tuning to measure the sensitivities of sublayers. SPRINT exploits the fast incompression tuning [Park et al., 2024a] in Section 2.4 to efficiently incorporate tunability information into the sublayer selection process. Figure 3(c) shows the sensitivity measurement process of SPRINT for the sth sublayer which we call as the evaluation target. SPRINT finds the closest MLP sublayer (dth sublayer in the Figure 3(c)) above the evaluation target. After that, SPRINT measures the sensitivity ζ (s) of the sth sublayer by computing the normalized distance between outputs X(d+1) and ̂ X(d+1) t of the MLP sublayer before and after pruning, respectively, as in Equation (5). We exploit ̂ X(d+1) t obtained by fast in-compression tuning to take the tunability into account.

Note that we use outputs of only MLP sublayers, since an MLP sublayer has three times more number of parameters than that of an MHA sublayer, and thus has a stronger tuning capability than MHA. For fast in-compression tuning, SPRINT finds ̂ W (d) t that minimizes the distance between the output ̂ X(d+1) = (̂ X(d) + ̂ W (d) ̂Z(d) ) after pruning and the output X(d+1) before pruning, as in Equation (3). ̂ W (d) and ̂Z(d) are the weight of the out projection and the intermediate activation of the dth sublayer after pruning, respectively. Note that each row of ̂ W (d) forms an independent subproblem and we tune the weights in only c% of rows to avoid overfitting. We select the rows with abundant outliers, which represent larger activations than others, to maximize the impact of tuning with the given percentage of rows. We exploit the sum of the outlier-aware weight-wise scores [Sun et al., 2023; Yin et al., 2024] of weights in each row for selection. We save the tuned weights of each sublayer during the importance scoring process and apply the tuned weights corresponding to the pruning of the least important sublayer.

3.4 Avoiding Unnecessary Computations

Observation 1. How can we minimize the computation for measuring the sensitivities of sublayers? Pruning a sublayer does not affect the sensitivity of other sublayers whose closest upper MLP sublayers are located below the pruned one. For instance, as shown in Figure 4(a), ζ (1) to ζ (3) do not need to be reevaluated if f (5) is pruned in the previous iteration. However, naively computing the sensitivities of all sublayers at each iteration entails redundant computation for sensitivities which are already obtained in the previous iteration.

Our solution 1 (Activation Checkpointing). We propose activation checkpointing to prevent unnecessary recomputations. Before starting the iterations, SPRINT places checkpoints between sublayers. At each iteration, SPRINT stores the activations at the checkpoints. SPRINT reuses the sensitivities from the previous iteration for each sublayer f whose closest upper MLP sublayer is beneath the pruned sublayer, since the sensitivity of f is not changed after pruning. Then, SPRINT updates the sensitivities of the remaining sublayers using the stored activation. For example, assume we pruned f (5) at iteration 1, as shown in Figure 4(b). Note that SPRINT reuses ζ (1) to ζ (3) from the iteration 1 in the iteration 2 since they are not changed. Then, SPRINT loads the stored activation X(4) and starts updating the sensitivities from f (4) .

The checkpoints are uniformly placed, and the number α of checkpoints controls the trade-off between the memory usage and the time consumption during pruning.

Observation 2. How can we minimize the cost of sensitivity measurement? SPRINT performs an in-compression tuning to evaluate the sensitivity of each sublayer. Naively computing the sensitivities of all sublayers as illustrated in Figure 4(c-i) leads to excessive number of tunings at each iteration, making the sensitivity measurement too expensive.

Our solution 2 (Fast Candidate Selection). We propose fast candidate selection to selectively measure the sensitivities, minimizing the number of tuning in the sensitivity measurement. Instead of evaluating tunability-aware sensitivities for all sublayers, SPRINT first selects the candidates of the least sensitive sublayers swiftly without tuning. Then, SPRINT measures the tunability-aware sensitivity only for the candidate sublayers. For example, in Figure 4(c-ii), SPRINT first finds two candidates without tuning and then selects the sublayer to prune with the tunability-aware evaluation, reducing the number of tunings from 6 to 2. This process can be viewed as reducing the computational costs by approximately finding the sublayer to prune.

To instantly select the candidate sublayers, SPRINT measures the pseudo-importance of each sublayer.The pseudosensitivity ˜ ζ (s) of sth sublayer is the normalized distance between outputs X(d+1) and ̂ X(d+1) of the MLP sublayer before and after pruning, respectively. Note that ̂ X(d+1) is obtained without tuning, unlike ̂ X(d+1) t in Equation (5). The pseudo-importance ˜ η(s) of the sth sublayer is ˜ ζ (s) /t(s) , where t(s) is the latency of the sublayer. SPRINT selects β sublayers with the least pseudo-importance scores, where β is a hyperparameter representing the number of candidates. A higher β leads to the more accurate search for the sublayer to prune while requiring a higher computational cost.

4 Experiments

We perform experiments to answer the following questions.

• Q1. Accuracy. How accurate is SPRINT compared to baselines with the similar acceleration level?
• Q2. Pruning Efficiency. How fast does SPRINT prune LLMs compared to baselines?
• Q3. Ablation Study. Does each main idea of SPRINT contribute to the performance?
• Q4. Pruning Pattern Analysis. Which sublayers are important to maintain the accuracy of LLMs?

4.1 Experimental Setup

Setup. We use Llama-2 [Touvron et al., 2023b] and Llama3 [Dubey et al., 2024] model families as pruning targets. We randomly sample 128 token sequences of length 2048 from Wikitext2 [Merity et al., 2016] dataset for sensitivity measurement and tuning. We use NVIDIA A100 80GB GPU for all experiments. We report zero-shot reasoning accuracies on ARC-Challenge, ARC-Easy [Clark et al., 2018], BoolQ [Clark et al., 2019], HellaSwag [Zellers et al., 2019], and PIQA [Bisk et al., 2020] benchmarks. We measure the latencies to generate 512 tokens from 1024 input tokens [Lin et al., 2024b] and report speedups of pruned models.

Baselines. We compare SPRINT with three sublayer pruning algorithms: ShortGPT [Men et al., 2024], SLEB [Song et al., 2024], and BlockPruner [Zhong et al., 2024]. ShortGPT and SLEB utilize the pairwise selection scheme while BlockPruner selects pruning targets individually as illustrated in Figure 1. We also include four fine-grained pruning algorithms as baselines for comprehensive analysis: SparseGPT [Frantar and Alistarh, 2023], Wanda [Sun et

• al., 2023], SliceGPT [Ashkboos et al., 2024], and LLMPruner [Ma et al., 2023]. These algorithms prune LLMs in parallelizable units smaller than sublayers such as channels or attention heads, resulting in insufficiently accelerated models.

Hyperparameters. We use random seeds from 0 to 2 and report the average values. We set the checkpointing hyperparameter α to 8 and the candidate hyperparameter β to 5 for all models.

4.2 Accuracy

Figure 5 shows the accuracies of SPRINT and baselines across various inference speedup settings. As shown in the figure, SPRINT achieves the best trade-off curve among all the methods, and significantly outperforms the competitors under 40% speedup setting with the maximum accuracy gap of 23.88%p. It is notable that SPRINT achieves the best performance among all the algorithms on challenging multiplechoice benchmarks, such as ARC-Challenge, ARC-Easy, and HellaSwag, across all models of various sizes.

4.3 Pruning Efficiency

Figure 6 visualizes compression time and average accuracy on five commonsense reasoning tasks of the pruned models generated by SPRINT and competitors with 40% speedup. We estimate the pruning time of BlockPruner for 70B models since it takes more than a week to prune them (see Supplementary Material for details). Brown dashed lines in Figure 6 denote the estimated pruning time.

Note that SPRINT offers the best accuracy-pruning time trade-off across all models. SPRINT prunes the LLM up to

40.01 times faster and achieves up to 6.21%p higher accuracy than BlockPruner. SPRINT further outperforms other methods with similar pruning times by larger margins.

4.4 Ablation Study

To prove the effectiveness of each main idea, we compare SPRINT with its three variants: SPRINT-l, SPRINT-t, and SPRINT-e. SPRINT-l is SPRINT without latency-aware importance scoring; it prunes the sublayer with the lowest sensitivity without considering the latency. SPRINT-t is SPRINT without tunability-aware sensitivity evaluation, measuring the sensitivity of each sublayer by directly comparing its output before and after pruning. SPRINT-e is SPRINT without activation checkpointing and fast candidate selection, calculating the importance scores of all sublayers without storing activations for each step of iterative pruning process.

Table 1 summarizes the performance of SPRINT and its variants when accelerating the Llama-3 8B by 40%. SPRINT and SPRINT-e outperform other variants in terms of accuracy, proving that both latency-aware importance scoring and tunability-aware sensitivity evaluation contribute to the accuracy. SPRINT is twice faster than SPRINT-e, confirming the effectiveness of activation checkpointing and fast candidate selection for the pruning cost. In summary, all three main ideas of SPRINT contributes to the performance.

4.5 Pruning Pattern Analysis

Figure 7 depicts the pruning patterns of SPRINT on Llama models. Orange, blue, and gray squares represent MHA, MLP, and pruned sublayers, respectively. We observe two main patterns related to the type and position of sublayers. First, SPRINT prunes more MHA sublayers than MLP sublayers in general; MLP sublayers are pruned only in 70B models. Second, SPRINT prunes sublayers located mainly in the upper-middle parts of models. These two patterns show that MLP sublayers and sublayers located near the bottom significantly contribute to the capability of LLMs. Moreover, as more extensive pruning occurs in the upper layers as observed by the second pattern, sensitivities of lower layers often do not need to be updated. Thus, our proposed activation checkpointing aligns well with the LLM's characteristics.

5 RelatedWork

We review techniques for accelerating LLMs: quantization, knowledge distillation, and dynamic inference. Quantization [Piao et al., 2022; Frantar et al., 2023; Lee et al., 2023; Lin et al., 2024a; Shao et al., 2024; Kim et al., 2025b; Kim et al., 2025a] reduces the bit-width of weights and activations in them; it accelerates computation by leveraging hardwares designed for low-bit operations. Quantization is compatible with pruning; unifying both methods achieves greater acceleration [Frantar and Alistarh, 2023].

Knowledge distillation (KD) [Ko et al., 2024; Yoo et al., 2019; Cho and Kang, 2022; Liu et al., 2024; Kim et al., 2021; Jeon et al., 2023; Jang et al., 2023] improves the accuracy of compressed models by transferring knowledge from uncompressed models. KD is also compatible with pruning, by effectively compensating for the error induced by pruning.

Dynamic inference [Schuster et al., 2022; Varshney et al., 2023; Raposo et al., 2024; Fu et al., 2024] accelerates LLMs via dynamically adjusting the amount of computations based on the input. Dynamic inference exhibits minimal accuracy degradation since it determines the amount of computations according to the importance of inputs. However, they have a significant drawback in that their efficiency is diminished when multiple inputs requiring different computations are fed [Song et al., 2024]. In contrast, sublayer pruning consistently preserves efficiency regardless of the number of inputs.

6 Conclusion

We propose SPRINT, an accurate sublayer pruning method for accelerating LLMs. SPRINT addresses the inaccurate sublayer selection problem of existing sublayer pruning methods by factoring in latency and tunability information. We propose activation checkpointing and fast candidate selection techniques to shorten the running time of SPRINT. We demonstrate that SPRINT achieves the best accuracy-speedup trade-off when pruning Llama-2 and Llama-3 models.

Contribution Statement

Seungcheol Park, Sojin Lee, and Jongjin Kim contributed equally to this work as co-first authors. Jinsik Lee and Hyunjik Jo provided helpful discussion for developing our method. U Kang supervised the project and carefully reviewed the manuscript.

Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No.RS-2020II200894, Flexible and Efficient Model Compression Method for Various Applications and Environments], [No.RS-2021II211343, Artificial Intelligence Graduate School Program (Seoul National University)], and [No.RS-2021-II212068, Artificial Intelligence Innovation Hub (Artificial Intelligence Institute, Seoul National University)]. This work was supported by Youlchon Foundation. The Institute of Engineering Research at Seoul National University provided research facilities for this work. The ICT at Seoul National University provides research facilities for this study. U Kang is the corresponding author. This work was improved by the helpful input and collaboration of researchers from LG AI Research.