Efficient Dynamic Ensembling for Multiple LLM Experts

1 Introduction

Large Language Models (LLMs) such as LLaMA [Touvron et al., 2023] and GPT-3.5 [Achiam et al., 2023] have exhibited remarkable performance across diverse tasks [Stark et al.,

2024; Hu et al., 2025; Wang et al., 2025], such as embodied intelligence [Mu et al., 2023]. However, their variations in architectures, model sizes, and training data result in distinct strengths and weaknesses in different tasks [Jiang et al., 2023; Lu et al., 2024]. Although training a larger LLM with more comprehensive data is possible to maintain excellent performance in all tasks, the cost is significant and often impractical. Consequently, it is crucial to assemble LLMs to enhance their generalization while minimizing the consumption of computational resources in practical applications.

Unfortunately, assembling knowledge of LLMs with limited computing cost is difficult partly for the following reasons. 1) Complex knowledge integration: Each LLM is typically trained on different datasets, which may include varying levels of quality, diversity, and bias. Harmonizing these LLMs requires aligning their understanding with knowledge bases, posing challenges to integration [Jiang et al., 2023; Lu et al., 2024]. 2) High computational complexity: LLMs are computationally intensive, requiring significant resources for inference. Combining LLMs increases this complexity, potentially necessitating more efficient algorithms to manage [Sheng et al., 2023; Wan et al., 2024].

Recently, LLM ensemble has emerged as a rapidly progressing research area to leverage the diverse strengths of multiple LLMs. Based on their strategies, most existing methods can be broadly divided into four types. The Mixtureof-Experts methods [Jiang et al., 2024; Tang et al., 2024] use a router network to select and activate a subset of experts to aggregate diverse expertise (see Figure 1 (a)), but they often require retraining, cannot integrate non-homologous LLM experts, and fail to leverage complementary knowledge among experts. The parameter merging methods [Yu et al., 2024; Matena and Raffel, 2022] merge the parameters of homologous LLMs into a single unified model but cannot assemble non-homologous LLMs. The rule-based methods [Dong et al., 2024; Du et al., 2023] assemble the advantages of LLM by manually designing task-specific roles or a fixed order. However, such a static setting makes it difficult for the integration to generalize in various domains. To avoid these issues, agent-based methods [Jiang et al., 2023] train an agent to integrate non-homologous LLMs with the various strengths, making them adaptable to various scenarios.

Despite the recent success of agent-based LLM ensemble methods in outperforming the best one among the LLMs across a wide range of downstream tasks, these methods still face certain limitations. Firstly, a common drawback in most existing methods [Jiang et al., 2023; Liu et al., 2024] is that they integrate LLMs based on the final outputs of all LLMs. These methods require prohibitively high computational costs to run all the LLM candidates and result in inefficient utilization of inference resources. Therefore, some easy samples that could be simply addressed by a single LLM have to run all the LLMs at expensive costs. For example, PairRanker [Jiang et al., 2023] requires significantly more parameters (> 117B) than a single LLM in inference, potentially leading to unbearable resource wastage. Secondly, while some agent-based methods implement a classifier to accomplish the integration by selecting only one LLM candidate at each time with very little inference cost (see Figure 1(b), left), e.g., ZOOTER [Lu et al., 2024], their performance is limited by the fact that they do not leverage the complementary knowledge among LLMs.

To address the above limitations, we propose a novel Dynamic Ensemble Reasoning paradigm for integrating the strengths of multiple LLM experts, called DER. Given that LLMs are always trained on diverse datasets, we hypothesize that they possess complementary knowledge, which can be sequentially assembled. Nevertheless, the exponential growth in possible combinations of routes renders it impractical to address this challenge through classification tasks alone. To overcome this, we view knowledge transfer as a sequential decision process. Specifically, we model the LLM ensemble as a Markov Decision Process (MDP), where a DER-Agent dynamically requests contributions from LLMs and transfers this knowledge to subsequent LLM candidates (see Figure 1(b), right). Moreover, we develop a reward function to train the DER-Agent to select optimal answering routes based on input questions, aiming to maximize performance while minimizing computational resources. Additionally, we introduce a Knowledge Transfer Prompt (KTP) to facilitate effective knowledge transfer among LLMs.

We summarize our main contributions as follows:

• We propose Dynamic Ensemble Reasoning (DER) for the LLM ensemble, modeling it as a Markov Decision Process (MDP) for efficient sequential knowledge

transfer. This approach dynamically selects optimal answering routes, integrating complementary knowledge from various LLMs to maximize performance with minimal computational resources. Experiments show DER integrates the strengths of different LLMs, achieving nearly a 7-fold parameter reduction compared to ensemble methods using the outputs of all LLMs.

• We introduce a Knowledge Transfer Prompt (KTP) to facilitate efficient knowledge sharing among LLMs and develop a reward function to train the DER-Agent. This ensures the DER-Agent leverages expert knowledge from previous LLMs, optimizing task performance while significantly reducing computational costs. Experiments show that more than 9% improvement is achieved on BARTScore using KTP and our reward function.

2 RelatedWorks

Mixture-of-Experts Methods integrate the knowledge of LLMs by selecting and activating a subset of experts through a router network. Jiang et al. [Jiang et al., 2024] propose a Sparse Mixture of Experts language model, which aggregates the knowledge of diverse experts by selecting two experts at each layer of the routing network to process the current state and combining their outputs. Tang et al. [Tang et al., 2024] propose the Weight-Ensemble MoE, which achieves assembling different expert knowledge by training a router to select and merge different LLM expert parameters. However, these methods often require re-fine-tuning of the MoE model, and it is difficult for experts to utilize the complementary knowledge of other experts.

Parameter Merging Methods assemble the advantages of LLMs by merging the parameters of multiple homologous LLMs into a single model. Matena and Raffel [Matena and Raffel, 2022] propose the "Fisher merging", which merges the parameters of models with the same structure and initialization, achieving the ability to assemble different LLMs. Yu et al. [Yu et al., 2024] introduce an operation called DARE to sparsify delta parameters of multiple homologous LLMs, and subsequently merge them into a single model by parameter averaging, realizing the ability to assemble LLMs. However, these methods cannot assemble non-homologous LLMs.

Rule-based Methods assemble LLMs by manually setting roles or fixed orders for specific tasks. Du et al. [Du et al.,

2023] propose LLM-debate, in which multiple LLMs get a consensus in multiple rounds of common debate. Dong et al. [Dong et al., 2024] use three LLMs, set up as analysts, coders, and testers, to collaboratively develop software in sequential execution. Despite the advantages of these to quickly assemble LLMs, this static setup is hard to generalize.

Agent-based Methods dynamically integrate the advantages of different non-homologous LLMs by training an agent and can be used in various scenarios. Jiang et al. [Jiang et al., 2023] propose the LLMs ensemble framework LLMBLENDER in expectation of consistently superior performance, which selects the TOP-K responses by PairRanker and mixes them to generate final outputs using the GENFUSER. Liu et al. [Liu et al., 2024] propose DyLAN, which enables the LLM ensemble through multiple rounds of interaction and an early stopping mechanism. However, they are assembled with the outputs of all candidate LLMs, which tends to waste significant computational resources. Lu et al. [Lu et al., 2024] propose ZOOTER, a reward-guided routing method that can directly and accurately assign each prompt to LLMs with expertise, using a small computational overhead. However, it is limited in answering performance because it does not utilize complementary knowledge between LLMs.

3 ProposedMethods

3.1 ProblemDefinition andMotivations

Problem Definition. Given an input question set {x1, . . . , xK} with K questions, the LLM ensemble aims to aggregate the strengths of different LLMs (experts) to consistently achieve superior performance. Specifically, given an input question x and a model pool {M1, . . . ,MN} with N LLMs, we aim to select k models to answer the question together to improve the quality of the answer.

Motivations. Existing methods [Lu et al., 2024; Yao et al., 2023] train an agent to select the most suitable LLM to answer each question individually (see Figure 1 (b), left). Although these methods can improve the answering performance, their performance is limited due to the underutilization of the different knowledge contained in LLMs. To overcome this issue, we aim to develop an effective knowledge aggregation strategy to achieve superior performance. Intuitively, leveraging knowledge transfer from one LLM to another can compensate for individual LLM shortcomings, fostering a collaborative integration of diverse knowledge. Moreover, there exists an optimal knowledge transfer route for each question. Unfortunately, the number of possible route combinations is as large as ∑mk=1 N k , where N is the number of LLMs, k is the route length, and m is the maximum route length. In this sense, finding an optimal route for each sample poses a severe challenge.

To address the above issue, we naturally model the knowledge transfer as a sequential decision process, given its sequential nature. To this end, we require selecting certain models consecutively from the pool to answer the question, with each model able to refer to the responses provided by its predecessors (see Figure 1 (b), right). In this way, we derive an ultimate response from the terminal model. Our goal lies in identifying an optimal pathway of model execution Route∗ :=[Mi⇒Mj⇒· · ·⇒Mk] to enhance the quality of the final answer with modest computational cost. As shown in Figure 2, we formulate the LLM ensemble as a Markov Decision Process (MDP) and train a DER-Agent to select an optimal answering route. At each time step t, the DER-Agent determines the next LLM to generate a response based on the input question and the current answer (initially absent). This procedure utilizes the Knowledge Transfer Prompt (KTP), facilitating the LLM to construct an answer that progressively integrates insights from the previous LLM. The newly formed answer serves as the next state, enabling the ongoing knowledge transfer. This process continues looping until the answer is satisfactory or the maximum trajectory length is reached.

3.2 Dynamic Ensemble as anMDP

We seek a general DER-Agent to select an optimal sequential execution route of LLMs for the input question x, obtaining the highest performance with the least possible computational resources. To this end, we formulate the selection of the sequential execution route of LLMs as Markov Decision Process (MDP) [Van Otterlo and Wiering, 2012]: < S, A, T , R, π >. The state space of the environment is S and the action space of the agent is A. At time step t, the agent takes the state st ∈ S as input and performs an action at ∈ A through the policy network π : S ×A → [0, 1]. The environment changes to the next state st+1 = T (st, at) according to the transition function T and a reward rt = R(st, at) is received with reward function R. The MDP is detailed as:

States S is a set of states which describe the environment. At time step t, the state consists of a question-answer pair: st = [Q : x,A : ˆ yt-1], where x is the input question and ˆ yt-1 is the answer from the selected LLM at time step t - 1 (the answer is None initially). Thus, the agent judges the quality of the current answer by the question-answer pair and predicts which next LLM to answer.

Actions A is a discrete set of actions the agent takes. The action space A = {1, 2, . . . ,N} is the index of each LLM. At time step t, the agent gives the action at ∈ A based on the state st to select a next LLMMat from the model pool {M1, . . . ,MN}.

Transition T (S, A) is a function T : S × A → S which maps a state st into a new state st+1. When the BERTScore P (ˆ yT ) for the answer ˆ yT reaches a manually-set threshold p0 or the maximum trajectory length Tmax is reached, this episode will be terminated and sT+1 is None. Otherwise, the selected LLM at time step t will answer the question x concerning the existing answer ˆ yt-1:

KTP(·) is the Knowledge Transfer Prompt (KTP) that we designed to stitch together the question and answer from the previous LLM Mat-1 and to promote the current LLM to answer the question x concerning the previous answer ˆ yt-1. The KTP is detailed in subsection 3.4.

Rewards R(S,A) is the reward function. In the LLM ensemble task, the reward can be considered as an evaluation of the quality of the answer ˆ y for the selected LLMMat . The details of the reward function are given in the subsection 3.3.

Policy πθ(a | s) : A× S → [0, 1] describes the behaviors of the agent. During the training process, the agent takes the current state st as input and outputs a probability distribution for each possible action at ∈ A = {1, 2, . . . , N}:

where fθ (st) is the output vector of the policy network with input st, and i denotes the index of the action. θ is the learnable parameters of the policy network.

The general diagram of the proposed DER is shown in Figure 2. Given an input question x, the DER is initialized with state s0 = [Q : x,A : None]. The agent takes s0 as input and gives an action a0 so that an LLMMa0 is selected to answer the question with ˆ y0. And then the reward r0 is calculated for agent optimization based on the answer's quality and computational resources. The state is updated to s1 = [Q : x,A : ˆ y0] with the answer ˆ y0. The above process will continue until the BERTScore P(ˆ yt) exceeds the threshold p0 or the maximum trajectory length is reached. Finally, the last answer ˆ yT is obtained, which is high-quality thanks to the knowledge transfer among LLMs.

3.3 Reward Function Design

In our designed MDP, the reward function is defined to reflect three aspects: the quality of the answer provided by the selected LLM, the increment quality of the answer, and the computational resources:

where P(·) is the BERTScore, which is commonly used to evaluate the quality of generated text and its high correlation with human judgment [Zhang et al., 2019]. The ˆ yt is the output answer of the selected LLMMat , C(·) is the computation cost of Mat , ∆P (ˆ y) = P(ˆ yt)-P (ˆ yt-1) is the increment of the BERTScore of the answer from t - 1 to t, and α, β is the coefficient to determine the ratio of computation cost and the increment of the score, respectively. In addition, we add additional rewards or penalties to allow the agent to complete the generation of routes in limited steps. Thus, the complete reward follows:

where p0 is the threshold of the BERTScore for which an environment gives an end, Tmax is the maximum step size, and γ is the bias for extra rewards or penalties. Note that in the testing phase P(ˆ yt) ≥ p0 or P(ˆ yt) < p0 is judged by one of our trained Terminator, which is a binary classifier.

3.4 Knowledge Transfer Prompt

We develop the Knowledge Transfer Prompt (KTP) to facilitate effective knowledge transfer among LLMs. The proposed KTP expects that the current LLM effectively uses the answer (knowledge) of the previous LLM ˆ yt-1 without being limited by it, to improve the performance of generating a better answer to the input x. To ensure that LLM experts follow the knowledge transfer settings, we introduce a role-playing mechanism [Kong et al., 2024] into the KTP as:

Knowledge Transfer Prompt:

[x] \n There is an answer to the question from another student: \n [ˆ yt-1] \n Using another student's answer as additional advice, you need to give a more satisfactory answer directly. DO NOT mention other students.

First, we treat the previous LLM's answer ˆ yt-1 as the "student's answer", thereby avoiding the overriding influence of

Algorithm 1 PPO Training for DER

Require: Prompt-Answer dataset D, DER-Agent πθ, critic Vϕ, replay buffer B, buffer size M, training iterations m,

actor θ and critic ϕ. 1: Initialize B, parameters θ and ϕ. 2: while Not converged do 3: for (question xi, answer yi) in D do 4: Collect a trajectory τ using old πθold and Vϕold , and put it into B. 5: if |B| =M then 6: for iteration = 1, 2, . . . ,M do 7: Uniformly sample τ ∈ B. 8: Calculate J (θ) via Eqn. (5). 9: Update θ to maximize J (θ). 10: Calculate TD error δt via Eqn. (6). 11: Update ϕ to minimize TD error δt. 12: end for 13: Empty the replay buffer B. 14: Update θold ← θ. 15: Update ϕold ← ϕ. 16: end if 17: end for 18: end while

where G(τ) is the total return of the trajectory τ = {st, at, rt, vt,Aπθold (st, at)} obtained by πθold and Vϕold , ρ = πθ(at|st) πθold (at|st) is the ratio of the probability of action at given by πθ and πθold for state st, and ϵ is a hyperparameter, usually set to 0.2 [Schulman et al., 2017]. The operation clip (ρ, 1- ϵ, 1+ ϵ) constrains ρ to the range [1- ϵ, 1+ ϵ], and Aπθold (st, at) = rt - Vϕold (st) is the advantage at t.

Training. The overview of the optimization process is presented in Algorithm 1. Specifically, given a Prompt-Answer

the answer content of the predecessor. We then ask the current LLM to refer to the "student's answer" to give a more satisfactory answer ˆ yt to question x via "you need to give a more satisfactory answer". In addition, the proposed KTP avoids LLM outputting role-playing messages and irrelevant information by "DO NOT mention other students".

3.5 Learning with Proximal Policy Optimization

We use the Proximal Policy Optimization (PPO) [Schulman et al., 2017] to optimize the parameters θ of the DER-Agent (policy) due to the stability and sample-efficiency as:

Actor. The actor (DER-Agent) is trained for LLM selection according to the question-answer pair. To enhance the ability of natural language understanding, we employ a pretrained OPT-125M [Zhang et al., 2022] with two Linear layers connected to the last hidden layer, where the output dim is the number of Candidate LLMs N. In the off-policy algorithm, the old policy πθold with old parameters θold is used to collect trajectories with the environment, while the policy πθ is updated using trajectories collected by πθold .

Critic. The critic Vϕ(s) is used to estimate the expected return vt of the state st and calculate the advantage, which aids the actor in learning more efficiently and stably. The critic is composed of an OPT-125M with two Linear layers. But the output dim is set to one. Besides, the old critic Vϕold (s) is used to collect trajectories, and the new critic Vϕ(s) is updated using the collected trajectories.

Learning Objectives. The goal of the learning is to maximize the expected long-term return J (θ):

dataset D, we use πθold and Vϕold to interact with the environment to collect a trajectory τ and compute the advantage Aπθold (st, at). We then put τ into the reply buffer B. When a certain number of trajectories (such as M) have been collected, they are used to train the actor and critic. In particular, we first uniformly sample sequences from the reply buffer B, and then calculate the expected long-term return J (θ) so that to optimize the parameters of the policy πθ. Besides, the Temporal Difference (TD) error δt is also calculated to optimize the parameters of the critic Vϕ:

where Gt is the total expected return starting from time step t. After training a certain number of times using the samples in the existing reply buffer B, we clear the reply buffer and update the parameters of the old policy πθold and critic Vϕold . Then we repeat the above operation until convergence.

4 Experiment

Datasets and LLM experts. Following the settings of PairRanker [Jiang et al., 2023], we use MixInstruct as the benchmark. In addition, we use the GSM8K [Cobbe et al., 2021] and Multidomain we constructed (see Appendix 2.3) for further evaluation. We select eleven LLM experts for the ensemble task: Open Assistant [LAION-AI, 2023], Vicuna [Chiang et al., 2023], Alpaca [Taori et al., 2023], Baize [Xu et al., 2023], MOSS [Sun et al., 2023], ChatGLM [Du et al., 2022], Koala [Geng et al., 2023], Dolly2 [Conover et al., 2023], Mosaic MPT [Team and others, 2023], StableLM [Stability-AI, 2023] and FLAN-T5 [Chung et al., 2024].

Baseline Methods. 1) Classifier-OPT (Appendix 2.5), 2) PairRanker [Jiang et al., 2023], 3) LLM-debate [Du et al., 2023], 4) ReConcile [Chen et al., 2024], 5) sampling-voting [junyou li et al., 2024], and 6) DyLAN [Liu et al., 2024].

4.1 Comparison Experiments

We compare our proposed DER, eleven LLM experts, ChatGPT, and state-of-the-art (SOTA) LLM ensemble methods to demonstrate our method superior performance. We conduct experiments on a variety of downstream tasks, including QA task (see Table 1), mathematical reasoning task (see Table 2), and multi-domain QA task (see Appendix 3.2).

DER is consistently better than single LLM. From Table 1, DER achieves better performance than a single LLM. Crucially, the DER on BARTScore is improved by 8.7% compared to Vicuna. In addition, DER reaches 98% of the ChatGPT performance on the BERTScore, while the inference parameters are only 10% of those of ChatGPT. We conclude that DER achieves better performance than the single LLM due to its ability to aggregate the complementary knowledge of diverse LLMs through knowledge transfer.

Trade-offs between performance and computational resources. As shown in Table 1, the proposed DER achieves better performance than the SOTA methods with a little cost. Specifically, the proposed DER reduces the computational overhead of LLMs inference by about 85% (117B → 17B) compared to PairRanker, while DER increases the BERTScore by about 2.7% (73.03 → 75.00) compared to PairRanker. In addition, our method is also the best performer on the GPT-Rank metric. The main reason is that the reward function of our design requires DER-Agent to aggregate the strengths of diverse LLMs on as few resources as possible while setting the maximum route length.

Superior performance on mathematical reasoning. From Table 2, DER outperforms SOTA methods on the mathematical reasoning task. Specifically, compared to PairRanker, DER reduces the inference cost by more than 77%

and increases the Accuracy by about 16%. We conclude that DER effectively aggregates the strengths of LLMs to generate better mathematical reasoning results after the knowledge transmission through them.

4.2 Ablation Studies

Effectiveness of Knowledge Transfer Prompt. We compare DER and DER (w/o KTP) to demonstrate the effectiveness of the KTP. From Table 3, the KTP effectively improves the performance of DER. Specifically, there is a 4.6% increase in BARTScore after using the KTP. This is strong support for the fact that the use of a role-playing prompt allows the LLM to leverage the knowledge of the previous LLM to produce a more satisfactory output [Kong et al., 2024].

Effectiveness of LLM ensemble using MDP. As shown in Table 3, compared with the randomly generated route method, the DER outperforms the randomly generated route method by 2.31 in BERTScore. The primary reason is that by modeling the LLM ensemble as an MDP, the trained DERAgent chooses an appropriate LLM based on the answer of the previous LLM to continue the answering.

Effects of the reward function R. We study the effects of β∆P(ˆ y) and γ in reward function on the performance of DER, by conditioning β = 0 or γ = 0 in Equation (3) and (4). As shown in Table 4, whenever β = 0 or γ = 0 both lead to about 3% performance degradation of DER on the BARTScore metric. This provides strong support for adding additional rewards/ penalties to our reward function (see Section 3.3) for answering the performance increase and whether or not completing the task within a finite step size improves DER's performance on the LLM ensemble task.

Effects of the Terminator. We study the effect of Terminator on DER at the maximum reachable step Tmax = 4. As shown in Table 5, DER with Terminator reduces the parameters by 39% while also improving the BERTScore by 1.68. This is because the route reaches the endpoint when the Terminator believes that the output answer is greater than or equal to the threshold p0 = 0.73 on the BERTScore metric (selection of p0 see Appendix 4.2). Therefore, using Terminator reduces the average computational resources required for inference of LLMs and results in better performance.

Effects of the maximum reachable step Tmax. As shown in Table 5, as the maximum reachable step increases, the average parameter of the LLMs inference also increases, but it does not make a significant difference to the performance. Therefore, we set the Tmax to 4. In addition, we count the length of answer routes generated by DER for all samples. As shown in Table 6, a substantial majority, i.e., 55.7% of DER's answer routes on MixInstruct have a length T ≤ 2, validating the practical efficiency of DER.

Effects of the number of experts selected for each state. We conduct experiments where two experts are selected in each state to demonstrate that selecting one expert per state is optimal. From Table 7, DER produces better-generated answers by selecting an expert per state, with a 65% reduction in inference parameters. The reason is that the knowledge gained from the non-optimal experts selected in each state affects the quality of the expert's answer in the next state.

4.3 Eaxmple of DER

From Appendix 5, the DER-Agent selects a suitable LLM to continue the answering task as the sequence decision process proceeds, and the answering performance ends up being the best. Notably, the next LLM expert tends to leverage the knowledge of the previous LLM expert to improve the output answer. Thus, DER assembles complementary knowledge among LLMs to obtain better output answers.

5 Conclusion

In this paper, we propose a novel dynamic ensemble reasoning method, called DER, to integrate the strengths of LLM experts dynamically conditioned on dynamic inputs. Specifically, we model the LLM ensemble as an MDP, where a DERAgent takes dynamic inputs, sequentially asks an LLM candidate to provide knowledge, and passes the knowledge to subsequent LLM candidates. We devise a reward function to train a DER-Agent to select an optimal answering route given the input questions, aiming to achieve the highest performance with as little computational cost as possible. We develop a KTP that enables the subsequent LLM to utilize the expert knowledge of the prior LLMs. Experiments demonstrate that DER integrates the advantages of LLMs with only 15% of the inference parameters compared to the LLM ensemble methods based on the output of all LLMs.

Acknowledgments

This work was partially supported by the Joint Funds of the National Natural Science Foundation of China (Grant No.U24A20327), Key-Area Research and Development Program of Guangdong Province (2018B010107001), and TCL Science and Technology Innovation Fund.

Contribution Statement

This work was a collaborative effort by all contributing authors. Jinwu Hu, Yufeng Wang, and Shuhai Zhang made equal contributions to this study and are designated as cofirst authors. Mingkui Tan and Bin Xiao, serving as the corresponding authors, are responsible for all communications related to this manuscript.