CollabLLM: From Passive Responders to Active Collaborators (Extended Abstract)

1 Introduction

Modern Large Language Models (LLMs) excel at generating high-quality single-turn responses when given well-specified inputs. However, real-world users often do not fully articulate their intents and sometimes initiate conversations with an imprecise understanding of their own needs [Taylor, 1968]. As a result, users routinely refine their requests post hoc through iterative corrections, which can increase frustration, hinder effective task completion, and reduce conversational efficiency [Amershi et al., 2019; Zamfirescu-Pereira et al., 2023; Wang et al., 2024; Kim et al., 2024]. Therefore, an open problem is to train models that guide users in clarifying their intents and achieving their goals.

∗This paper is an extended abstract of Wu et al. [2025]; the full paper additionally provides ablations, a case study, out-of-domain generalization results, and qualitative user feedback.

Here we introduce COLLABLLM, a novel and general training framework that improves the ability of LLMs to effectively collaborate with humans in multiturn scenarios [Gao et al., 2019; Balog and Zhai, 2023; Rahmani et al., 2023]. The key innovation of COLLABLLM is to promote LLMs' forward-looking behavior that leads to long-term collaboration gains (Figure 1). We introduce a collaborative simulation module that samples future conversations with users to estimate the long-term impact of model responses across multiple turns, a measure we term the Multiturn-aware Reward (MR). The MR function evaluates responses by incorporating both extrinsic metrics, such as task-specific success, and intrinsic metrics, such as efficiency, to holistically assess collaboration quality (cf. Section 3). By fine-tuning with RL algorithms [Rafailov et al., 2023; Schulman et al., 2017] on MRs, COLLABLLM promotes responses that lead to better task completion and efficiency in later conversation stages.

We also introduce three challenging multiturn tasks for training and evaluation in simulated environments: MediumDocEdit-Chat (document editing), BigCodeBench-Chat (code generation), and MATH-Chat (math problem solving). On the three test sets, our approach improves task accuracy metrics by 18.5% and interactivity by 44.3% over our best baselines. We further perform a large-scale and real-world user study with 201 Amazon Mechanical Turkers, who are asked to write documents with an AI assistant; COLLABLLM achieves impressive improvement with 17.6% increase in user satisfaction and yields user time savings of 10.4%.

2 ProblemFormulation

In contrast to many existing tasks that are single-turn and require no human involvement beyond the initial query, our problem formulation reflects a real-world setting in which a user's underlying (implicit) goal is defined as g in a multiturn conversational task. The conversation unfolds over multiple turns tj = {uj,mj}, where uj is the user input andmj is the model's response at each turn j = 1, . . . ,K, where K is the number of turns in the conversation.

At the j-th turn, the model generates its response based

…

Online generation

Figure 1: COLLABLLM Framework: Given a context 1 , the model generates a response 2 to maximize long-term collaboration gains, termed Multiturn-aware Rewards (MR). During training, MRs are estimated via 3 collaborative simulation, which forward-samples conversations with simulated users. Finally, 4 reinforcement fine-tuning is applied using the MRs.

on the previous conversation turns t1:j-1 = {t1, . . . , tj-1} and the current user response uj. For simplicity, we define historical conversation at j-th turn as thj = t1:j-1 ∪ {uj}, therefore, mj = M(thj ). The objective is to generate a sequence of model responses {mj}Kj=1 that effectively and efficiently achieve goal g, where goal achievement is assessed based on user satisfaction or an external evaluation function. Formally, we define the objective as R∗ (t1:K | g), where R∗ incorporates the achievement of task success and user experience factors such as time cost.

3 Unified Collaborative LLMTraining

Key Motivations. Established LLM training frameworks, such as Reinforcement Learning from Human Feedback (RLHF) [Ouyang et al., 2022], focus on maximizing immediate rewards for single-turn tasks. This causes a misalignment between their single-turn objective and real-world multiturn objective R∗ (t1:K | g). Precisely, the model's accumulative single-turn reward ∑︁j=K j=1 R(mj | thj ) may not imply a higher final reward R∗ (t1:K | g). For example, if the user's goal g is to write an engaging article, an RLHF model may produce locally helpful sections (e.g., an introduction or a conclusion) that fail to cohere across turns.

Instead, effective multiturn collaboration requires model responses that optimally contribute to the final reward. The model should align its responses with the user's goal g by considering their impact on the trajectory t1:K-e.g., asking "Should I maintain an engaging tone in the conclusion like the introduction?" rather than drafting in isolation.

3.1 Multiturn-aware Rewards

Our key insight is that effective multiturn collaboration relies on forward-looking strategies. Given a context 1 , the model should consider how its response 2 influences the subsequent turns of the conversation. To capture this, we design a 3 collaborative simulation module to estimate this impact. By 4 fine-tuning to distinguish between future conversations from different responses, the model better aligns with the overarching goal g. This aligns with causal effect estimation [Pearl,

2009; Pearl et al., 2016]: MR is an interventional estimate of a response's causal contribution to long-term reward, unlike trajectory-level methods that learn from observed conversations. More specifically, we define:

Multiturn-aware Reward (MR): The multiturn-aware reward for model response mj at the j-th turn is given by:

where t1:j denotes the conversation history up to and including the j-th turn (equivalent to thj ∪ {mj}), and tfj = tj+1:K represents the forward trajectory of turns following the j-th turn. The distribution P(tfj | t1:j) models the possible forward conversations conditioned on the prior conversation history.

Computing Equation 1 requires two components, detailed below: (a) a conversation-level reward function R∗ (t | g) to evaluate an arbitrary multiturn conversation t, and (b) a sampling strategy for forward conversations P(tfj | t1:j).

Conversation-level Reward Function

We approximate the conversation-level reward R∗ (t | g) with a combination of extrinsic (goal-specific) and intrinsic (goalagnostic) metrics:

where Rext(t, g) focuses on task success, and Rint(t) evaluates user experience including efficiency and engagement.

• Extrinsic Reward Rext(t, g) measures how well the conversation achieves the user's goal g. Formally:

where Extract(t) extracts the final solution or response from the conversation t, especially for tasks requiring revisions or multi-step answers. yg is the reference solution for the goal g, e.g., the ground truth solution for a math problem. And S(·, ·) evaluates task-specific metrics like accuracy or similarity. This ensures the conversation contributes directly to achieving the desired goal.

• Intrinsic RewardRint(t) prioritizes conversations that enhance user experience, defined as:

where we encourage conversational efficiency by penalizing excessive tokens that users read and write, with λ controlling the penalty severity. This efficiency measure is bounded by 1 to maintain balance with other metrics. The second term, RLLM(t), is assigned by an LLM-based judge [Zheng et al., 2023] on a 0-1 scale, evaluating uservalued objectives such as engagement / interactivity. Notably, additional conversational aspects, such as clarity, can be further integrated into the objective.

Forward Sampling

To compute Eq. 1, we require samples from P(tfj | t1:j), the distribution of forward conversation conditioned on the conversation history. A simple approach is to use Monte Carlo sampling, where the conversation is extended turn-by-turn until it concludes. However, this can be computationally expensive for computing reward for every model response. For a scalable approximation, we introduce a window size w as a hyperparameter to limit the maximum number of forward turns considered in tfj (we use w = 2 in all experiments). This reduces the computational cost while maintaining sufficient context.

Gathering forward conversations from human participants at scale is impractical, so we introduce a user simulator U.

User Simulator: A user simulator U : T → U maps a conversation history t to a user response u, generating a distribution P(u | t) that simulates realistic user behavior.

Specifically, we prompt an LLM to role-play as users (each with an implicit goal g), following the same language style as previous user turns and injecting typical user behaviors such as evolving needs, limited background knowledge, or clarification requests [Park et al., 2024]. Unlike prior work that uses simulators mainly for synthetic dialogue data generation [Shi et al., 2019; Tseng et al., 2021; Hong et al., 2023; Hu et al., 2023; Faltings et al., 2023], COLLABLLM leverages them in forward sampling to estimate long-term response effects.

3.2 Optimization & Synthetic Datasets

With the conversation-level reward function and forward sampling strategy, we can compute MR for any model response without requiring an additional reward model, which is often costly and slow to train. Further, we employ reinforcement learning (RL) methods such as PPO [Schulman et al., 2017] and DPO [Rafailov et al., 2023] to guide the model in multiturn conversations. Moreover, MR can generate high-quality synthetic conversations for both supervised fine-tuning (SFT) and DPO: for SFT, it iteratively selects topranked responses to build realistic, goal-directed conversation histories; for DPO, it constructs pairwise comparisons by ranking responses at each turn, distinguishing "chosen" and "rejected" pairs based on MR scores.

4 Experimental Setup

For fine-tuning and evaluation, we create three multiturn datasets using publicly available data across diverse domains [Hendrycks et al., 2021; Zhuo et al., 2024; Chiusano, 2024]: collaborative document editing, coding problem assistance, and multiturn mathematics problem solving.

To build a multiturn environment, we employ GPT-4o-mini as a user simulator LLM to role-play realistic user behaviors, given the target problem and conversation history. The three datasets are:

MediumDocEdit-Chat: Document editing requires iterative feedback and refinements across multiple turns to ensure coherence and alignment with user intent. We sample 100 Medium articles as goal documents, which are summarized into target problems to guide the user simulator. After each interaction, task performance is evaluated using the BLEU score, measuring similarity between the extracted document and the original articles.

BigCodeBench-Chat: Coding tasks inherently require multiturn interactions, such as clarifying requirements and debugging. We sample 600 coding problems from BigCodeBench [Zhuo et al., 2024] as the target problems given to the user simulator. For evaluation, we compute the average Pass Rate (PR) of code at the end of the interactions.

MATH-Chat: Math problem solving often requires addressing implicit assumptions, verifying intermediate steps, and clarifying reasoning. We sample 200 level-5 math problems from MATH [Hendrycks et al., 2021] to prompt the user simulator, which interacts with the LLMs. Task success is measured by the accuracy (ACC) of the final solution.

In addition to the above task-specific metrics, we incorporate two task-agnostic scores across all datasets: 1) Average Token Count, which quantifies the average number of tokens generated by the LLM per conversation, reflecting interaction efficiency. 2) Interactivity (ITR), which evaluates engagement levels using an LLM judge (Claude-3.5-Sonnet), with scores rescaled to an upper bound of 1.

Fine-tuning COLLABLLMs. COLLABLLMs are based on Llama-3.1-8B-Instruct [Llama Team, 2024] with LoRA finetuning [Hu et al., 2022]. We train four model variants: 1) Offline models: SFT and Offline DPO are fine-tuned on pregenerated multiturn conversational datasets (500 conversations per task) guided by Multiturn-aware Rewards (MR) (cf. Section 3.2). 2) Online models: PPO and Online DPO are further trained from the SFT and Offline DPO models, respectively. During online fine-tuning, the model joins the collaborative simulation to compute MRs, which dynamically adjust model preference.

Baselines. We compare COLLABLLMs against (1) the pretrained Llama-3.1-8B-Instruct (Base), (2) the base model with proactive prompt engineering (Proactive Base), which encourages follow-up and clarification questions.

5 Results of Simulated Experiments

We present the results in Table 1 and the takeaways are: Prompt engineering is helpful, but limited in terms of performance gains and flexibility. Proactive Base improves base model performance by encouraging follow-up questions and clarifications. For example, it increases BLEU on MediumDocEdit-Chat from 32.2% to 35.0% and reduces read tokens by 0.31k compared to the base model. However, these gains are modest and do not fully address the challenges of multiturn collaboration. We observe that prompting strategies remain rigid, relying on predefined instructions rather than adapting dynamically to user needs. For instance, the model sometimes asks unnecessary clarification questions that disrupt the conversation flow.

COLLABLLM improves task performance, efficiency, and engagement. Averaged across the three datasets, COLLABLLM (Online DPO) achieves 18.5% higher task-specific performance, 13.3%more efficient conversations, and 44.3% enhanced interactivity over Proactive Base (Table 1, last row). For MediumDocEdit-Chat, the Online DPO model increases ITR from 0.46 to 0.92. For MATH-Chat, Online DPO decreases token count per conversation from 3.40k to 2.37k.

6 Real-world User Study

Setup. We conduct a large-scale user study using Amazon Mechanical Turk with 201 participants. Each participant is assigned a document type-randomly selected to be either blog post, creative writing, or personal statementand chooses a topic from a predefined set. Participants then engage in at least eight turns of conversation with an anonymized AI assistant, which can be Base, Proactive Base, or COLLABLLM. We also record the total interaction duration to assess efficiency. Participants rate the final document quality and overall interaction on a 1-10 scale.

Results. COLLABLLM achieves an average document quality score of 8.50, a 17.6% relative gain over Base. Specifically, 91.4% of participants rate COLLABLLM's document quality as "good" (score 8-9), and 56.9% as "very good" (score 9-10), compared to 88.5% and 39.3% for Base (Llama-3.1-8B-Instruct), respectively. Similarly, 63.8% of participants find COLLABLLM highly engaging, while only 42.6% report the same for Base. Moreover, COLLABLLM improves task efficiency, reducing time spent by 10.4% compared to the Base model and by 15.6% relative to Proactive Base. Overall, COLLABLLM enhances collaboration through iterative refinement; future work should improve personalization, creativity, and real-time knowledge integration.

7 RelatedWork

Non-collaborative LLM training. Existing LLM training frameworks, including pre-training, supervised fine-tuning (SFT), and reinforcement learning (RL) [Rafailov et al., 2023; Schulman et al., 2017; Ouyang et al., 2022; Lee et al., 2024], primarily optimize for next-turn response quality. While effective for single-turn objectives, these approaches fail to capture how responses influence user intent discovery and task success [Amershi et al., 2019; Zamfirescu-Pereira et al., 2023; Wang et al., 2024; Kim et al., 2024].

Learning-based methods for multiturn interaction. Beyond prompting, prior studies have explored supervised finetuning [Andukuri et al., 2024], RL fine-tuning [Chen et al., 2024; Zamani et al., 2020; Erbacher and Soulier, 2023], and active learning [Pang et al., 2024] to train models to ask clarification questions. Several studies extend RLHF to multiturn settings by optimizing trajectory-level rewards [Shani et al., 2024; Zhou et al., 2024; Gao et al., 2024; Shi et al., 2024; Zhang et al., 2025]. Other works [Xu et al., 2023; Deng et al., 2024] leverage self-chat or self-play to enhance model adaptation. However, these methods primarily rely on post-hoc trajectory-level data, learning from observed conversations rather than explicitly modeling the causal effect of individual responses on task success. Additionally, they often overlook open-ended tasks such as document generation [Faltings et al., 2023; Jiang et al., 2024], where user responses can be highly diverse, and users may have limited capacity to read and refine lengthy model outputs.

8 Conclusion

Multiturn human-LLM collaborations are increasingly prevalent in real-world applications. Foundation models should act as collaborators rather than passive responders, actively uncovering user intents in open-ended and complex tasksan area where current LLMs fall short. The key insight of COLLABLLM is making LLMs more multiturn-aware by using forward sampling to estimate the long-term impact of responses. Through simulated and real-world evaluations, we show COLLABLLM is effective, efficient, and engaging across three diverse tasks; the full paper [Wu et al., 2025] further demonstrates zero-shot generalization to out-of-domain tasks (Abg-CoQA), advancing human-centered LLMs.