Enhancing User-Oriented Proactivity in Open-Domain Dialogues with Critic Guidance

1 Introduction

Humans are naturally more engaged in conversations relevant to their interests [Oertel et al., 2020]. To meet humans' chatting preferences, the open-domain dialogue task aims to build chatbots that produce engaging and meaningful natural language responses [Kann et al., 2022]. It yields practical value in various human-computer interaction applications, such as personal assistants [Luo et al., 2022], education [Rodrigues et al., 2022] and social robotics [Grassi et al., 2022]. Recently, the rise of large language models (LLMs) [Zhao et al., 2023; Hu et al., 2024] has led to unprecedented popularity in dialogue systems. LLM-based dialogue systems have demonstrated extraordinary context understanding and coherent response generation capabilities [Deng et al., 2023a], significantly advancing the development of open-domain dialogue.

Despite excellent dialogue capabilities of LLMs, they still have limitations due to the inability of proactivity. The proactive chatbot is designed to go beyond passive response to user inputs, but actively explore the user's chatting interests and lead the topics to the user's preferences. Since LLM-based methods heavily rely on the existing training conversations and knowledge to provide suggestions and complete tasks [Achiam et al., 2023], they typically generate responses to user queries in a passive manner [Liao et al., 2023]. They tend to help solve problems rather than proactively learn about the background and chatting interests of the user (Figure 1, left). Existing methods enhance proactivity by topic planning and shifting, or topic-aware response generation [Tang et al., 2019; Deng et al., 2023b; Deng et al., 2023c; Wang et al., 2024]. Although they can actively lead the chatting topic, these target-oriented methods focus more on target-centered topic switching, resulting in a lack of proactive attention toward the users themselves (Figure 1, medium). This may not fully align with the original intention of open-domain chatbots, which is to enhance the user's chat experience and create engaging dialogues. To address this, we delve into the proactivity of exploring user's background and chatting interests and take proactivity to lead the conversation toward user-centered topics, namely the User-oriented Proactivity (Figure 1, right). It enables the chatbot to deliver responses that are not only relevant but also resonate with the user's chatting interests and background, thereby fostering more satisfying interaction.

In this paper, we propose a User-oriented Proactive Chatbot (UPC) to address the lack of user-oriented proactivity. We build a chatbot that explores the user's background and chatting interests, and actively leads the conversation towards user-centered topics. Specifically, since assessing proactivity by humans is costly, LLM can serve as a judge that aligns with humans [Zheng et al., 2024]. We appoint an LLM to be a critic and design evaluation metrics for this proactivity. Besides, the lack of a high-quality training corpus for a proactive chatbot puts us in a dilemma of making bricks without straw. To this end, we introduce a critic-guided dialogue corpus generation paradigm. We employ the critic to guide dialogues between the chatbot and different user agents, generating a dialogue corpus with enhanced user-oriented proactivity. Since communication difficulty varies across users, the chatbot may struggle with users who have uncommon backgrounds or preferences in the early training stages. Therefore, we propose a communication difficulty-aware iterative curriculum learning approach. It adapts the chatbot to easy-tocommunicate users in early training iterations and incrementally learns the chatbot with more challenging users, achieving steady performance improvement. Finally, we construct the ISCO-800, a dataset with 800 user backgrounds, to create diverse user agents. Our main contributions are as follows:

1. Design of a critic to assess user-oriented proactivity. To address the lack of user-oriented proactivity in the chatbot, we develop a critic to quantify and assess this proactivity. It also guides high-quality corpus generation for training. The real user evaluation verifies its alignment with humans.
2. Iterative curriculum learning is proposed for chatbot training. Recognizing that communication difficulty varies among different users, we define adaptation difficulty and introduce an iterative curriculum learning method. It first adapts the chatbot to easy-to-communicate users in early training iterations and progressively includes more challenging ones. Experiments show that this paradigm improves average performance by 8.2% compared to the original LLM.
3. Construction of a user background dataset ISCO-

800. To train a chatbot capable of interacting with users from various backgrounds. We construct a user background dataset that includes 800 types of user background information. It provides realistic background information, including occupation, hobby, education, and personality, enabling the chatbot to interact with a wide range of users.

2 RelatedWorks

Significant efforts have been made to enhance the human-like performance of open-domain dialogue systems. Based on key issues, existing studies relevant to our work can be broadly categorized into three areas: coherent dialogue, personalized dialogue, and target-oriented dialogue.

Coherent dialogue aims to improve contextual coherence during conversations. Memochat [Lu et al., 2023] uses memorization-retrieval-response cycles to teach the LLMs to memorize and retrieve past dialogues with structured memos, leading to enhanced consistency. Sun et al. [Sun et al., 2023] propose a hybrid latent variable method. It combines discrete and continuous latent variables to improve both semantic correlation and diversity. Although these methods improve coherence, they do not prioritize the proactive, user-centric dialogue that is central to our work.

Personalized dialogue focuses on building a chatbot endowed with a persona [Tu et al., 2023; Chen et al., 2023], or modeling the persona of the other party [Chen et al., 2024; Zhong et al., 2024]. For example, ORIG [Chen et al., 2023] proposes a model-agnostic framework to fine-tune the persona dialogue model. It enables the dialogue models to learn robust representations and improve the consistency of response generation. In addition to improving the chatbot's role-playing abilities, Memorybank [Zhong et al., 2024] adds long-term memory functionality for LLMs by summarizing past interactions to capture user personalities. However, unlike the static personality, the user's chatting interests evolve dynamically during a conversation. These methods often struggle to actively explore and engage with these changing interests, limiting their ability to generate topics that align with what the user is currently interested in.

Target-oriented dialogue improve proactivity by planning and steering the dialogue to pre-set topics [Tang et al., 2019; Deng et al., 2023b; Deng et al., 2023c; Wang et al., 2024]. For example, TRIP [Wang et al., 2024] generates the dialogue path with a bidirectional planning method to drive the conversation to the goal. ProCoT [Deng et al., 2023c] proposes the proactive chain-of-thought prompting scheme, which augments LLMs with the goal planning capability. Although they actively lead the conversation towards system-side goals, they lack the ability to understand the user's chatting interests, resulting in low engagement and a diminished willingness to continue the conversation. In contrast, our method emphasizes user-oriented proactivity by focusing on topics that are centered around the user's background and interests.

3 ProblemDefinition

Given a chatbot C𝜃 parameterized by 𝜃, we consider the task of having a multi-turn open-domain conversation between the chatbot and each user U𝑖 from a user set {U𝑖} 𝑁 𝑖=1 . The resulting dialogue corpus is denoted as {D𝑖} 𝑁 𝑖=1 with the total

conversation with 𝑇 turns, and 𝑢 𝐵𝑜𝑡 𝑖,𝑡 , 𝑢 𝑈𝑠𝑒𝑟 𝑖,𝑡 are the utterances of the chatbot and user at turn 𝑡, respectively.

Existing methods often passively respond to the user queries or focus on target-oriented topics. This lack of user-oriented proactivity can make users feel unappreciated, reducing their satisfaction and willingness to continue the conversation in human-computer interactions. Therefore, our goal is to develop a chatbot that proactively explores the user's chatting interests and guides conversations toward user-centered topics. To evaluate this kind of proactivity, the primary evaluation metric is whether the user finds the chatbot's response interesting. In addition, background relevance plays a crucial role, as the user typically focuses on topics related to their context. Finally, the value of the chatbot's response is an important factor, as it directly influences the quality of the conversation. To this end, We aim to build a chatbot with user-oriented proactivity from the perspective of the user's chatting interests 𝑆 𝑖,𝑡 𝑖𝑛𝑡. , background relevance 𝑆 𝑖,𝑡 𝑟 𝑒𝑙 . and response value 𝑆 𝑖,𝑡 𝑣𝑎𝑙. , respectively. These metrics gauge the chatbot's ability to proactively explore the user's background, engage with the user's desired topics, and provide responses that are both interesting and valuable.

4 User-oriented Proactive Chatbot

The overall framework for UPC training is shown in Figure 2. We first construct a critic to evaluate user-oriented proactivity, then train the chatbot in an iterative curriculum learning process. In each iteration, the chatbot engages in a dialogue with different user agents, each with a distinct background from our ISCO-800 dataset. During the conversation, the chatbot receives evaluations from the critic, which guides the generation of a high-quality corpus with improved proactivity. Since the communication difficulty varies across users, we assess the difficulty based on the quality of the generated corpus, so as to fine-tune the chatbot with the corpus corresponding to easy users. Then, the fine-tuned chatbot is used to start a new iteration of dialogue corpus generation. We repeat this iterative process to improve the performance until convergence.

4.1 Evaluation of User-oriented Proactivity

To evaluate the user-oriented proactivity S𝑖,𝑡 from the three perspectives of user's interest level, background relevance, and response value, we follow the LLM-as-a-judge method [Zheng et al., 2024] and construct the critic J by prompting the ChatGPT to score the abilities. Specifically, we let the critic obtain the user's background and the dialogue history D𝑖,𝑡-1 between the user and the chatbot. Then we use the 5-point scoring system to evaluate the current response 𝑢 𝐵𝑜𝑡 𝑖,𝑡 of the chatbot. It is formalized:

Figure 3 illustrates the prompt of critic. We provide descriptions for scores of 1, 3, and 5 to improve the rating accuracy.

4.2 Critic-guided Dialogue Corpus Generation

We employ the critic to guide dialogues between the chatbot and user agents, generating a dialogue corpus with enhanced user-oriented proactivity. We use Qwen1.5-72B-Chat,

Prompt of the critic

You are now playing a character named {name}, and you are chatting with another person named Bot. You need to evaluate and rate the Bot's response based on its performance.

This is your background information: \n{background}

The historical dialogue between you and Bot is: \n{history}

Bot's next response is: \n{response}

Using the description below, combined with the historical conversation records, evaluate and rate the response of the Bot with score 1 to 5:

• Relevance to your background of Bot's next response. Score: 1, 2, 3, 4, 5: [1: <The content only has a little connection to my stated interests or background>, 3: <The content generally relates to my interests or background, with a fair level of relevance throughout the conversation>, 5: <The content is highly tailored to my specific interests and background. It may also expand on topics that are relevant to me>], because...
• Value of Bot's next response. Score: 1, 2, 3, 4, 5: [1: <The content only has a little substantive information. It may include meaningless repetition, incorrect information>, 3: <The content can somewhat meet my needs, although it may not be completely comprehensive or in-depth>, 5: <The content is insightful or deeply analytical, greatly facilitating my understanding>], because...
• Your level of interest in the Bot's next response. Score: 1, 2, 3, 4, 5: [1: <I only have a little interest in the response>, 3: <The response is reasonably interesting, though it may not be my favorite subject>, 5: <I am deeply fascinated by this response. It is highly engaging and something I look forward to>], because...
• Be cautious in giving a score of 4 or above. Only use a score of 5 when the content is truly exceptional and exceeds your expectations.

You can only output three sentences, each following the format below:

Relevance to your background of Bot's next response: (score), because...

Value of Bot's next response: (score), because...

Level of interest in the Bot's next response: (score), because...

Figure 3: The prompt of the critic.

an LLMwith strong role-playing abilities [LMSYS, 2024], to act as user agents. Each user agent is assigned a unique background from our ISCO-800 dataset, described in Section 4.4. The system prompt of the user agent is shown in Appendix E. We then put the chatbot and each user agent into multiple rounds of open-domain conversation. Notably, the chatbot does not have prior access to the user's background but is instead prompted to actively learn about the background of the user during the conversation, and find topics of potential interest for the user. The system prompt is shown in Appendix E. In the first round, we prompt the chatbot to greet first and let the user give a brief self-introduction. Then they engaged in open-ended conversation. The critic evaluates each response from the chatbot, excluding the initial greeting. Each response is scored on three dimensions: user interest, background relevance, and response value. Scores range from 1 to 5, with corresponding reasons provided. For responses scoring below 4, the chatbot is prompted to regenerate based on the critic's feedback, as shown in Appendix E. The regeneration continues until all scores are 4 or higher, or until the maximum number of regenerations is reached. Our critic-guided approach ensures that the chatbot generates high-quality responses by trial and error like humans [Young, 2009], progressively enhancing user-oriented proactivity. Besides, the proportion of high-quality responses increases with the evolution of the chatbot. The corpus generation pipeline in each training iteration 𝑘 is detailed in Algorithm 1.

Algorithm 1 Dialogue corpus generation in iteration 𝑘.

Require: The chatbot C𝜃𝑘 in iteration 𝑘, the user agent U𝑖 from ISCO-800 dataset {U𝑖 } 𝑁 𝑖=1 . The critic J. The maximum re-gen attempts 𝑅, dialogue turns 𝑇 and score buffer S𝑖 .

1: for 𝑖 = 1, ..., 𝑁 do 2: for 𝑡 = 1, ..., 𝑇 do 3: if 𝑡 == 1 then 4: The chatbot C𝜃𝑘 gives an greeting utterance 𝑢 𝐵𝑜𝑡 𝑖,𝑡 . 5: The user agent U𝑖 gives brief self-introduction 𝑢 𝑈𝑠𝑒𝑟 𝑖,𝑡 . 6: Add (𝑢 𝐵𝑜𝑡 𝑖,𝑡 , 𝑢 𝑈𝑠𝑒𝑟 𝑖,𝑡 ) to the dialogue history D𝑖,𝑡 . 7: else 8: The chatbot C𝜃𝑘 gives response 𝑢 𝐵𝑜𝑡 𝑖,𝑡 . 9: Compute scores S𝑖,𝑡 for current 𝑢 𝐵𝑜𝑡 𝑖,𝑡 via Eqn. (1). 10: Cache scores, utterances: S ′ 𝑖,𝑡 ← S𝑖,𝑡 , 𝑢 ′𝐵𝑜𝑡 𝑖,𝑡 ← 𝑢 𝐵𝑜𝑡 𝑖,𝑡 . 11: Initialize 𝑟 = 0. 12: while (𝑆 𝑖,𝑡 𝑖𝑛𝑡. , 𝑆 𝑖,𝑡 𝑟𝑒𝑙. or 𝑆 𝑖,𝑡 𝑟 𝑒𝑙 . < 4) 𝑎𝑛𝑑 𝑟 <= 𝑅 do 13: Re-generate utterance 𝑢 𝐵𝑜𝑡 𝑖,𝑡 , 𝑟 = 𝑟 + 1. 14: Compute scores S𝑖,𝑡 for 𝑢 𝐵𝑜𝑡 𝑖,𝑡 via Eqn. (1). 15: end while 16: The user agent U𝑖 gives response 𝑢 𝑈𝑠𝑒𝑟 𝑖,𝑡 . 17: Add (𝑢 ′𝐵𝑜𝑡 𝑖,𝑡 , 𝑢 𝐵𝑜𝑡 𝑖,𝑡 , 𝑢 𝑈𝑠𝑒𝑟 𝑖,𝑡 ) toD𝑖, Add (S ′ 𝑖,𝑡 , S𝑖,𝑡 ) to S𝑖. 18: end if 19: end for 20: end for 21: return Dialogue corpus {D𝑖} 𝑁 𝑖=1 and scores {S𝑖} 𝑁 𝑖=1 in iter. 𝑘.

4.3 Communication Difficulty-Aware Iterative Curriculum Learning

Since the training data is generated through interactions between the chatbot and the user, timely improvement of the chatbot helps enhance the generated data and final performance. Therefore, it is crucial to conduct iterative training that uses the model from the previous iteration to generate data for the next iteration. Besides, since the communication difficulty varies among different users [Lindqvist, 2015], similar to human learning, fine-tuning models benefit from an easy-to-difficult curriculum during model training [Gao et al., 2024; Hu et al., 2025]. Therefore, we propose a communication difficulty-aware iterative curriculum learning framework that consists of a Communication Difficulty Measurer and Training Scheduler.

Communication Difficulty Measurer. Our difficulty measurer captures two aspects: how well the chatbot performs and whether it can improve. The first checks if the chatbot can produce acceptable responses for the user. The second evaluates whether the chatbot can make meaningful progress from the feedback. Therefore, we use the scores described in Section 4.1 and their improvement with the critic as the Difficulty Measurer 𝐷𝑖 𝑓 (𝑢 𝐵𝑜𝑡 𝑖,𝑡 ):

the condition 𝑃 is formalized:

Algorithm 2 Iterative Curriculum Learning.

Require: The chatbot C𝜃 with parameter 𝜃. User set {U𝑖} 𝑁 𝑖=1 . The maximum number of iterations 𝐾 and dialogue turns 𝑇. 1: for 𝑘 = 1, . . . , 𝐾 do 2: Collect a corpus {D𝑖} 𝑁 𝑖=1 according to Algorithm 1. 3: Initialize D ∗ = ∅. 4: for 𝑖 = 1, . . . , 𝑁 do 5: 𝑖𝑠 𝑒𝑎𝑠𝑦 ← true 6: for 𝑡 = 1, . . . , 𝑇 do 7: Calculate 𝐷𝑖 𝑓 (𝑢 𝐵𝑜𝑡 𝑖,𝑡 ) via Eqn. (2). 8: if 𝐷𝑖 𝑓 (𝑢 𝐵𝑜𝑡 𝑖,𝑡 ) == true then 9: 𝑖𝑠 𝑒𝑎𝑠𝑦 ← false 10: break 11: end if 12: end for 13: Add D𝑖 to D ∗ if 𝑖𝑠 𝑒𝑎𝑠𝑦. 14: end for 15: Fine-tuning the model C𝜃𝑘 with D ∗ . 16: end for 17: return Chatbot C𝜃𝐾 .

where 𝑇𝑟𝑢𝑒 represents difficult and the reverse is easy. 𝛼 and 𝛽 are hyperparameters, specified in Appendix B. A sample is considered difficult if any metric is below 𝛼 or the number of boosted metrics is fewer than 𝛽. These criteria identify user samples that are either hard to generate high-quality responses or hard to improve.

Training Scheduler. We adjust the generated training corpus at each training iteration 𝑘 with the Difficulty Measurer for the collected multi-round dialogue corpus. Specifically, for the dialogue corpus D of different users, we follow Eqn. (2) to measure the difficulty of the corpus {D𝑖} 𝑁 𝑖=1 . The easy corpus constitutes the training dataset D ∗ , while the difficult corpus will be discarded and its corresponding users will be engaged in dialogues in the next round. Notably, as the chatbot is updated, its ability to adapt to a wider range of users improves, leading to an increase in the amount of easy corpus. The training pipeline is shown in the Algorithm. 2.

4.4 User Background Dataset ISCO-800

To train a chatbot capable of interacting with users from diverse backgrounds, we construct a dataset called ISCO800, containing 800 pieces of user background information. Specifically, to ensure the dataset is representative of global occupations, we refer to the ISCO-08 classification by the International Labour Organization [Organization, 2008]. We randomly select 40 out of 43 sub-major occupation groups from ISCO-08 as the source data. The details of these occupation groups are listed in Appendix C. We designed a prompt in Appendix C to guide GPT-4 in generating realistic user backgrounds for each occupation. These backgrounds include names, career histories, educations, personalities, and hobbies. Each user's background consists of 50 to 100 words. To ensure realism, not all users have positive personalities or smooth careers. We generate 20 different backgrounds for each occupation group, resulting in 800 user backgrounds. Statistics of ISCO-800 and comparison with other related datasets in terms of the user background are in Appendix C.

5 Experiment

5.1 Experimental Settings

Implementation Details. We implemented UPC with Qwen1.5-32B-Chat [Bai et al., 2023] as the base LLM. We use user backgrounds from ISCO-800 to form the user agents and each engages in 5-turn dialogues with the chatbot. The 800 user agents are divided into training, validation, and test sets (500, 100, and 200 users, respectively) for dialogue generation. They are not duplicated in the occupation groups. This ensures the model is always evaluated on unseen domains during testing. See Appendix B and C for more details.

Compared Methods. We compare UPC with existing open-domain dialogue methods, including the universal LLMs, recent target-oriented open-domain dialogue methods (ProCoT [Deng et al., 2023c] and TRIP [Wang et al., 2024]) and other LLM-based methods (BlenderBot 3 [Shuster et al., 2022], MemoChat [Lu et al., 2023] and MemoryBank [Zhong et al., 2024]). For a fair comparison, we adapt them to the user-oriented dialogue tasks by fine-tuning the models or using prompt strategies. See Appendix B for details.

Metrics. We use our designed scoring system (including the background relevance 𝑅𝑒𝑙., user's interest level 𝐼𝑛𝑡., and response value 𝑉𝑎𝑙.) to evaluate the chatbot's user-oriented proactivity. Notably, the critic used for evaluation includes gpt-3.5-turbo-0125 or gpt-4-turbo-2024-04-09. In addition, we also adopt the widely used perplexity (PPL) [Jelinek et al., 1977] to measure the quality of the generated corpus.

Evaluation. In the test phase, we obtain the user backgrounds of the test set from ISCO-800 and construct different user agents. Then we let the chatbot start 5 turns of opendomain dialogue with each user agent. The chatbot does not obtain the user's background information in advance. We use the critic to evaluate metrics of user-oriented proactivity of each turn of the chatbot's response for the generated dialogue corpus and obtain the averaged scores.

5.2 Comparison Experiments

We compare our UPC with other open-domain dialogue methods and the results are shown in Table 1. Our proposed UPC, with only 32B parameters, outperforms all other methods in terms of 𝑅𝑒𝑙., 𝐼𝑛𝑡., 𝑉𝑎𝑙., and PPL, demonstrating its effectiveness. The detailed analyses are as follows.

UPC outperforms universal LLMs with fewer parameters. With GPT-3.5 as the critic, our UPC outperforms the strongest baseline Llama-3-70B-Instruct by 3.32%, 4.42%, and 4.03% on 𝑅𝑒𝑙., 𝐼𝑛𝑡. and 𝑉𝑎𝑙.. With GPT-4 as the critic, our UPC outperforms the strongest baseline Qwen1.5-72BChat by 1.39%, 1.33% and 3.69% on 𝑅𝑒𝑙., 𝐼𝑛𝑡. and 𝑉𝑎𝑙.. Our method also reduces the PPL by 21.64% compared with Llama-3-70B-Instruct and 29.36% compared with Qwen1.572B-Chat. It indicates that our UPC improves the performance even with half of the model parameters. We attribute the improvement to the critic-guided corpus construction paradigm that facilitates the LLM to generate user-interested dialogue data. Besides, the iterative curriculum learning method helps adapt to different users from easy to hard, enhancing the chatbot's user-oriented proactivity. Moreover, the GPT-4-based critic exhibits similar scoring trends to the GPT3.5-based critic. It suggests that although we use GPT-3.5based critic during training, we still obtain a consistent judgment with the better-performing GPT-4 in the test. Therefore, we use GPT-3.5-based critic in ablation studies.

UPC outperforms the target-oriented methods. UPC outperforms the strongest baseline TRIP by 7.45%, 8.50%, and 8.25% on 𝑅𝑒𝑙., 𝐼𝑛𝑡. and 𝑉𝑎𝑙. (Critic: GPT-3.5) and 10.99%, 7.51%, and 12.97% (Critic: GPT-4). UPC also reduces the PPL by 14.70%. We attribute this to the fact that although it has proactivity in target topic switching, the topic is not necessarily the one that the user is interested in, leading to lower performance of user-oriented proactivity.

UPC outperforms other LLM-based methods. UPC outperforms the strongest MemoChat by 5.40%, 6.19%, and 4.08% on 𝑅𝑒𝑙., 𝐼𝑛𝑡. and 𝑉𝑎𝑙. (Critic: GPT-3.5) and 14.63%, 8.23%, and 7.85% (Critic: GPT-4). Our method also reduces the PPL by 9.31%. We attribute this to the reason that MemoChat's focus on maintaining consistency offers limited improvement in open-domain conversations where topics shift constantly, whereas our trial-and-error strategy yields better performance across a diverse range of topics.

5.3 Ablation Studies

We conducted ablation experiments with the GPT-3.5-based critic to validate the effectiveness of our proposed iterative curriculum learning. We compared different training strategies in our iterative curriculum learning, described as follows:

SFT: An initial LLM generates a dialogue corpus through multiple dialogues with each user agent for supervised finetuning, without the critic's involvement. SFT+CDC (Criticguided Dialogue Corpus Collection): Based on SFT, a critic evaluates the chatbot's responses and requests regenerations for low-rated responses, producing a high-quality corpus for fine-tuning. We repeat the above two strategies for the same number of iterations as UPC to ensure a consistent amount of training data. CDC+IFT (Iterative Fine-Tuning): The chatbot engages in dialogue with each user agent, incorporating critic feedback. The collected corpus is used for initial fine-tuning. Then the fine-tuned chatbot re-engages with each user agent and collects the corpus. This process of corpus collection and fine-tuning undergoes the same iterative rounds as UPC, gradually enhancing the performance of the chatbot. CDC+IFT+CL (Curriculum Learning): Building on CDC+IFT, we additionally use our Communication Difficulty Measurer to identify easy samples after each dialogue generation, and we only use easy samples for fine-tuning.

Effectiveness of CDC. As in Table 2, UPC (SFT+CDC) outperforms UPC (SFT) in all metrics, as the critic feedback improves the dialogue corpus quality, resulting in better finetuning and enhanced LLM performance [Zhou et al., 2024]. Besides, UPC (SFT) performs even inferior to that of the original LLM. We attribute this to the poor-quality corpus generated during training data collection without feedback.

Effectiveness of IFT. As in Table 2, compared to only CDC, training with CDC and IFT improves the user-oriented proactivity of chatbots by 0.97%, 1.80%, and 3.27% on 𝑅𝑒𝑙 ., 𝐼𝑛𝑡., and 𝑉𝑎𝑙., respectively. It indicates that re-collecting the dialogue corpus data using the fine-tuned model helps to improve the quality of the generated data and improves the performance of the model in the next round of fine-tuning.

Effectiveness of CL. As in Table 2, UPC has significant improvement compared to UPC (w/o CL). Specifically, there are 6.12%, 5.48% and 4.44% increases on 𝑅𝑒𝑙., 𝐼𝑛𝑡., and 𝑉𝑎𝑙. respectively. In conclusion, curriculum learning helps the model learn the essential features of the task more quickly and improves the generalization ability [Wang et al., 2021].

5.4 More Discussions

Online Chatbot Interaction with Real Users. To illustrate the practicality of UPC, we deploy UPC and one of the strongest baselines Llama-3-70B-Instruct on the Internet and recruited 62 participants to have free chat with each of the two chatbots. Participants are unknown to the names of the chatbots. The participants are first advised to chat with the two chatbots separately with any identity and topics. Then they are required to compare the performance of the two chatbots. See Appendix D for detailed settings. As in Figure 4, over 75% of participants preferred UPC in the overall performance as well as the 𝑅𝑒𝑙., 𝐼𝑛𝑡. and 𝑉𝑎𝑙.. It indicates our UPC has enhanced performance and aligns with human preferences. Most Participants consider that UPC pays more attention to their background and chatting interests, gives more valuable responses, and leads to a better chatting performance. It reveals the practical value in real chatting scenarios.

Performance evolution during iterative curriculum learning. To illustrate the performance improvement process during iterative curriculum learning, we present our designed metrics on the validation set and the rate of easy samples in the training data at each iteration in Figure 5. It shows that performance and the rate of easy samples gradually improve with each iteration, converging at the 4 𝑡ℎ iteration. This suggests that iterative curriculum learning progressively enhances model performance by training the chatbot to adapt to users with increasing communication difficulty. Additionally, the maximum number of iterations is set to 4 due to convergence. We further calculate the regeneration rate and average number of regenerations during the training process. As in Table 3, both the regeneration rate and average regenerations decrease as the iteration proceeds. It indicates that through iterative curriculum learning, the cost of regeneration is reduced as the iteration progresses. Besides, the chatbot increasingly delivers satisfactory responses on the first attempt.

Effects of LLM category and size. To verify the applicability of UPC, we conduct experiments on LLMs of different categories and sizes. In Table 4, for various sizes in Qwen1.5

series, UPC consistently improves performance, with gains of 5%∼10%. For LLMs of similar size but different types, e.g., Qwen1.5 and Vicuna, UPC also yields improvements. Besides, stronger base LLMs like original Qwen1.5-14B and Qwen1.5-32B show greater gains than Vicuna-33B. In summary, UPC works well across LLM categories and sizes, and brings larger gains on stronger base LLMs.

5.5 Case Study

In Appendix F, we provide test set samples to compare the performance of UPC with GPT-3.5-Turbo baseline. After the introduction of the user, UPC ask specific questions about the user's background, like "Innovative supply chain management solutions". When the user mentions blockchain technology, UPC asks about its implementation. The responses are highly interactive and relevant to the user's interests, keeping the user engaged in the discussion. In contrast, the dialogues of GPT-3.5 are more generic, with the chatbot relying on simple pleasantries like "How can I assist you today?" and failing to address the user's specific background. The responses are shallow, do not effectively explore the user's interests, and result in superficial dialogues that neither engage the user nor encourage further discussion. Thus, our UPC exhibits better user-oriented proactivity. Another case between our UPC and target-oriented method TRIP is provided in Appendix F.

6 Conclusion

We propose a User-oriented Proactive Chatbot to address the lack of user-oriented proactivity in open-domain dialogue. We first construct a critic to evaluate the user-oriented proactivity. Then we use the critic to guide dialogues between the chatbot and user agents, generating a corpus with enhanced user-oriented proactivity. We introduce a user background dataset ISCO-800 to ensure the diversity of user backgrounds. Finally, we train the chatbot with an iterative curriculum learning strategy to adapt to different users from easy to hard. Experiments demonstrate that our proposed training method is applicable to different LLMs, improving useroriented proactivity in open-domain dialogues. We hope to bring new insight into the improvement of dialogue quality for LLMs and human-computer interaction experience.

Acknowledgments

This work was supported in part by the National Key R&D Program of China under Grant No. 2022YFB4500600, in part by the Joint Funds of the National Natural Science Foundation of China under Grant No. U24A20327, in part by the Guangdong Provincial Key Laboratory of Human Digital Twin under Grant No. 2022B1212010004, and in part by Major Key Project of Peng Cheng Laboratory (PCL) under Grant No. PCL2023A08.

Contribution Statement

This work was a collaborative effort by all contributing authors. Yufeng Wang and Jinwu Hu made equal contributions to this study and are designated as co-first authors. Xiaofen Xing and Qingfang Zheng, serving as the corresponding authors, are responsible for all communications related to this manuscript.