“Think Before You Speak”: Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs

1 Introduction

Task-oriented Dialog Systems (TDS) assist users in completing specific tasks (e.g., restaurant booking) using natural language, and have been applied to various commercial services [Li et al., 2017; Zhang et al., 2020; Qin et al., 2021; Zhang et al., 2021]. In a TDS, the dialog policy, which decides the next system actions according to the current dialog context, plays a vital role as it dramatically influences the dialog efficiency (e.g., the conciseness and smoothness).

Prior work on dialog policy typically assumes that the agent only produces one action per turn [Mnih et al., 2015; Peng et al., 2018], which leads to lengthy dialogs and increases the risk of task failure. To address these weaknesses, some recent works have investigated Multi-Action Dialog Policy Learning (MADPL) that generates multiple actions concurrently as the current system response to boost the expressive power and efficiency of the TDS [Shu et al., 2019; Jhunjhunwala et al., 2020]. MADPL can be formulated as a multi-label classification or sequence generation problem, and thus solved by Supervised Learning (SL) based methods that imitate action combinations from labeled dialog samples [Shu et al., 2019].

∗Corresponding Author

Despite their success, existing SL-based methods often generalize poorly towards real-world human-machine dialog flows [Jhunjhunwala et al., 2020]. Task-oriented dialogs are known to share the "one-to-many" nature that various appropriate responses may exist under the same context [Huang et al., 2020]. Accordingly, in the multi-action setting, various action combinations could be considered appropriate under the same context, which are mostly not covered in limited human-labeled dialog datasets [Jhunjhunwala et al., 2020]. With straightforward imitation, existing SL-based methods will be restricted to a small subspace of the entire action space, leading to limited generalization ability towards unseen dialog scenarios.

Some existing studies try to address the above issue from three different perspectives: 1) Interactive Learning [Jhunjhunwala et al., 2020], which performs data augmentation through human-machine interactions; 2) Reinforcement Learning (RL) [Takanobu et al., 2019; Zhu et al., 2020], which enables learning from online human-interacting dialogs; and 3) Adversarial Learning (AL) [Takanobu et al.,

1 A dialog policy responses by predicting atomic dialog actions represented as "Domain-Intent(Slot)" phrases. We omit the domain ("restaurant") for clarity.

2019; Li et al., 2020], which provides extra supervision signals with a machine-learned discriminator. However, the performance gain is obtained at the cost of additional supervision. For example, they require tons of human work to label dialogs, build real-world environments such as user simulators, and design learning strategies. Moreover, RL and AL-based algorithms suffer from the unstable training process [Suzuki and Ogata, 2020; Tian et al., 2020], bringing extra deployment costs in real-world applications.

In this work, we present Planning Enhanced Dialog Policy (PEDP), an SL-based multi-task learning framework for improved MADPL. Unlike prior methods that directly imitate complex action combination patterns from limited multiaction dialog turns, we propose to further learn such patterns from single-action dialog fragments. We find that one turn in the multi-action dialog usually corresponds to several turns in the single-action setting (i.e., fragments) to achieve the similar (or identical) dialog state transition (see Fig. 1). This phenomenon inspires us to learn to simulate singleaction dialog fragments for enhancing multi-action prediction through discovering and incorporating contextually relevant dialog actions. To materialize this idea, we integrate model-based planning into the decision process and propose a Single-action Dialog Planning module. It plans several single-action paths (dialog fragments) based on the current dialog state before multi-action prediction. Each path looks ahead several turns through "self-play" between a discrete dialog policy model and a world model that imitates user behaviors. We further propose an Ensemble Prediction module that aggregates the multi-action prediction probabilities from multiple paths to reduce the impact of low-quality dialogs. Unlike seeking external data sources, our method better uses existing dialog samples. It performs multi-action prediction in a reasonable and explainable way, thus boosting performance towards unseen dialog flows.

To evaluate the proposed method, we conduct experiments on the MultiWOZ [Budzianowski et al., 2018] and the SGD [Rastogi et al., 2020] datasets. The experimental results demonstrate that the extension of planning brings significant improvement to our model's performance and dialog efficiency, resulting in a promising task success rate of 90.6%, improving 3% compared to the state-of-the-art methods.

2 RelatedWork

Multi-Action Dialog Policy Learning (MADPL). Recent efforts have been made to learn a multi-action dialog policy for task-oriented dialogs where the agent can produce multiple actions per turn to expand its expressive power. Some attempts [Zhu et al., 2020; Gordon-Hall et al., 2020] (primarily DQN-based methods) incorporate the top-k most frequent dialog action combinations in the dataset into the output space and simplify MADPL as a multi-class classification problem. However, the expressive power and flexibility of the dialog agent are severely limited. Considering the sequential dependency among the acts, Shu et al. (2019) proposes to formulate MADPL as a sequence generation problem. However, the order in which people express their intentions is complicated and has not been properly annotated in the existing human-to-human dialog datasets 2 . The discrepancy between the task setting and data annotation can lead to misleading performance. Recent works [Shu et al., 2019; Li et al., 2020] typically cast MADPL as a multi-label classification problem and address it with supervised learning. However, the complex action combinations can exponentially enlarge the output space [Jhunjhunwala et al., 2020]. The limited coverage of the output action space of the existing dialog corpus will greatly hinder SL-based methods' performance. Jhunjhunwala et al. (2020) addresses this problem by data augmentation through interactive learning. The problem can also be partially alleviated through RL-based [Zhu et al., 2020] and AL-based methods [Takanobu et al., 2019; Li et al., 2020] by allowing the model to explore more potential dialog scenarios. Unlike existing works that "memorize" action combinations from limited multi-action samples, we mimic the human decision process and discover dialog actions to express before condensing them into a brief response through simulating single-action dialogs.

Planning in Dialog Policy Learning. Planning refers to the process that uses a model of the environment to improve a policy. Peng et al. (2018) first introduces background planning for task-completion dialog policy learning. They propose to use Deep Dynamic Q-network, in which a world model is trained to mimic real-world users and provide extra simulated experience for training. The integration of planning can substantially improve data efficiency. Since the world model is not perfect and may generate erroneous experiences (state-action pairs) that hinder policy learning. Su et al. (2018) proposes to train a discriminator to filter out low-quality simulated experiences. Wu et al. (2019) design a switcher-based mechanism to balance the use of real and simulated experiences automatically. Wang et al. (2020) proposes decision-time planning that adopts a rollout algorithm (Monte-Carlo Tree Search) to boost the accuracy of action value estimations. In summary, previous work has focused on RL-based single-action dialog policy learning. Instead, we address SL-based multi-action dialog policy learning and extend planning to generate anticipatory single-action dialog trajectories for further combination.

3 Methodology

In this section, we introduce the Planning-Enhanced Dialog Policy (PEDP) framework, an SL-based multi-task learning framework for improved MADPL.

3.1 Task Formulation

We regard MADPL as a multi-label classification task, i.e., given the current dialog state that summarizes the dialog history, we want to predict a macro-action that is a set of independent atomic dialog actions that serves as the current system response. Following [Li et al., 2020], each atomic dialog action is a concatenation of domain name, action type, and slot name, e.g., "hotel-inform-area".

2 The sequential order of the actions in the annotated samples is usually inconsistent with the expressive order in text responses.

3.2 Overview

The PEDP framework is inspired by the human dialog strategy of "think before you speak" that conceives relevant content to express before deciding the current response. We first discover contextually relevant dialog actions by looking ahead several turns under the single-action dialog dynamics (i.e., Single Action Dialog Planning) and then use the planning result as guidance to predict the atomic dialog actions that should be executed concurrently (i.e., macro-action).

Formally, let st denote the current dialog state representing the dialog history up to the current dialog turn. We encode st into a dialog state embedding ht. Given the current state embedding ht, we plan K independent single-action dialog paths, and the k-th dialog path is represented by a vector h(k) , k = 1, . . . ,K. Our model then decodes each dialog path to a probability distribution over atomic dialog actions, i.e., p(k) . Finally, these distributions {p(k) }Kk=1 are aggregated to form a unified distribution, from which atomic dialog actions in the macro-action At are sampled.

Figure 2 depicts the architecture of PEDP, which mainly consists of four modules: 1) State Encoding, which encodes the dialog state into a compact vector; 2) Single-Action Dialog Planning, which plans several independent dialog paths with each one looking ahead several steps considering only the single-action dialog dynamics; 3) Path-wise Decoding, which predicts the multi-action probability distribution for each dialog path; and 4) Ensemble Prediction, which aggregates the probabilities and samples the final macro-action. In what follows, we elaborate on each of these modules in PEDP.

3.3 State Encoding

The dialog state is a summary of the dialog history. Inspired by [Li et al., 2020], we propose to include the following information into dialog state st at time step t: 1) embedding of returned entities for a query, 2) the last user action, 3) the last system action, and 4) the belief state that contains the requested slots and informed slots of user and agent.

To densify the dialog state, we apply a Fully-connected Feed-forward Network (FFN) to obtain a compact dialog state embedding ht. The FFN consists of two linear transformations with a ReLU activation in the middle layer, i.e.,

In what follows, this dialog state embedding ht will serve as the initial dialog state embedding for planning.

3.4 Single-Action Dialog Planning

As discussed in the introduction, the straight-forward multilabel imitation of action co-occurrence suffers from the poor generalization ability issue due to the lack of labeled multiaction training data. To address this issue, we propose a single-action dialog planning module that further leverages single-action dialog samples to model contextual action cooccurrence patterns. It looks ahead several steps under singleaction dialog dynamics to predict the next dialog state, which will serve as the additional information to enhance macroaction prediction. In what follows, we describe how to model the dialog state transition in a single-action dialog.

Given the initial dialog state embedding ht at the current dialog turn t, we plan K independent single-action dialogs to discover contextually related atomic dialog actions. In each dialog, a discrete policy model and a world model interact to look ahead several steps. Let h(k) t,n denote the dialog state embedding at the n-th step in the k-th dialog for n = 0, . . . , Nk where Nk is the length of the k-th dialog, and h(k) t,0 = ht, h(k) t,Nk = h(k) . The last dialog state embedding h(k) estimates the hidden vector of the future dialog state st+1 and summarizes the planned single-action dialog. In what follows, we describe how to plan a single step from h(k) t,n to obtain h(k) t,n+1 . To simplify the notations, we omit the dialog index k and dialog turn t.

Given the dialog state embedding hn at step n, we want first to predict a single action an and then use it to update hn to obtain hn+1. This can be achieved by cyclical interactions between a discrete policy model (DP) that predicts an atomic dialog action under the current-step dialog state and a world model that simulates user behavior to update the current-step dialog state, i.e.,

Here, DP is implemented as a single linear layer followed by a Gumbel-Softmax function [Jang et al., 2016] parameterized by τd. The Gumbel-Softmax function draws an atomic dialog action sample from a categorical distribution, diversifying the planned dialogs. τd is selected to balance the approximation bias and the magnitude of gradient variance. The world model is implemented using a GRU to model dialog state transitions, and Emb(an) denotes the embedding vector of atomic dialog action an.

We observe that both executing a macro-action and sequentially executing the corresponding atomic dialog actions will lead to a similar (or identical) state transition (see Fig. 1). Accordingly, we assume that the planned information is adequate once such a state transition is achieved, and propose to determine if to stop planning by comparing the initial and the next dialog state embeddings, i.e., h0 and hn+1 (see Fig. 2b). This is also modeled using a neural network, i.e.,

where cn is a binary variable, ":" denotes vector concatenation, and FFN is a 2-layer fully-connected feed-forward network using the ReLU activation function in the middle layer.

Note that the single-action dialog planning module estimates dialog state transitions in the dialog state embedding space from h0 to hN, and it demands supervision from the labeled structural dialog state transition samples from st to st+1. To enable model training, we incorporate a recovery model for mapping the state embeddings back to the structured state-space. Formally, we have,

Here, Recover is implemented by a 2-layer FFN and is only used during the training stage.

3.5 Path-wise Decoding

Leveraging the planned single-action dialog, we predict the probability distribution over atomic dialog actions.

Given the initial and the last dialog state embeddings h0 and h(k) , we decode the multi-action probability distribution p(k) with a neural network applied to each path separately and identically. Since our task formulation posits no dependency among atomic dialog actions, following [Li et al., 2020], we perform binary classification for each specific action in the atomic dialog action space independently to decide whether it is selected. Specifically, we instantiate the decoder p(k) = [p(k) 1 : . . . : p(k) M ], where k refers to the planned path and M is the size of the action space. Each p(k) m ,m = 1, . . . ,M is a vector computed as:

Future efforts at exploring different decoders may further improve the performance.

3.6 Ensemble Prediction

The performance of the proposed model is affected by the quality of the planned dialogs. To reduce the impact of low-quality dialogs, we propose to ensemble the multi-action probability distributions with an aggregation function, i.e.,

where Aggr(·) is the mean average in our case.

Task-oriented dialogs share a critical "one-to-many" nature that different responses may be proper under the same context, making it essential to incorporate stochastic factors to build a multi-action dialog policy. We empirically verified that this stochasticity could be achieved with probability sampling. In our case, we apply the Gumbel-Sigmoid function to sample the macro-action, that is,

Here GumbelSigmoid(·) is a modification of the GumbelSoftmax function, regarding sigmoid as a softmax with two logits p and 0. τ denotes the temperature factor, g1 and g2 are Gumbel noises.

3.7 Training

We use a multi-task objective to train all the modules jointly.

Task 1: Discrete Act Prediction (DAP) For the planned single-action sequence a = (a0, . . . , aN-1), we aim to maximize the log-likelihood (MLE) for the joint probability p(a|h0), which can be factorized as:

where θ and ϕ denotes trainable parameters for the discrete dialog policy model and the world model, respectively.

Task 2: Stop Flag Prediction (SFP) We define the objective of predicting the stop flag sequence c = (c0, . . . , cN-1) as MLE for the joint probability p(c|h0), which can be factorized as:

where γ parameterizes the stop prediction model, the joint probability of pϕ,θ(hn+1|hn) is factorized as pϕ(hn+1|an, hn)pθ(an|hn) of state transition and discrete act prediction.

Task 3: State Recovery (SR) We consider a state recovery objective to supervise state encoding and state transition. Precisely, we predict the current dialog state st and next dialog state st+1 with the initial dialog state embedding h0 and the last state embedding hN, respectively, i.e.,

where η and ζ denotes trainable parameters for state encoder and the Recover, respectively. The joint probability pϕ,θ(hn+1|hn) is the same as explained in Task 2.

Task 4: Multi-Action Prediction (MAP) Finally, we introduce a multi-action prediction objective to supervise all the modules in our PEDP framework. That is:

where ω denotes trainable parameters for the decoder. The rest is the same as explained in Task 3.

Full Objective The overall loss is the weighted sum of four losses: two cross-entropy losses of DAP and SFP and two binary cross-entropy losses of SR and MAP 3 .

4 Experiments

4.1 Settings

Evaluation Metrics We perform the automatic evaluation from two perspectives:

1) Interactive Evaluation, which evaluates task completion quality through dialog interaction with the user simulator. We use Inform F1, Inform Recall, Match, Turn, and Success as evaluation metrics. Inform F1 and Inform Recall evaluate whether all the requested information (e.g., hotel address) has been provided. Match evaluates whether the booked entities match the indicated constraints (e.g., the cheapest restaurant in town). Turn and Success evaluate the efficiency and level of task completion of dialog agents, respectively. Specifically, a dialog is considered successful if all the requested information is provided (i.e., Inform Recall = 1) and all the entities are correctly booked (i.e., Match = 1).

2) Standard Evaluation, which measures the generalization ability with the test set of the training corpus. We report the sample-wise Precision, Recall, and F1 score of the macroactions, where each atomic dialog action is assumed to be correct if it is included in the ground truth.

Datasets We conduct experiments on two dialog datasets.

1. MultiWOZ [Budzianowski et al., 2018] is a multidomain dialog dataset with 10, 438 dialogs covering 7 distinct domains and 24 slots. We apply the agenda-based user simulator as the interaction environment for interactive evaluation.

2) SGD [Rastogi et al., 2020] is published to build scalable multi-domain conversational agents. It has 16, 142 dialogs spanning 16 domains and 214 slots. We conduct a standard evaluation on SL-based methods to address the scaling concerns. We do not report interactive evaluation results since the corresponding official user simulator is unavailable.

Baselines Three types of baselines are explored: 1) Supervised Learning (SL), 2) Reinforcement Learning (RL), and 3) Adversarial Learning (AL). We ignore DQN-based methods because these methods do not suit the multi-label classification setting in our task formulation (see Sec.2 for details).

3 We refer the reader to Appendix A and B for data usage and model implementation details.

• (SL) DiaMultiClass [Li et al., 2020] is a FFN with Sigmoid as the last activation function.
• (SL) DiaSeq [Li et al., 2020] encodes the multi-hot dialog state vector with a 2-layer FFN and sequentially decodes the macro-action with an RNN.
• (SL) DiaMultiDense [Li et al., 2020] performs binary classification for each specific action with the same network described in Section 3.7.

(SL) gCAS [Shu et al., 2019] is a novel DiaSeq variant that uses tangled RNNs to predicts the macro-action by an autoregressive generation of intent and slots.

• (RL) GP-MBCM [Gaˇsi´c et al., 2015] is a Gaussian process based Multi-domain Bayesian Committe Machine that decomposes the policy into domain-specific policies.
• (RL) ACER [Wang et al., 2016] is the Actor-Critic RL policy with Experience Replay. ACER is sample efficient and is well adapted for large discrete action spaces.
• (RL) PPO [Schulman et al., 2017] denotes Proximal Policy Optimization, a simple and stable RL algorithm that applies a constant clipping mechanism.
• (AL) ALDM [Liu and Lane, 2018] is the Adversarial Learning Dialog Model. ALDM estimates a reward at the end of the session to optimize the dialog policy.
• (AL) GDPL [Takanobu et al., 2019] equips PPO with a "rewarder" for joint reward estimation and policy optimization. GDPL is regarded as the SOTA method.
• (AL) DiaAdv [Li et al., 2020] further improves the DiaMultiDense model with adversarial fine-tuning.

4.2 Model Evaluation

Interactive Evaluation We report the performance of each approach that interacts with the user simulator in Table 1.

We observe that our PEDP framework achieves the highest performance in the task success by 3% compared to the second-best model DiaAdv and by 7.4% compared to the SOTA GDPL method on account of the substantial improvement in match rate and inform recall. This demonstrates that PEDP can effectively incorporate more task-relevant information to enrich the response and lead to task success.

4 Results reused from [Li et al., 2020]

We also observe that PEDP performs moderately in terms of average turn. A possible reason is that the model fails to generate denser dialog action combinations. As shown in Table 1, PEDP reaches a comparable F1 score and a higher recall, indicating a lower precision score and implying the incorporation of redundant information that does not contribute to task completion. We report how this affects user experience in the human evaluation.

Standard Evaluation We report the standard performance of the state-of-the-art GDPL and DiaAdv and all the SL-based methods in Table 2.

We observe that our PEDP framework outperforms the others in terms of all evaluation metrics. For the F1 score, PEDP achieves 8.37 ∼ 32.74% improvements on account of the substantial improvement in precision (6.03 ∼ 26.89%) and recall (1.31 ∼ 36.78%) over the baselines.

We also observe that the standard and interactive performance are negatively correlated for the baseline methods (see Table 1). We attribute this to the fact that standard and interactive objectives are fundamentally in conflict. While multiple action combinations can be considered appropriate during dialog interaction, they may not match the ground truth in the test samples. Unlike existing methods, PEDP achieves the best of both worlds, demonstrating our method's effectiveness on MADPL.

Ablation Study We conduct ablation experiments to investigate the effects of two components in our model (Singleaction Dialog Planning and Ensemble Prediction) and study how stochastic factors affect model performance (action sampling). We merge the results in Table 1 and 2.

The results show that: 1) The single-action dialog planning module contributes more significantly for performance improvement while the ensemble prediction module effectively stabilizes the model and leads to a lower standard deviation, and please see Appendix C for hyper-parameter sensitivity experiments for the number of paths K. 2) Sampling usually leads to a performance gain of match rate and inform recall but reduces the inform precision, indicating the incorporation of redundant information in the response. For GDPL, sampling harms the performance. A possible reason is that GDPL, which learns from the user simulator for evaluation, has adapted to the simulated dialog scenarios. Sampling instead may lead to sub-optimal decisions.

Human Evaluation Automatic evaluation only measures part of the agent's performance (e.g., Success for the level of task completion). It may fail to consider other aspects for assisting real users (e.g., inappropriate reply). Thus, we conduct a human study to fully evaluate system performance. Our PEDP method is compared with three state-of-the-art dialog policy models: GDPL, DiaAdv, and DiaSeq (reported to offer an elegant user experience [Li et al., 2020]). Following [Takanobu et al., 2019; Li et al., 2020], for each comparison pair, we randomly sample 100 user goals and present the generated dialog samples from the two agents. The dialog actions are translated into human-readable utterances with the language generation module from ConvLab [Zhu et al., 2020]. We hire nine postgraduates as judges to select the dialogs that provide a better user experience (each sample is graded nine times rather than three in existing works [Takanobu et al., 2019; Li et al., 2020]), "Tie" can also be selected for the equally performed ones. We refer the reader to Appendix D and E for the case study and the corresponding error analysis.

In Table 3, we see that our PEDP framework outperforms the baseline models. As mentioned in robust evaluation (see Section 5.4), our PEDP model shares a lower inform precision (0.79) than DiaSeq (0.82) and GDPL (0.87), which indicates the incorporation of redundant information. However, we find that the performance gap does not severely harm the user experience. A possible reason is that human users tend to focus on goal-related information during task-oriented dialog. Compared to dealing with redundant information, they are more intolerant towards the failure of providing the requested information.

5 Conclusion

This paper presents a supervised learning-based PEDP framework, which incorporates model-based planning for conceiving what to express before deciding the current response through simulating single-action dialogs. With multi-task learning of single-action dialog dynamics and multi-action prediction, our method fully uses the training corpus and makes reasonable and explainable decisions, thus improving performance towards unseen task-oriented human-machine dialog flows. Extensive experiments show that our method significantly and consistently outperforms state-of-the-arts with a favorable task success rate of 90.6%.

Acknowledgments

This work was supported in part by National Natural Science Foundation of China (61902305, 61922067), Shenzhen Basic Research Grant (JCYJ20170816100819428), and MoECMCC "Artifical Intelligence" Project (MCM20190701).