Learning to Select Knowledge for Response Generation in Dialog Systems

1 Introduction

End-to-end neural generative models attract much attention as a potential solution to open-domain dialogue systems. The sequence-to-sequence (Seq2Seq) model [Shang et al., 2015; Vinyals and Le, 2015; Cho et al., 2014b] has achieved success in generating fluent responses. However, it tends to produce less informative responses, such as "I don't know" and "That's cool", resulting in less attractive conversations.

Variety of improvements [Zhou et al., 2018; Ghazvininejad et al., 2018; Liu et al., 2018] have been proposed toward informative dialogue generation, by leveraging external knowledge, including unstructured texts or structured data such as knowledge graphs. For example, the commonsense model proposed in [Zhou et al., 2018] took commonsense knowledge into account, which is served as knowledge background to facilitate conversation understanding. The recently created datasets Persona-chat [Zhang et al., 2018] and Wizardof-Wikipedia [Dinan et al., 2018] introduced conversationrelated knowledge (e.g., the personal profiles in Persona-chat) in response generation where knowledge is used to direct conversation flow. Dinan et al. [2018] used ground-truth knowledge to guide knowledge selection, which demonstrates improvements over those not using such information. However, ground-truth knowledge is difficult to obtain in reality.

Most of existing researches focused on selecting knowledge based on the semantic similarity (e.g., graph attention [Zhou et al., 2018]) between input utterances and knowledge. This kind of semantic similarity is regarded as a prior distribution over knowledge. However, a prior distribution cannot effectively guide appropriate knowledge selection since different knowledge can be used to generate diverse responses for the same input utterance. In contrast, given a specific utterance and response pair, the posterior distribution over knowledge, which is inferred from both the utterance and the response (instead of the utterance only), can provide effective guidance on knowledge selection since the actual knowledge used in the response is considered. The discrepancy between the prior and posterior distributions brings difficulties in the learning process: the model could hardly select appropriate knowledge simply based on the prior distribution and without response information, it is difficult to obtain the correct posterior distribution during the inference process. This kind of discrepancy would stop the model from learning to generating proper responses by utilizing appropriate knowledge.

The problems caused by this discrepancy are illustrated in Table 1, which is a dialogue from [Zhang et al., 2018]. In this dataset, each agent is associated with a persona profile, which is served as knowledge. Two agents exchange information based on the associated knowledge. Given an utterance, different responses can be generated depending on whether appropriate knowledge is used. R1 utilizes no knowledge and thus ends up in a less informative response, while other responses are more informative since they incorporate external knowledge. However, among the knowledge, both K1 and K3 are relevant to the utterance. If we simply select knowledge based on the utterance (i.e, prior information) without knowing that K3 is used in the true response (i.e., posterior information), it is difficult to generate a proper response since appropriate knowledge might not be selected. If the model is trained by selecting wrong knowledge (e.g., K2 in R2) or knowledge irrelevant to the true response (e.g., K1), it can be seen that they are completely useless since they cannot provide any helpful information. Note that it is also important to properly incorporate knowledge in response generation. For example, though R3 selects correct knowledge K3, it results in a less relevant response due to inappropriate usage of knowledge. Only R4 makes appropriate selection of knowledge and incorporates it properly in generating responses.

To tackle the aforementioned discrepancy, we propose to separate the posterior distribution from the prior distribution. In the posterior distribution over knowledge, both utterances and response are utilized, while the prior distribution works without knowing responses in advance. Then, we try to minimize the distance between them. Specifically, during the training process, our model is trained to minimize the KL divergence between the prior distribution and the posterior distribution so that our model can approximate the posterior distribution accurately using the prior distribution. Then, during the inference process, the model samples knowledge merely based on the prior distribution (i.e., without any posterior information) and incorporates the sampled knowledge into response generation. It is proved that through this process, the model can effectively learn to generate proper and informative responses by utilizing appropriate knowledge.

The contributions of this paper can be summarized below:

• We clearly state and analyze the discrepancy between the prior and posterior distributions over knowledge in knowledge-grounded dialogue generation, which has not been sufficiently studied in the previous work.
• We propose a novel neural model which separates the posterior distribution from the prior distribution. We prove that our knowledge selection mechanism is effective for appropriate response generation.
• Our comprehensive experiments demonstrate that our model significantly outperforms the existing ones by incorporating knowledge more properly and generating appropriate and informative responses.

2 Model

In this paper, we focus on training a neural model with an effective knowledge selection mechanism. Given an utterance X = x1x2 . . . xn (xt is the t-th word in X) and a collection of knowledge {Ki}Ni=1 (where the ground-truth knowledge information is unknown), the goal is to select appropriate knowledge from the collection and to generate a response Y = y1y2 . . . ym by incorporating the selected knowledge.

2.1 Architecture Overview

The architecture overview of our model is presented in Figure 1 and it consists of four major components:

• The utterance encoder encodes X into an utterance vector x, and feeds it into the knowledge manager.
• The knowledge encoder takes as input each knowledge Ki and encodes it into a knowledge vector ki. When response Y is available, it also encodes Y into a vector y.
• The knowledge manager consists of two sub-modules: a prior knowledge module and a posterior knowledge module. Given the previously encoded x and {ki}Ni=1 (and y if available), the knowledge manager is responsible to select an appropriate ki and feeds it (together with an attention-based context vector ct) into the decoder.
• The decoder generates responses based on the selected knowledge ki and the attention-based context vector ct.

2.2 Encoder

We implement the utterance encoder using a bidirectional RNN with a gated recurrent unit (GRU) [Cho et al., 2014a], which consists of two parts: a forward RNN and a backward RNN. Given utterance X = x1 . . . xn, the forward RNN reads X from left to right and then, obtains a left-to-right hidden state -→h t for each xt while the backward RNN reads X in a reverse order and similarly, obtains a right-to-left hidden state ←- h t for each xt. These two hidden states are concatenated to form an overall hidden state ht for xt:

where [·; ·] represents a vector concatenation. To obtain an encoded vector x for utterance X, we utilize the hidden states and define x = [-→hT ; ←- h 1]. This vector will be fed into the knowledge manager to facilitate knowledge selection and it will also serve as the initial hidden state of the decoder.

Our knowledge encoder follows the same structure as the utterance encoder, but they do not share any parameters. Specifically, it encodes each knowledge Ki (and response Y if available) into a vector ki (and y, respectively) using a bidirectional RNN and uses it later in the knowledge manager.

2.3 Knowledge Manager

Given the encoded utterance x and the encoded knowledge collection {ki}Ni=1 , the goal of the knowledge manager is to select an appropriate ki. When the response y is available, the model also utilized it to obtain ki. The knowledge manager consists of two sub-modules (see Figure 2): a prior knowledge module and a posterior knowledge module.

In the prior knowledge module, we define a conditional probability distribution over knowledge, denoted by p(k|x):

Intuitively, we use the dot product (i.e., attention [Bahdanau et al., 2014]) to measure the association between ki and the input utterance x. A high association means that ki is relevant to x and thus, ki is likely to be selected. Note that p(k|x) is conditioned only on x and thus, it is a prior distribution over knowledge since it works without knowing the response. However, there can be different knowledge that are relevant to the utterance, and thus, it is difficult to select knowledge simply based on the prior distribution in training.

Motivated by this, in the posterior knowledge module, we define a posterior distribution over knowledge, denoted by p(k|x, y), by considering both utterances and responses:

where MLP(·) is a fully connected layer. Compared with the prior distribution, the posterior distribution is sharp since the actual knowledge used in the true response Y can be captured.

According to the distributions defined above, we sample knowledge using Gumbel-Softmax re-parametrization [Jang et al., 2016] (instead of the exact sampling) since it allows back propagation in non-differentiable distributions. Specifically, in the training process, knowledge is sampled based on the posterior distribution, which is inferred from the true response, and thus it is more likely to obtain appropriate knowledge via this distribution. In the inference process, the posterior distribution is unknown since responses are not available. Thus, knowledge is sampled based on the prior distribution.

Clearly, the discrepancy between prior and posterior distributions introduces great challenges in training the model: it is desirable to select knowledge based on the posterior distribution, which, however, is unknown during inference. In this paper, we propose to approximate the posterior distribution using the prior distribution so that our model is capable to select appropriate knowledge even without posterior information. For this purpose, we introduce an auxiliary loss, namely the Kullback-Leibler divergence loss (KLDivLoss), to measure the proximity between the prior distribution and the posterior distribution. The KLDivLoss is defined to be

where θ denotes the model parameters.

When minimizing KLDivLoss, the posterior distribution p(k|x, y) can be regarded as labels and our model is instructed to use the prior distribution p(k|x) to approximate p(k|x, y) accurately. As a consequence, even when the posterior distribution is unknown in the inference process (since the actual response Y is unknown), the prior distribution p(k|x) can be effectively utilized to sample appropriate knowledge so as to generate proper responses. To the best of our knowledge, it is the first neural model, which incorporates the posterior distribution as guidance, enabling accurate knowledge lookups and high quality response generation.

2.4 Decoder

The decoder generates response word by word sequentially by incorporating the selected knowledge ki. We introduce two variants of decoders. The first one is a "hard" decoder with a standard GRU and the second one is "soft" decoder with a hierarchical gated fusion unit [Yao et al., 2017].

Standard GRU with Concatenated Inputs

Let st-1 be the last hidden state of the decoder, yt-1 be the word generated in the last step and ct be an attention-based context vector of the encoder (i.e., ct = ∑ni=1 αt,ihi where αt,i measures the relevancy between st-1 and the hidden state hi of the encoder). The hidden state of the decoder at time t:

where we concatenate yt-1 with ki. This decoder is said to be a hard decoder since ki is forced to participate in decoding.

Hierarchical Gated Fusion Unit (HGFU)

HGFU provides a softer way to incorporate knowledge and it consists of three major components, namely an utterance GRU, a knowledge GRU and a fusion unit.

The former two components follow the standard GRU structure, which produce hidden representations for the last generated yt-1 and the selected knowledge ki, respectively:

Then, the fusion unit combines them to produce the hidden state of the decoder at time t [Yao et al., 2017]:

where r = σ(Wz[tanh(Wysyt ); tanh(Wkskt )]) and Wz, Wy and Wk are parameters. Intuitively, the gate r controls the contributions of syt and skt to the final hidden state st, allowing a flexible knowledge incorporation schema.

After obtaining the hidden state st, the next word yt is generated according to the following probablity distribution:

2.5 Loss Function

Apart from the KLDivLoss, two additional loss functions are used in our model: the NLL loss captures the word order information while the BOW loss captures the bag-of-word information. All loss functions are also elaborated in Figure 2.

The objective of NLL loss is to quantify the difference between the true response and the response generated by our model. It minimizes Negative Log-Likelihood (NLL):

where θ denotes the model parameters and y<t denotes the previously generated words.

The BOW loss is adapted from [Zhao et al., 2017] to ensure the accuracy of the sampled knowledge ki by enforcing the relevancy between the knowledge and the true response. Specifically, let w = MLP(ki) ∈ R|V | where |V | is the vocabulary size and we define p(yt|ki) = exp(wyt ) ∑v∈V exp(wv) . Then, the BOW loss is defined to minimize

In summary, unless specified explicitly, the total loss of a given a training example (X, Y, {Ki}Ni=1 ) is

3 Experiments

3.1 Dataset

We conducted experiments on two recently created datasets, namely the Persona-chat dataset [Zhang et al., 2018] and the Wizard-of-Wikipedia dataset [Dinan et al., 2018].

In Persona-chat, each dialogue was constructed from a pair of crowd-workers, who chat to know each other. To produce meaningful conversations, each worker was assigned a persona profile, describing their characteristics, and this profile serves as knowledge in the conversation. There are 151,157 turns (each turn corresponds to an utterance and a response pair) of conversations in Persona-chat, which we divide into 122,499 for train, 14,602 for validation and 14,056 for test. The average size of a knowledge collection (the average number of sentences in a persona profile) in this dataset is 4.49.

Wizard-of-Wikipedia is a chit-chatting dataset between two agents on some chosen topics. One of the agent, also known as the wizard, plays the role of a knowledge expert and has access to a retrieval system for acquiring knowledge. The other agent acts as a curious learner. From this dataset, 79,925 turns of conversations are obtained and 68,931/3,686/7,308 of them are used for train/validation/test. The test set is split into two subsets, Test Seen and Test Unseen. Test Seen contains 3,619 turns of conversations on some overlapping topics with the training set, while Test Unseen contains 3,689 turns on topics never seen before in train or validation. Note that in this paper, we focus on the scenarios where ground-truth knowledge is unknown. Thus, we did not use the ground-truth knowledge information provided in this dataset. The average size of a knowledge collection accessed by the wizard is 67.57.

3.2 Models for Comparison

We implemented our model, namely the Posterior Knowledge Selection (PostKS) model, for evaluation. In particular, two variants of our model were implemented to demonstrate the effect of different ways of incorporating knowledge:

• PostKS(concat): the hard knowledge-grounded model with a GRU decoder where knowledge is concatenated.
• PostKS(fusion): the soft knowledge-grounded model where knowledge is incorporated with a HGFU.

We compared our models with three baselines:

• Seq2Seq: an attention Seq2Seq that does not have access to external knowledge [Vinyals and Le, 2015].
• MemNet(hard): a memory network from [Ghazvininejad et al., 2018], where knowledge is sampled based on prior semantic similarity and fed into the decoder.
• MemNet(soft): a soft knowledge-grounded model from [Ghazvininejad et al., 2018], where knowledge is stored in memory units that are decoded with attention.

In our adaption of all baselines, we used the same RNN encoder/decoder as PostKS. Among them, Seq2Seq is compared for demonstrating the effect of introducing knowledge in response generation while MemNet based models, which also have access to knowledge, are compared to verify that the effectiveness of our novel knowledge selection mechanism.

3.3 Implementation Details

Our encoders and decoders have 2-layer GRU structures with 800 hidden states for each layer, but they do not share any parameters. We set the word embedding size to be 300 and initialized it using GloVe [Pennington et al., 2014]. The vocabulary size is 20,000. We used the Adam optimizer with a mini-batch size of 128 and the learning rate is 0.0005.

We trained our model with at most 20 epochs on a P40 machine. In the first 5 epochs, we minimize the BOW loss only for pre-training the knowledge manager. In the remaining epochs, we minimize over the sum of all losses. After each epoch, we save a model and the model with the minimum loss is selected for evaluation. Our models and datasets are all available online: https://github.com/ifr2/PostKS.

3.4 Automatic and Human Evaluation

We adopted several automatic metrics to perform evaluation and the result is summarized in Table 2. Among them, BLEU1/2/3 and Distinct-1/2 are two widely used metrics for evaluating the quality and diversity of generated responses. Knowledge R/P/F1 is a metric adapted from [Dinan et al., 2018], which measures the unigram recall/precision/F1 score between the generated responses and the knowledge collection. Specifically, given the set of non-stopwords in Y and in the knowledge collection {Ki}Ni=1 , denoted by WY and WK, we define Knowledge R(ecall) and Knowledge P(recision) to be

and Knowledge F1 = 2 · Recall·Precision Recall+Precision .

As shown in Table 2, our models outperform all baselines significantly (p < 0.00001) by achieving the highest scores in most of the automatic metrics. Specifically, compared with Seq2Seq, incorporating knowledge is shown to be helpful in generating diverse responses. For example, Distinct-1/2 on Persona-chat is increased from 0.026/0.074 (Seq2Seq) to 0.048/0.126 (PostKS(concat)), meaning that the diversity is greatly improved by augmenting with knowledge. Besides, when comparing with existing knowledge-grounded baselines, our models demonstrate their ability on incorporating appropriate knowledge in response generation. In particular, comparing PostKS(fusion) against MemNet(soft) on Persona-chat (they are soft knowledge-grounded models except that we use both prior and posterior information to facilitate knowledge selection), we achieve higher BLEU and Distinct scores. This is because that the posterior information is better utilized in our models to provide effective guidance on obtaining appropriate knowledge, resulting in responses with better quality. Compared with knowledge selection on Persona-chat, selecting appropriate knowledge on Wizard-ofWikipedia is more challenging due to a larger knowledge collection size. Nevertheless, our models perform consistently better than most baselines. For example, PostKS(fusion) has higher knowledge R/P/F1 compared with all MemNet based models on Wizard-of-Wikipedia, indicating that it can not only select appropriate knowledge, but also ensure that knowledge is better incorporated in the response generated. Finally, we observe that PostKS(fusion) performs slightly better than PostKS(concat) in most cases. This verifies that soft knowledge incorporation is a better way of introducing knowledge to response generation since it allows for more flexible knowledge integration and less sensitivity to noise.

In human evaluation, three annotators were recruited to rate the overall quality of the responses generated by each model. The rating ranges from 0 to 2, where 0 means that the response is completely irrelevant, 1 means that the response is acceptable but not very informative, and 2 means that the response is natural, relevant and informative. We randomly sampled 300 responses for each model on each dataset, resulting in 4,500 responses in total for human annotation. We reported the average rating in Table 2. The agreement ratio (Fleiss' kappa [Fleiss, 1971]) is 0.48 and 0.41 on Personachat and Wizard-of-Wikipedia, showing moderate agreement. According to the result, both of our models, PostKS(concat)

and PostKS(fusion), are remarkably better than all existing baselines in terms of human rating, demonstrating the effectiveness of our novel knowledge selection mechanism.

3.5 Case Study

Table 3 shows two example responses. For the lack of space, we only display three pieces of knowledge on each dataset. In the example from Persona-chat, the utterance is asking whether the agent likes music. Without access to external knowledge, Seq2Seq produces a bland response which does not contain any useful information. MemNet(hard) tries to incorporate knowledge, but, unfortunately, it selects the wrong knowledge, leading to an irrelevant response about broccoli rather than music. The remaining models generate responses with the help of the correct knowledge. Among them, our PostKS(fusion) and PostKS(concat) models perform better since they are more specific by mentioning exactly the rock music. In particular, our soft knowledge-grounded model, PostKS(fusion), performs noticeably well since it does not only answer questions, but also raises a relevant question about the favorite band, allowing evolving conversations. The example fromWizard-of-Wikipedia is about country music. Similar to Persona-chat, our models enjoy superior performance by producing informative and relevant responses.

3.6 Further Evaluation of Knowledge Selection

To further verify the effectiveness our knowledge selection mechanism, we apply it on the best performing Transformer model, Lost in Conversation (LIC), in ConvAI2 NeurIPS competition [Dinan et al., 2019] and the result is reported in Table 4 (following [Dinan et al., 2018], Perplexity (PPL) is reported). After integrating our mechanism, all metrics are greatly improved. In particular, we achieve a threefold improvement on knowledge R/P/F1 on Persona-chat, which verifies the usefulness of our knowledge selection mechanism in incorporating knowledge in response generation.

4 RelatedWork

The success of Seq2Seq motivates the development of various techniques for improving the quality of generated responses. Examples include diversity promotion [Li et al., 2016] and unknown words handling [Gu et al., 2016]. However, the problem of tending to generate generic words still remains since they do not have the access to external information.

Recently, knowledge incorporation is shown to be an effective way to improve the performance of neural models. Long et al. [2017] obtained knowledge from texts using a convolutional network. Ghazvininejad et al. [2018] stored texts as knowledge in a memory network to produce more informative responses. A knowledge diffusion model was also proposed in [Liu et al., 2018], where the model is augmented with divergent thinking over a knowledge base. Large scale commonsense knowledge bases were first utilized in [Zhou et al., 2018] and many domain-specific knowledge bases were also considered to ground neural models with knowledge [Xu et al., 2017; Zhu et al., 2017; Gu et al., 2016].

However, most existing knowledge-grounded models condition knowledge simply on conversation history, which we regard as a prior distribution over knowledge. Compared with the posterior distribution over knowledge, which further considers the actual knowledge used in the true responses, the prior distribution has a larger variance and thus, existing models can hardly select appropriate knowledge simply based on the prior distribution during the training process. In comparison, we carefully analyze the discrepancy between prior and posterior distributions and our model has been effectively taught to select appropriate knowledge and to ensure that knowledge is better utilized in generating responses.

Our work is related to conditional variation autoencoders (CVAE) [Zhao et al., 2017] where a recognition network is used to approximate a posterior distribution, but we have the following differences. Firstly, we focus on different problems. In this paper, we focus on knowledge-grounded conversations where our model employs a novel knowledge selection mechanism while CVAE aims at capturing the discourselevel diversity. Secondly, CVAE learns a distribution in a latent space where the meanings of latent variables are difficult to interpret, while we explicitly define the distributions over knowledge based on the semantic similarity on utterances and responses, which has better understandability.

5 Conclusion

In this paper, we present a model with a novel knowledge selection mechanism, which is the first neural model that makes use of both prior and posterior distributions over knowledge to facilitate knowledge selection. We clearly state and analyze the discrepancy between prior and posterior distributions, which has not been studied before. Extensive experiments on both automatic and human metrics demonstrate the effectiveness and usefulness of our model. As for future work, we plan to extend our knowledge selection mechanism for selecting knowledge in multi-turn conversations.

Acknowledgments

We would like to thank Siqi Bao, Chaotao Chen and Huang He for their help and valuable suggestions on this paper.