Generating Multiple Diverse Responses with Multi-Mapping and Posterior Mapping Selection

1 Introduction

Recently, generative models built upon sequence-to-sequence (Seq2Seq) framework [Sutskever et al., 2014; Shang et al., 2015] have achieved encouraging performance in opendomain conversation, with their simplicity in learning the mapping from input post to its response directly. However, an input post in human conversation is open to multiple potential responses, which is typically regarded as a one-to-many problem. Modeling diverse responding regularities as a oneto-one mapping, the Seq2Seq models inevitably favor general and trivial responses [Li et al., 2016a]. Thus the rich and diverse content in human conversation can not be captured.

To address this problem, work from [Zhao et al., 2017; Serban et al., 2017] combines Seq2Seq with Conditional Variational Auto-Encoder (CVAE) and introduces a Gaussian latent distribution to build the one-to-many relationship. By drawing samples from the Gaussian latent distribution, multiple responses can be generated. However, the Gaussian latent distribution is not compatible to the multi-modal 1 nature of diverse responses and lack of interpretability.

For these issues, recent approaches [Zhou et al., 2017; Zhou et al., 2018a; Tao et al., 2018; Gu et al., 2019] resort to incorporation of multiple latent mechanisms, each of which could model various responding regularities for an input post, as shown in Figure 1. For example, Zhou et al. [2017; 2018a] introduce multiple latent embeddings as language responding mechanisms into Seq2Seq framework, and output responses of different language styles by choosing different embeddings; Tao et al. [2018] augment Seq2Seq model with multi-head attention mechanism, and generate responses that focus on specific semantic parts of the input post with different heads of attention. Although these methods have shown potential to capture the multi-modal nature of diverse responses, they still fail to fulfill the one-to-many relationship, due to their inaccurate optimization of latent mechanisms. As shown in Figure 1, given a target response, the optimization is distributed to each latent mechanism. However, for more accurate modeling, we assume only the latent mechanism corresponding to the target response should be selected for optimization. For example, given a questioning response, we should only optimize the latent mechanism that models interrogative responding regularities rather than the other irrelevant ones. Although in some methods the optimization to each latent mechanism is guided by a weight from the input post, the weight is inaccurate to represent the selection of the corresponding latent mechanism, considering the semantic gap between the input post and the target response. With such a rough optimization, the latent mechanisms are not guaranteed to capture the diverse responding regularities.

In this paper, in order to capture the one-to-many relationship, we propose to augment the Seq2Seq framework with a multi-mapping mechanism, employing multiple mapping modules as latent mechanisms to model the distinct semantic mappings between an input post and its diverse responses. More importantly, to avoid the rough optimization of latent mechanisms in previous methods, in training time we incorporate a posterior mapping selection module to select the corresponding mapping module according to the target response. By explicitly leveraging the information in the target response (i.e. posterior information), it is easier to select the accurate mapping module. Then only the selected mapping module is updated given the target response. Moreover, to facilitate the optimization of posterior mapping selection, we further propose an auxiliary matching loss that evaluates the relevance of post-response pair. Compared with the simple embedding mechanism and the multi-head attention mechanism whose diversity is limited to the semantic attentions in the input post, the proposed multi-mapping mechanism is more flexible to model different responding regularities. And it also introduces multi-modal capacity and interpretability over the Gaussian latent distribution. With the posterior mapping selection to ensure the accurate optimization of mapping modules, our model is more effective to capture diverse and reasonable responding regularities.

1 Multi-modal means the property with multiple modes.

Our contributions can be summarized as follow:

• We propose a multi-mapping mechanism to capture the one-to-many relationship with multiple mapping modules as latent mechanisms, which is more flexible and interpretable over previous methods.
• We propose a novel posterior mapping selection module to select the corresponding mapping module according to the target response during training, so that more accurate optimization of latent mechanisms is ensured. An auxiliary matching loss is also introduced to facilitate the optimization of posterior mapping selection.
• We empirically demonstrate that the proposed multimapping mechanism indeed captures distinct responding regularities in conversation. We also show that the proposed model can generate multiple diverse, fluent and informative responses, which obviously surpasses the other existing methods.

2 Model

2.1 Model Overview

Following the conventional setting for generative conversation models [Shang et al., 2015; Li et al., 2016b], we focus on the single-round open-domain conversation. Formally, given an input post X = (x1, x2, ..., xT ), the model should generate a natural and meaningful response Y = (y1, y2, ..., yT ′ ).

To address the one-to-many problem in conversation, we propose a novel generative model with multi-mapping mechanism and posterior mapping selection module. The multimapping mechanism employs multiple mapping modules to capture the various underlying responding regularities between an input post and its diverse responses. The posterior mapping selection module leverages the posterior information in target response to identify which mapping module should be updated, so as to avoid the rough optimization in previous methods. The architecture of our model is illustrated in Figure 2 and it consists of following major components:

Post Encoder encodes the input post X into a semantic representation x and feeds it into different mapping modules.

Response Encoder encodes the target response Y into a semantic representation y for posterior mapping selection.

Multi-Mapping consists of K mapping modules and maps post representation x to different candidate response representation {mk}Kk=1 through different mapping module Mk, respectively.

Posterior Mapping Selection selects the z-th mapping module that is corresponding to the target response in the training time.

Response Decoder generates the response based on the candidate response representation mk.

2.2 Encoder

The post encoder employs a one-layer bidirectional Gated Recurrent Unit (GRU) [Cho et al., 2014] to transform the input post X into a sequence of hidden state ht as follows:

where [·; ·] denotes the concatenation of states, -→ht and ←- h t are the forward and backward hidden states at time t, e(xt) is the embedding of word xt. The semantic representation of input post is summarized as x = [-→hT ; ←- h 1].

The response encoder, which encodes the target response Y to the semantic representation y, follows the same structure as post encoder but with different learnable parameters.

2.3 Multi-Mapping

For one-to-many relationship, we introduce a multi-mapping mechanism to capture the different responding regularities, with multiple mapping modules bridging the post encoder and response decoder. Specifically, we employ a linear mapping function as the mapping module for simplicity and leave more advanced mapping structures as future work. Formally, the model maps the post representation x to the K different candidate response representations {mk}Kk=1 through different mapping modules as follows:

where Wk and bk are the learnable parameters of the k-th mapping module Mk .

2.4 Posterior Mapping Selection

To ensure accurate optimization of mapping modules in training time, it is necessary to identify which mapping module is responsible for the generation of target response and only update the corresponding mapping module given the target response. Thus, we incorporate a posterior mapping selection module to explicitly select the corresponding mapping module by leveraging the information in target response. With the guidance of target response, we assume it is easier to find the corresponding mapping module for accurate optimization. Specifically, we introduce a categorical distribution π to denote the selection of mapping module conditioned on the target response. And the selection probability of the k-th mapping module is based on its relevance to the target response, which is measured by the dot product between the representations of candidate response and target response as follows:

Then for a target response, the corresponding mapping module can be sampled according to their relevance. Given that the z-th mapping module Mz is selected, only the corresponding candidate representation mz is fed into the response decoder for further decoding optimization. Therefore, optimization of irrelevant mapping module is not conducted and more accurate optimization of latent mechanisms is ensured. In order to back-propagate through the discrete sampling in posterior mapping selection, we leverage the Gumbel-Softmax reparametrization [Jang et al., 2017].

2.5 Decoder

The response decoder employs an uni-directional GRU with the selected candidate representation mz as its initial state and update its hidden state as follows:

where st is the hidden state of decoder at time t, e(yt-1) is the embedding of the last generated word yt-1, and ct is the attended context vector at time t and defined as the weighted sum of hidden states of the post encoder: ct = ∑Ti=1 αt,ihi, where αt,i is the attention weight over hi at time t:

where v, Wh andWs are learnable parameters. Then at time t, the generation probability conditioned on the input post X and the selected mapping module Mz is calculated as:

where y<t denotes the previous generated words.

The objective of the response generation is to minimize the negative log-likelihood of the target response Y conditioned on the input post X and the selected mapping module Mz as follows:

where p(Y |X,Mz) = ∏T ′ t=1 p(yt|y<t,X,Mz) is the conditional probability of target response.

2.6 Auxiliary Objective

Although the posterior mapping selection module is designed to select the corresponding mapping module by referring to the target response, we find that its raw implementation quickly converges to selecting the same mapping module and thus the proposed model falls back to the vanilla Seq2Seq. We conjecture that in the early training, the response encoder is inefficient to capture the semantic information in target response. So the posterior mapping selection fails to provide an accurate selection of mapping module and the model falls into a local optima that focuses on single mapping module. To address this issue, we introduce an auxiliary objective from the response retrieval task to improve the semantic extraction of response encoder. The auxiliary objective namely matching loss is to evaluate the relevance of post-response pair. Specifically, given an input post X and a target response Y , their relevance probability is estimated by the dot product of their semantic representations:

where r is the label denoting if the response Y is relevant to the post X, and σ is a sigmoid function. Following the previous work [Shang et al., 2018], we adopt negative sampling to train this auxiliary task so as to release the burden of human annotation. Particularly, for the input post X and golden response Y , we randomly sample another response Y - in training set as a negative sample. Formally, the matching loss LM is defined as the negative log-likelihood of relevance for the golden response and negative response:

With this auxiliary matching loss, the response encoder is more efficient to capture the semantic information from the target response and provide a better relevance measurement for the accurate posterior mapping selection.

2.7 Training and Generation

Overall, the total loss function of our model is a combination of the generation loss LG and the matching loss LM:

All the parameters are simultaneously updated with backpropagation.

After optimization with posterior mapping selection, the model is able to capture distinct responding regularities and generate various candidate responses with different mapping modules. For response generation, we assume each mapping module is reasonable and just randomly pick mapping module for responding to avoid selection bias. More advanced response selection such as reranking is left as future work.

3 Experiment

3.1 Datasets

We evaluate the proposed model on two public conversation dataset: Weibo [Shang et al., 2015] and Reddit [Zhou et al., 2018b] that maintain a large repository of post-response pairs from popular social websites. After basic data cleaning, we have above 2 million pairs in both datasets. The statistics of datasets are summarized in Table 1.

3.2 Implementation Details

The vocabulary size is limited to 40,000 and 30,000 in Weibo and Reddit dataset, respectively. The hidden size in both encoder and decoder is set to 1024. Word embedding has size 300 and is shared for both encoder and decoder. We initialize the word embedding from pre-trained Weibo embedding [Li et al., 2018] and GloVe embedding [Pennington et al., 2014] for Weibo and Reddit dataset, respectively. The temperature of Gumbel-Softmax trick is set to 0.67. All model are trained end-to-end by the Adam optimizer [Kingma and Ba, 2015] on mini-batches of size 128, with learning rate 0.0002. We train our model in 10 epochs and keep the best model on the validation set for evaluation. The code is available at: https://github.com/LittleChencht/MMPMS.

3.3 ComparedMethods

We compare our model with several state-of-the-art generative conversation models in both single and multiple (i.e. 5) response generation:

Seq2Seq [Bahdanau et al., 2015]: The standard Seq2Seq architecture with attention mechanism.

MMI-bidi [Li et al., 2016a]: The Seq2Seq model using MaximumMutual Information (MMI) between inputs and outputs as the objective function to reorder generated responses. We adopt the default setting: λ = 0.5 and γ = 1.

CVAE [Zhao et al., 2017]: The Conditional Variational AutoEncoder model with auxiliary bag-of-words loss.

MARM [Zhou et al., 2017]: The Seq2Seq model augmented with mechanism embeddings to capture latent responding mechanisms. We use 5 latent mechanisms and generate one response from each mechanism.

MHAM [Tao et al., 2018]: The Seq2Seq model with multihead attention mechanism. The number of heads is set to 5. Following the original setting, we combine all heads of attention to generate a response for the single response generation, and generate one response from each head of attention for the multiple response generation. Although the constrained MHAM is reported better performance in the original paper, on our both datasets we see negligible improvement in single response generation and much worse performance in multiple response generation due to the lack of fluency. So we only adopt the unconstrained MHAM as baseline.

MMPMS: Our proposed model with Multi-Mapping and Posterior Mapping Selection (MMPMS). We set the number of mapping modules to 20.

3.4 EvaluationMetrics

For automatic evaluation, we report: BLEU [Chen and Cherry, 2014]: A widely used metric for generative dialogue systems by measuring word overlap between the generated response and the ground truth. Dist-1/2 [Li et al., 2016a]: Ratio of distinct unigrams/bigrams in the generated responses, which can measure the diversity of generated responses.

Since empirical experiments [Liu et al., 2016] have shown weak correlation between automatic metrics and human annotation, we consider the careful human judgment as major measurement in the experiments. In detail, three annotators are invited to evaluate the quality of the responses for 300 randomly sampled posts. Similar to [Zhou et al., 2017], for each response the annotators are asked to score its quality with the following criteria: (1) Bad: The response is ungrammatical and irrelevant. (2) Normal: The response is basically grammatical and relevant to the input post but trivial and dull. (3) Good: The response is not only grammatical and semantically relevant to the input post, but also meaningful and informative. Responses on normal and good levels are treated as "Acceptable". Additionally, to evaluate the diversity of multiple responses generation, for the 5 responses generated for a single post, the annotators also annotate the number of distinct meanings among the acceptable responses, namely Diversity. The average Fleiss' kappa [Fleiss and Cohen, 1973] value is 0.55 and 0.63 on Weibo and Reddit, respectively, indicating that the annotators reach moderate agreement.

3.5 Evaluation Results

The evaluation of single response generation are summarized in Table 2. As shown, our model achieves the best performance in human evaluation and Dist on both datasets, especially the visible enhancement in Good ratio (0.24 vs. 0.15 on Weibo and 0.36 vs. 0.25 on Reddit) compared with the best baseline, indicating our model can generate more informative and diverse responses. Notably, Seq2Seq performs the best in BLEU but poor in human evaluation on Weibo, which further verifies the weak correlation of BLEU to human judgment.

Table 3 shows the evaluation results of multiple response generation. As can be seen, our model outperforms baseline methods by a large margin in human evaluation on both datasets. More importantly, the Diversity measure of our model reaches 2.61 on Weibo and 3.16 on Reddit, much higher than other baseline methods, which demonstrates the superiority of our model to generate multiple diverse and high-quality responses. This can also be supported by the examples in Table 4 2 , where the multiple candidate responses returned by our model are much more relevant and diverse.

However, CVAE fails to generate appropriate responses (i.e. low Acceptable ratio) even though it achieves relatively high Dist scores. It seems that its generation diversity comes more from the sampling randomness of the prior distribution rather than from the understanding of responding regularities. And we conjecture that the lack of multi-modal property in Gaussian distribution makes it hard to capture the one-tomany relationship among human conversation.

Instead, MARM performs the worst in response diversity with the lowest score of Diversity and Dist while it obtains a high Acceptable ratio. As shown in Table 4, the responses from MARM are similar and trivial. And the word overlap among the 5 candidate responses is up to 94% and 96% on Weibo and Reddit, respectively, showing that each mechanism embedding converges to similar and general responding regularities. We attribute this to the lack of accurate selection of latent mechanism for the target response during training. Since each mechanism embedding is roughly optimized with the same target response, they are prone to learn similar and general responding relationships. This result further validates the importance of our proposed posterior mapping selection.

2 Due to space limitation, responses from Seq2Seq and MMI-bidi are omitted, considering their lack of one-to-many mechanism and low diversity in multiple response generation.

It is also interesting to find the degradation of MHAM in multiple response generation over single response generation. According to Table 4, the responses from different heads of attention are diverse but ungrammatical and irrelevant. The reason may also lies in the absence of accurate selection of latent mechanism during training. Since the response generation is optimized with a combination of all heads rather than the head corresponding to the target response, each head of attention is coupled together and fails to capture independent responding regularity. So the model ends up with inappropriate responses when only single head of attention is utilized in multiple response generation, but ends up with appropriate responses when combining all heads of attention in single response generation. Another potential reason is that there is no enough distinct semantic information in the input post for the model to attend separately.

3.6 Analysis onMappingModules

We also conduct analysis to explore the responding regularities that the mapping modules have captured. For the 200 posts sampled from the Weibo test set, we obtain the candidate representation mk from different mapping modules and apply t-SNE [van der Maaten and Hinton, 2008] for visualization. As shown in Figure 3, the candidate representations are highly clustered by their corresponding mapping modules, indicating the ability of various mapping modules to model various responding regularities by mapping the post representation to significantly different response representations.

For more intuitive understanding, we identify the keywords of different mapping modules from their responses for the Weibo test set. We assume that for a mapping module its keyword should appear frequently in its output responses but rarely occur in other mapping modules. Then, the importance of a word w in the mapping module Mk is measured by p(w|Mk) = N w k / ∑Ki=1 N w i , where N w k is the number of times that word w occurs in the responses fromMk. In addition, only the keywords occurring frequently are considered, namely N w k > 40. Table 5 illustrates the keywords of several representative mapping modules. As we can see, Map1, whose keywords are mainly question words (e.g. where, what and the question mark), probably represents interrogative responding regularity. Map-2 tends to respond with mood words (e.g. wow, lol, haha). The keywords of Map-3 are composed of intensity words (e.g. too, so) and modifier words (e.g. horrible, cute, pretty), showing that it tends to respond with surprise and emphasis. And Map-4 is more likely to return responses in English. The keywords of Map-5 are mainly subjective words (e.g. I, think, believe, want), indicating that it tends to generate responses with respect to personal opinion or intention. These results verify that each mapping module can capture diverse and meaningful responding regularities.

e E2vrQ`/b

Table 5: Keywords of different mapping modules.

4 RelatedWork

The safe response problem [Li et al., 2016a] in Seq2Seq models [Sutskever et al., 2014; Shang et al., 2015] remains an open challenge. In order to generate multiple diverse responses, many approaches resort to enhanced beam search [Li et al., 2016a; Li et al., 2016b]. But these methods are only applied to the decoding process and limited by the semantic similarity in the decoded responses. Another line of research turns to the different factors that determine the generation of diverse responses, such as sentence function [Ke et al., 2018], specificity [Zhang et al., 2018], dialogue act [Xu et al., 2018] and keywords [Gao et al., 2019]. However, such methods require annotations and can capture only one aspect of one-to-many relationship. Work from [Zhao et al., 2017; Serban et al., 2017] combines Seq2Seq with Conditional Variational Auto-Encoder and employs a Gaussian distribution to capture discourse-level variations. But it is observed that these methods suffer from the posterior collapse problem [Bowman et al., 2016]. Moreover, the Gaussian distribution is not adaptive to the multi-modal nature of diverse responses.

The most relevant work to ours is those incorporating multiple latent mechanisms for one-to-many relationship. Zhou et al. [2017; 2018a] propose a mechanism-aware machine that introduces multiple latent embeddings as language responding mechanisms. Tao et al. [2018] propose a multihead attention mechanism to generate diverse responses by attending to various semantic parts of an input post. Gu et al. [2019] incorporate a Gaussian mixture prior network and employ Gaussian component as the latent mechanism to capture the multi-modal nature of diverse responses. However, without an accurate selection of the latent mechanism corresponding to the target response, these methods suffer from a rough optimization of latent mechanisms. Given a target response, instead of optimizing the corresponding latent mechanism, Zhou et al. [2017] just roughly optimize each mechanism embedding while Tao et al. [2018] optimize a rough combination of all heads of attention according to the input post. Whereas, our model maintains a more accurate selection of the corresponding latent mechanism by referring to the posterior information in the target response. Although posterior information is also utilized in [Gu et al., 2019], it is for the optimization of the prior Gaussian component which is roughly inferred by the input post, rather for the accurate selection of the Gaussian component.

5 Conclusion

In this paper, we augment the Seq2Seq model with a multimapping mechanism to learn the one-to-many relationship for multiple diverse response generation. Particularly, our model incorporates a posterior mapping selection module to select the corresponding mapping module according to the target response for accurate optimization. An auxiliary matching loss is also proposed to facilitate the optimization of posterior mapping selection. Thus each mapping module is led to capture distinct responding regularities. Experiments and analysis support that our model works as expected and tends to generate responses of diversity and high quality.

R Acknowledgments

We thank Rongzhong Lian, Siqi Bao and Huang He from Baidu Inc. for their constructive advice.