PCVAE: Generating Prior Context for Dialogue Response Generation

1 Introduction

Researchers from both academic and industrial communities have paid increasing attention to open domain dialogue responses generation since it is promising in real-world applications. In this task, for a given context, usually, there are more than one valid responses, which is the so-called one-tomany problem [Csaky et al., 2019]. For CVAEs [Sohn et al., 2015], they have been widely used in dialogue responses generation. The conventionally used continual latent variables in CVAE make it convenient to sample in the latent space for generating different responses, however, they are not suitable for generating distinct and specific responses but generic results. In CVAE, a context is mapped to distribution in latent space, and we sample a latent variable in latent space according to the distribution. This scheme tends to capture latent variables around the center of the distribution. However, the distinct responses are far away from each other [Sun R1: You Know the deal.

∗Equal Contribution.

et al., 2021], which causes them hardly be sampled. To deal with this problem, we propose to introduce a novel discrete variable called prior context as shown in Figure 1. Specifically, we employ vector quantization to quantize the encoded responses into discrete latent variables (codewords) as prior context. Thus, during the test time, we can sample from possible codewords to aid the model to generate distinct and specific responses. Although vector quantization for discrete latent variables has been studied and applied in various areas, two critical issues that remain to be addressed to achieve our goals: i) It is necessary to ensure distinctness and diversity of prior context for generating more specific responses. However, the training of vector quantization is unstable, and the frequently happened codebook collapse problem causes only a small portion of discrete latent variables to be used, which inevitably leads to sub-optimal performance. ii) selecting an appropriate prior context according to the input context is extremely important for superior performance, however, so far it is a rarely explored problem. To resolve it, a straightforward way is to employ an autoregressive model to predict them. However, in our early experiment, simply training a model such as a GRU [Cho et al., 2014] is unable to reach convergence, which causes poor results in testing. To address the above two challenges, we present a novel CVAE model, namely Prior Context Variational AutoEncoder (PCVAE), to model prior context with two key components:

1. We propose active codeword transport (ACT) to actively pull the input embedding towards unused codeword, which not only resolves the codebook collapse problem but also improve the distinctness and diversity of prior context.

(2) For the non-convergence problem, we conjecture the dependency between codewords may be initially unordered, which is different from a natural language where the next words are subject to the previous words, causing difficulty for a autoregressive style model to learn them. To deal with it, we design an autoregressive codeword arrangement (ACA) that regularizes the conditional probability distribution between codewords to fit the autoregressive patterns predicted by a GRU, which is crucial for successful training.

In our experiments, we compare our model with various dialogue CVAE baselines on two authoritative dialogue datasets and conduct further experiments to verify the effectiveness of our proposed model.

To conclude, our contributions can be summarized as follow:

• We introduce the prior context for generating distinct and specific responses in dialogue generation and propose PCVAE that models prior context with discrete latent variables. To the best of our knowledge, PCVAE is the first model to learn and select prior context automatically without manual intervention.
• We propose ACT which resolves the codebook collapse problem and prompts the model to automatically discover potential prior context. Meanwhile, ACA is designed to deal with the non-convergence problem in the training of the autoregressive model, which is crucial for selecting appropriate codeword combinations of prior context for a given context.
• Empirical experiments demonstrate the effectiveness of our model. Further analyses reveal the unique advantages of our methods.

2 RelatedWork

Dialogue Generation. The dialogue generation in the open domain is a challenging task. Early works [Graham, 2015; Sordoni et al., 2015] suffered from the generic response problem. To tackle this problem, there are two major approaches including improving the architecture of the neural dialog model and introducing external knowledge. In this paper, we focus on the former one which includes enhancing the model with attention mechanism [Bahdanau et al., 2015; Luong et al., 2015], applying Reinforce Learning [Liu et al., 2020; Zhang et al., 2018a], GAN [Feng et al., 2020; Zhang et al., 2018b], and variational reasoning [Zhao et al., 2017; Gao et al., 2019; Sun et al., 2021; Zhao et al., 2018].

Vector Quantization. Vector quantization in VAE is first proposed by [van den Oord et al., 2017] for image generation. The process of quantization is selecting a vector (codeword) that is closest to the input vector from a random-initialized vector table (codebook). We refer to the selected vector as the quantized vector of the input.

However, vector quantization suffers from the notorious codebook collapse problem. The codebook collapse is that the input vectors are only mapped to a very small portion of the codewords, which results in the inferior representation of the discrete latent variables. The existing methods that deal with codebook collapse include random restart [Dhariwal et al., 2020; Lancucki et al., 2020] that reinitializes the codeword in the codebook to improve the usage and Population-Based Training (PBT) [Jaderberg et al., 2017; Dieleman et al., 2018] that dynamically adjusts the hyperparameters in the objective function of vector quantization. However, the random restart inevitably changes the indexes of codewords which disturbs the process of selecting prior context and the PBT method can only prevent the decrease of codeword usage but we expect more unused codewords could be utilized. In a word, no existing method is well fitted for our needs. Thus, we propose an ACT to satisfy our demand.

Conditional Variational Autoencoder. The CVAE [Sohn et al., 2015; Yan et al., 2016] is a variational reasoning model that uses a conditional signal (context) to generate more specific data (responses). CVAEs have been widely applied in dialogue response generation. Previous work that introduces manually defined information including dialog act [Zhao et al., 2017], word-level representation [Gao et al., 2019] can be viewed as utilizing a discrete latent variable with external explicit semantic meaning. Except for methods using manually predefined information, there are unsupervised methods including DI-VAE and DI-VST [Zhao et al., 2018] focus on improving the interpretability of learned discrete latent variables, and SpeaCVAE [Sun et al., 2021] uses the clustering method to find group information to resolve the one-to-many and many-to-one problem.

Our work differs from these as follows: (1) We focus on automatically discovering potential prior context, while previous work uses a fixed number of discrete latent variables which is rather limited in the real world. (2) we improve the quality of the codebook in vector quantization and overcome the well-known codebook collapse problem. (3) a comprehensive solution is proposed to properly select prior context.

3 ProposedMethods

We define x ∈ X as a response utterance, c ∈ C as a given context. In a dialogue CVAE, the goal is to model p(x) = ∫ p(x|c)p(c)dc. In our model, we further introduce the y ∈ Y and z ∈ Z that are latent variables of prior context and response, respectively. Thus, this goal can be rewritten as modeling p(x) = ∫ p(x|z, c)p(z, c)dzdc and p(z, c) = ∫ p(z|y, c)p(y|c)p(c)dy. We employ neural networks to model those distributions. We refer to the continual latent variables pϕ(z|y, c) as a prior network and we introduce a recognition network pθ(z|x, c) to approximate the true posterior distribution q(z|y, c). Here, ϕ and θ represent the parameters of the prior network and recognition network, respectively. Both pϕ(z|y, c) and pθ(z|x, c) are assumed to follow isotropic Gaussian distribution. The p(x|z, c) generation network follows a Dirac distribution and the p(y|c) is a discrete latent variable that follows a sequential conditional probabilistic distribution modeled by a prior context planning network. The overview of our model is shown in Figure 2.

In the following, we will first introduce the encoding phrase including prior network, recognition network. Then we introduce decoding phrase including generation network. Then we illustrate how our model learns prior context automatically with vector quantization to generate discrete latent variables y and prior context planning network to learn p(y|c). Finally, we illustrate active codeword transport and autoregressive codeword arrangement in detail.

3.1 Encoding

In this section, we show how to use a prior network to encode context and a recognition network to encode responses for obtaining a distribution of their associated latent variables. A two-layer GRU is used to encode input context as hc and response utterance as hx. Then, to obtain the latent variables that capture deeper semantic meaning, the hc and hx are compressed through ND layers of MLPs, which results in ec and ex, respectively. After that, a prior network and a recognition network are employed to obtain the parameters of their corresponding continual latent variables distribution, which can be described as follows:

where [·, ·] means concatenation of variables, e′c is a conditional signal to guide the generation in decoding, y is learned discrete latent variable for prior context, which will be described later. We define pϕ(z|y, c) ∼ N(µc, σ2 c I) and pθ(z|x, c) ∼ N(µx, σ2x I). The reparameterization trick [Kingma and Welling, 2014] is used to sample a latent variable z from the pθ(z|x, c) in training phrase and pϕ(z|y, c) in testing phrase, respectively. We employ KL divergence LK = DKL(pθ(z|x, c)∥pϕ(z|y, c)) to make pϕ(z|y, c) approximate to pθ(z|x, c).

3.2 Decoding

In this section, we introduce the decoding process of our model that utilizes the output of the prior network and recognition network. In generation network, the conditional signal

Encoder Network Z e′c and z is concatenated as the input ed = [e′c , z]. The information of responses are reconstructed from ed through ND layers of MLP. The final output is used as the initial states of a two-layer GRU to generate the expected responses. We use negative log-likelihood LG as the objective function of generation.

3.3 Prior Context Learning

VectorQuantization. We employ a codebookwith random initialization codewords vi and i ∈ {1, cdots,NE}. The input ex and ec are transformed into an input vector for quantization as ey = MLP([ex, ec]). After that, ey is chunked into NK parts ey,j with the same size of vi. We achieve quantization by quantizing function as follows:

when I(ey,j, vk) = 1, we map ey,j to vk. Then we concatenate all selected vk to obtain the discrete latent variable y. We apply the straight-through estimator [Bengio et al., 2013] to train the codewords as:

Where the stop-gradient operation sg is used since the above selecting approach is intractable. β is a weight coefficient.

Piror Context Planning. The indexes of selected codewords can be viewed as an ordered index sequence. We employ a GRU p(y|c) to predict y, as we do not have the groundtruth response to acquire ey in the testing phase. To learn the p(y|c), we can obtain it through marginalizing the x in p(y|c, x) in training as follows:

where the p(y|c, x) is the probability distribution of quantizing function I . Then we can train the GRU in a autoregressive In manner as follow:

Autoregressive Codeword Arrangement. Intuitively, the ground-truth indexes in training are non-differentiable, which leads to an uncontrolled conditional probability distribution of indexes. It may cause unordered dependency, as we have mentioned. We propose to directly pull the p(y|c) and p(y|c, x) to each other as:

However, the gradient of the p(y|c, x) is intractable, which prevents us from training it to obey autoregressive style distribution. To resolve the problem, we introduce a probability predicting network Ψ(ey) to mimic the p(y|c, x) and we train Ψ by approximating the p(y|c, x) in equation 1. The above steps can be summarized as follows:

Notably, introducing Ψ(ey) makes all parameters involved in generating ey affected. Since ey determines p(y|c, x) in the

forward pass, p(y|c, x) will also tend to obey autoregressive

style distribution, despite it being intractable.

Active Codeword Transport. The general idea of ACT is

to move the ey (target) to the unused codewords (source), and

thus the currently unused codewords would be selected and them as a target set T while ey in the same batch are as a

where j ∈ {0, 1, · · · ,NK}, i ∈ {0, 1, · · · ,NB}, vq means the vector quantization operation as we have defined, (j) in S indicate additional repeated elements to balance the number of elements between T and S. After finding targets, the second problem is how to assign the source to the target appropriately. We must avoid different ey being transported to the same unused codeword while minimizing the total moving distance. We can formalize this problem as an optimal transport problem and employ Wasserstein distance to resolve it:

where the transport plans π that distributes the mass in φ to match that in ν. The ground metric d(s, t) = ∥s - t∥22 provides the cost of moving a unit of mass from s ∼ φ to t ∼ ν. However, the above equation is intractable, therefore we tend to employ a sinkhorn divergence [Cuturi, 2013] to get an approximate optimal transport solution Lact = W∗ (φ, ν) for training.

3.4 Training Objective

In PCVAE, the training objective includes six parts: (1) response generation loss LG, (2) posterior approximating loss LK, (3) vector quantization loss Lvq, (4) prior context planning loss Lpcp, (5) autoregressive codeword arrangement loss Laca, and (6) active codeword transport loss Lact. The total loss is as follows, where λ1,λ2 are weight factors:

How would

You know the deal. you like it? Yes, I just have short hair. Decoder 4 Experiment Datasets. We employ two authoritative datasets for our experiment, including MultiWoz [Zang et al., 2020] for crossdomain task-oriented dialogue and Cornell Movie [DanescuNiculescu-Mizil and Lee, 2011] for open-domain dialogue. Specifically, we use MultiWoz 2.2, which contains 3,406 single-domain dialogues and 7,032 multi-domain dialogues, and all dialogues are task-oriented. The Cornell Movie consists of 220,579 conversational exchanges between 10,292 pairs of movie characters. We further convert them into two turn dialogue datasets that the model has to generate a response given three context utterances. Although on single turn dialogue the one-to-many situations appear more frequently, it may just contain an uninformative utterance such as "ok" where too many acceptable responses exist.

Baselines. We choose the Seq2Seq model, CVAE, and various dialogue CVAEs as baselines. kgCVAE [Zhao et al., 2017] uses manually predefined dialog acts as additional latent variables. SepaCVAE [Sun et al., 2021] uses an unsupervised clustering method to obtain group information to guide the generation. DCVAE [Gao et al., 2019] replaces conventional continual latent variables with discrete latent variables and adopts the predefined word-level knowledge. Note that we do not compare PLM/RL/GAN-based methods since we focus on the improvement from introducing prior context, and we can easily replace our backbone with other architectures for better performances.

Metrics and Evaluation. We employ several widely used metrics, including BLEU-1, BLEU-4 [Papineni et al., 2002], Distinct-1, Distinct-2 [Li et al., 2016], and METEOR [Banerjee and Lavie, 2005]. All results are the mean values of five runs with different random seeds.

Implementation Details. We use word embeddings with 200 dimensions and hidden states with 300 dimensions for encoding and decoding GRU. We initialize the word embedding from Glove embedding [Pennington et al., 2014] and use the NLTK tokenizer [Bird et al., 2009]. The number of layers ND of MLPs for compression and reconstruction is set to 2 with hidden sizes ranging from 200 to 300. We use NK = 4 codebooks and NE = 8192 codewords. The γm used in moving mean predicting is set to 0.95. The β used in vector quantization is set to 0.25. In training, we use batch size NB = 192 and Adam optimizer with an initial learning rate of 1e-3 for both of the datasets. We decrease the learning ratio by 0.8 when the worse valid loss is obtained in the validation phase and stop training as the learning rate is down to 1e-5. For other models, we adopt their official code if available. Otherwise, we adapt their key techniques to our model. For a fair comparison, we replace their encoder and decoder with the same as our model.

4.1 Responses Generation Performance

The experiment results are shown in Table 1 1 . As we can see, PCVAE outperforms strong baselines significantly on both datasets. The higher BLEU and Distinct implies the effective of specific prior context, which is beneficial for improving the diversity and distinctness of the generated responses. Moreover, PCVAE obtains more performance gain on opendomain dataset (Corncell Movie) than multi-domain taskoriented dataset (MultiWoz), which implies that our model can better handle the one-to-many problem. Thus, we conjecture the performance gains of PCVAE mainly come from automatically discovered potential prior context, while other models can only rely on their limited signals. Further, we find the really used codeword in testing on MultiWoz are about 1400 while on corncell movie them are about 7400, which means our model can utilize more prior context when potential response are more diverse and distinct. It also confirms that our model benefits from the prior context.

1 kgCVAE is not tested on Corncell Movie dataset since the dialog cat is unavailable.

4.2 Ablation Study

In this section, we evaluate the effectiveness of our proposed components. Specifically, we introduce several variants of PCVAE by discarding certain components. PCVAE-VQ removes both ACT and ACA with only vector quantization left, and PCVAE-None further removes the vector quantization to evaluate the backbone performances. PCVAE-ACT and PCVAE-ACA remove ACA and ACT, respectively. The ablated results are shown in Table 2. We can observe that: (1) simply applying vector quantization to the model can not bring any improvement. (2) Performances of all models without ACA are close and disappointing. We believe that this phenomenon is caused by the non-convergence problem of the prior context planning network that prevents a model from utilizing appropriate prior context. (3) Comparing the performance gap between PCVAE-ACT and PCVAE-VQ with that between PCVAE and PCVAE-ACA, although the ACT enables model access to better prior context, we can not obtain a satisfactory result without a proper selection. In turn, once we can appropriately select prior context, our model would fully benefited from prior context discovered by ACT, which significantly improves the performance of our model.

4.3 Qualitative Analysis

The randomly sampled responses generated by PCVAE and baseline models are shown in Table 3. The two samples are from Corncell Movie and MultiWoz, respectively. In the first sample, all other models response something related to the air tickets or trip, while only PCVAE properly answer the question whether to see it off. We conjecture this is mainly because our prior context can provide pertinent detail rather than the related general topic information, and it enables our model to generate more specific response. In the second sample, the context is asking a receipt. We can find that SepaCVAE and DCVAE realize the general meaning of context (purchasing) but fail to figure out the distinctness between ordering and requiring a receipt. For kgCVAE, although it provides a acceptable response for the given context, "it will be ready soon" is rather generic. In contrast, our response is specific and informative. We believe this is because prior context provided by our model contain more distinct information, which can aids our model generate response with points and avoid generic result.

4.4 Analysis

Effect of Prior Context. We further evaluate the effect of prior context on performance. To this end, we change NE to vary the representation capacity of a codebook and restrict the influence of prior context to see how it related to the model performance. We conduct the experiment on the MultiMoz dataset and show the results in Table 4. As we can see, increasing NE always leads to better performances, which verifies our intuition that prior context enables PCVAE to generate more diverse and relevant responses. Additionally, We observe that the performance gains gradually saturated as we keep increasing NE. We believe this is because more available codewords make it harder to select prior context properly. It also means that we can always use a relatively large NE to obtain competitive performances.

Active Codeword Transport. We evaluate the effectiveness of ACT for overcoming the codebook collapse problem. To this end, we measure the codebook usage and the mean Lvq of vector quantization in testing. The NE is set to 8192. We also compare four different training setups: (1) Standard: without any heuristics; (2) RS: applying random restarts; (3) PBT: applying population based training. (4) ACT: applying active codeword transport. The experiment result is shown in Table 5. As we can see, the ACT method achieves the highest codeword usage, which demonstrates its superior performance among various previous methods. At the same time, ACT also reduces the mean vector quantization loss, which is beneficial for improving the quality of the codebook.

4.5 Human Evaluation

In this section, we provide a human evaluation of our model. Following [Sun et al., 2021], we randomly sample 200 responses generated by different models on the test set of MultiWoz, respectively. The samples are provided to three annotators with linguistic backgrounds, and we ask them to rank the generated responses considering fluency, diversity, and relevance, respectively. Ties are permitted. Fluency measures the closeness to words from humans, diversity measures the amount of specific information, and relevance measures semantic relevance to the context. The results are shown in Table 6. As we can see, although the fluency score of each model is close, PCVAE outperforms other methods significantly on diversity and relevance. It implies that PCVAE can generate more specific responses about the given context attributed to our superior prior context.

5 Conclusion

This paper proposes a novel hierarchical deep CVAE named PCVAE to automatically learn high-quality prior contexts for generating distinct and specific responses. Specifically, we introduce prior context, a discrete latent variable which aids model resolve the one-to-many problem in dialogue generation effectively. Moreover, we propose active codeword transport and autoregressive codeword arrangement. The former is to discover potential prior context, the later is to effectively train a prior context planning network to select appropriate prior context for a given context. These mechanisms are essential for instantiating our model and achieving superior performance. The experimental results show that PCVAE outperforms strong baselines significantly and further analyses demonstrate the effectiveness of our proposed methods.