MuiDial: Improving Dialogue Disentanglement with Intent-Based Mutual Learning

1 Introduction

Online communication platforms such as Gitter and Slack have been widely adopted by developers as the main communication channel to discuss events, issues, tasks, and personal experiences. Despite the convenience of sharing and collaborating in real-time, there are often multiple conversations occurring among a large group of participants concurrently, thus making it difficult to detect topics, analyze user behaviors and summarize contents automatically.

Various automatic dialogue disentanglement approaches have been proposed with the goal to separate utterances into dialogues and better facilitate the usage of massive live chat data. These approaches share two common aspects to achieve dialogue disentanglement [Zhu et al., 2021]. One aspect is the utterance-to-utterance prediction (U2U), which predicts the "reply-to" probability between two utterances. U2U can find utterances with similar semantic features, which improves the accuracy of disentanglement. The other aspect is the utterance-to-thread prediction (U2T) that predicts which dialogue-thread a given utterance belongs to. U2T can analyze some weak semantic related utterances (e.g., "Greeting" and "Feedback"), thus improving the completeness of dialogues.

∗Corresponding Author

Although there are some hybrid approaches that aim to combine U2U and U2T predictions together [Tan et al., 2019; Liu et al., 2021; Pappadopulo et al., 2021], the two predictions are still trained independently, and the learning is hardly shared between them comprehensively.

In this paper, we propose MUIDIAL, a novel intent-based dialogue disentanglement with mutual learning. First, we enhance the utterance embedding by integrating the intent embedding and five heuristic features with textual utterance embedding. Figure 1 shows how the user-intent can benefit the disentanglement performance. As shown in the figure, it is difficult to distinguish between u5 and u6 in terms of the traditional semantic features (e.g., timestamp, speaker or text message). However, if we consider intent, "Information Giving" is usually used to reply to "Original Question", not "Information Giving", which means u6 and u3 tend to be disentangled into the same dialogue, rather than u5. Second, we design two new types of utterance prediction components, U2UDIAL and U2TDIAL, where U2UDIAL leverages the pointer mechanism to predict the "reply-to" relationships and U2TDIAL leverages the attention-based state transition mechanism to predict the "belong-to" relationships. Finally, we utilize a new mechanism to share the learning ability between U2UDIAL and U2TDIAL via mutual learning [Zhang et al., 2018], which is found successful in several AI research areas [Liu et al., 2019; Yang et al., 2020] but has not been commonly facilitated in dialogue disentanglement.

To evaluate the effectiveness of our approach, we conduct an exploratory study on four cross-domain benchmark datasets: IM (Instant Messenger), Reddit, IRC (Ubuntu Internet Relay Chat), and Gitter. The results show that MUIDIAL outperforms SOTA baselines on all the datasets.

Our major contributions are summarized as follows:

• We enhance the utterance embedding for dialogue disentanglement with user intents and five heuristic features.
• We adopt mutual learning in the dialogue disentanglement setting, where we propose MUIDIAL, a novel intent-based dialogue disentanglement model, which takes the advantages of both U2U and U2T predictions with mutual distillation.
• We demonstrate the effectiveness and efficiency of MUIDIAL on commonly used benchmark datasets.

2 Methodology

There are four main steps to construct MUIDIAL, as shown in Figure 2, including: 1) the intent-based representative learning to embed the input utterances; 2) multiple thread probability computation using both U2U and U2T predictions; 3) mutual learning to optimize parameters for both predictions; and 4) mutual predicting to output convergent disentanglement results via voting.

2.1 Intent-based Utterance Embedding

We first embed utterances with text, heuristic, and user-intent encoders; then we learn its context into the context-aware utterance embedding to obtain a rich representation.

Individual Utterance Embedding

Given a set of utterances ui, a slice of multi-party chat is defined as [u1, u2, ..., un]. Each utterance is a tuple ui = ⟨ti, si,wi⟩, where wi = [wi1,wi2...,wim] is the word sequence posted by speaker s at the time t. The dialogues in one chat slice are noted as D = [D1,D2, ...,DK], and each utterance is associated with a dialogue-thread Dk. We learn the representative of individual utterance ui with three encoders (ui, ri, and ϵi), and concatenate them as the individual utterance embedding ξi = [ui; ri; ϵi].

Text Encoder. Given the word sequencewi in utterance ui, we embed it into ui by using pre-trained BERT [Devlin et al., 2019] model.

Heuristic Feature Encoder. To enrich the utterance embedding with high-level semantic information, we extract five predefined features from each utterance ui = ⟨ti, si, wi⟩ in a heuristic manner.

1. Speaker: si = [I (k = Index (si))], where I is the indicator function that returns 1 if the element subscript of si is consistent with the identifier of si.
2. Time Difference: ∆ti = ti - t0, where ti is the timestamp of the current utterance, and t0 is the timestamp of the first utterance in the same chat.
3. Topic: τi = GloVe (LDA (wi)), where LDA is the topic extraction model pre-trained on 10K topics, and GloVe is the word embedding model pre-trained on 30K words.
4. Entity: ei = GloVe (LSTM-CRF (wi)), where LSTMCRF is the bi-directional entity extraction model, and GloVe is the pre-trained word embedding model.
5. Mention: mi = M(si), where mention history M is a diagonal matrix for all speakers [Yu and Joty, 2020]. We set the M(si, sj) as the frequency that speakers si and sj mention each other.

The output of the heuristic feature encoder ri is the concatenation of the five features:

User-intent Encoder. We use the concatenation of text embedding ui and heuristic features ri as the input of user-intent encoder. First, we use a BiLSTM [Hochreiter and Schmidhuber, 1997] model to encode the utterance information into a deep contextual representation hϵi . Then, we use a multi-layer perceptron (MLP), which includes three fully-connected layers (FCNN) and uses ReLU as the activation function to encode the user-intent embedding ϵi.

Following the previous user-intent study [Qu et al., 2018], we define six intents that commonly appear in live chats, as described in Table 1. The output of user-intent encoder ϵi is a 6-dimensional vector that records the weights of the six user intents according to each dimension. Finally, we use the Softmax function to predict the intent probability:

pint(ui) = p([OQ, FQ, IG, IS, FB,OT] |ui) = Softmax (ϵi) (4) where pint(ui) indicates the probability of intent prediction.

Context-aware Encoder

This component aims to account for representing the full utterance context in one chat slice. Given a sequence of individual utterance embeddings [ξ1, ξ2, ..., ξn] in one chat slice, we use a BiLSTMmodel to further embed the contextual information for each utterance ui.

where hξi is the output of the intent-based utterance embedding for utterance ui.

2.2 Multiple Thread Probabilities Computation

Given the intent-based embedding of all the utterances in the chat slice [hξ1 , ..., hξn ] as input, we first construct U2UDIAL and U2TDIAL to respectively output the "reply-to" and "belong-to" probability matrices. Since the two probability matrices have different dimensions, we then design the TBT algorithm to transform them into the same dimension.

Utterance-to-Utterance Prediction (U2UDIAL)

U2UDIAL uses the pointer module [Nguyen et al., 2020] to predict the "reply-to" probability between the current ui and its former utterances uj≤i. Given a sequence of the intentbased utterance embedding [hξ1 , hξ2 , ..., hξn ], where hξi ∈ R1×δ (δ indicates the size of the dialogue embedding vector), we use a fully-connected neural network (FCNN) to map the hidden representation to new hidden vectors. We then use the dot-product to calculate the similarity and normalize the result with Sigmod activation:

where WU ∈ Rδ×δ represents the trainable parameters of FCNN and the "reply-to" probability matrix U ∈ Rn×n indicates the probability that ui → uj is a "reply-to" relationship.

Utterance-to-Thread Prediction (U2TDIAL)

U2TDIAL uses the attention-based state-transition mechanism [Chen and Manning, 2014] to predict the "belongto" probability. Given the intent-based utterance embedding hξi , we randomly initialize S1 ∈ RK×δ as S1 =

[S11,S12, ..., S1K], where K is the number of dialoguethreads in a chat slice and S1k is the sub-vector to represent the feature of one dialogue cluster. At timestamp i, we choose the dot-product and Softmax to measure the similarity between hξi and its state Si, and input the result to another FCNN to predict the "belong-to" probability.

where matrix WT ∈ Rδ×δ represents a set of trainable parameters of coupling measurement, andW′T ∈ RK×K is the trainable parameter of FCNN. The kth element of the weight vector αi ∈ RK×1 denotes the attention weight between ui and dialogue-thread Dk. T ∈ Rn×K indicates the probability that ui belongs to the thread Dk. Finally, we use αi to weight hξi and update Si with self-attention mechanism [Yang et al., 2016] to aggregate the previous embedding:

where Wattn ∈ Rk×δ and battn ∈ RK×1 are trainable parameters for calculating the attention weights.

Thread-based Transformation (TBT)

Given the two probability matrices U and T outputted by U2UDIAL and U2TDIAL, the TBT function can be defined as TBT ({U, T}) → p(ui, uj), where p(ui, uj) indicates the probability of whether the current utterance ui and the former utterances uj should be classified into the same dialoguethread. Figure 3 visualizes the details of TBT function, which consists TBTU and TBTT . For "reply-to" probability U, TBTU transforms the U2UDIAL thread probability with an iterative dynamic programming algorithm:

For "belong-to" probability T, TBTT transforms the U2TDIAL thread probability with dot-product:

The output probabilities of TBT include both Rn×n lower triangular matrices, with 1 on the diagonal. The other element values are distributed in (0, 1).

2.3 Mutual Learning

We calculate four losses with two probabilities and optimize the MUIDIAL mutually. The goal is to optimize the intent embedding together with the thread probability prediction.

Loss Functions

The four losses include intent loss, U2UDIAL loss, U2TDIAL loss, and KL loss.

Intent Loss. Given the intent probabilities produced by the user-intent encoder pint (ui), we use the cross-entropy loss to calculate the intent loss Lint:

where H(·, ·) is the cross-entropy function, Y int i is the truth intent label of ui, and T is the batchsize of training datasets.

U2UDIAL and U2TDIAL Losses. Similarly, we use the Cross-Entropy loss to calculate the U2UDIAL loss Lthd U and U2TDIAL loss Lthd T with thread probabilities.

where Y thd ij indicates the truth label of whether ui and its former utterance uj≤i are in the same dialogue-thread.

KL Loss. In order to measure the mutual difference between two output distributions, we calculate KullbackLeibler (KL) divergence between two thread-based probabilities as the KL loss:

where KL (·||·) is the function of KL divergence.

Mutual Optimization

With the above losses, we iteratively optimize the parameters of MUIDIAL by performing the two optimizations mutually:

• Optimize LU = αLint+βLKL+(1-α-β)Lthd U where α and β are the loss-balancing parameters.
• Optimize LT = αLint+βLKL+(1-α-β)Lthd T , where α and β are the loss-balancing parameters.

We employ gradient descent to minimize LU and LT . The two optimization processes are repeated simultaneously until the model is convergent.

2.4 Mutual Prediction

Since pU (ui, uj≤i) and pT (ui, uj≤i) may give different disentanglement predictions, we adopt a vote&construct strategy to determine the final disentangled dialogues. Given the multi-party chat slice [u′1 , u′2 , ..., u′n ] in the test dataset, we separately predict the probabilities pU(u′i , u′j≤i ) and pT (u′i , u′j≤i ). Then, we predict that u′i and u′j≤i are in the same thread if and only if both probabilities satisfy p(u′i , u′j≤i ) > η, where η is the cutoff-value to separate the positive and negative classes. We follow the existing works to tune η until MUIDIAL achieves the optimal performance, and choose the cutoff-value as η = 0.6. Finally, we use the predicted relationships to construct independent dialogues D and output the disentangled results.

3 Experiments

3.1 Experiment Settings

Datasets. We adopt four benchmark datasets for the dialogue disentanglement task. As shown in Table 3, IM and Reddit contain social chats collected fromMicrosoft Messenger and Reddit forum respectively, whereas IRC [Kummerfeld et al., 2019] and Gitter focus on technical chats collected from Ubuntu IRC chat log and Gitter chat rooms from eight open-source communities respectively.

Ground-truth Labeling for Intent. We manually label the intent category for each utterance on the sampled training and validate dataset from the four benchmark datasets, as shown in the last row in Table 3. To guarantee the correctness of the labeling results, we carefully assemble the labeling team with four members (2PhD/1MS students and 1 research fellow) and develop a widely accepted process in our annotation. Cohen's Kappa is measured to indicate the level of agreement among team members. The average Cohen's Kappa score is 0.84, which proves the annotation agreement 1 .

Evaluation Metrics. We use four commonly used metrics to evaluate the dialogue disentanglement results: Normalized Mutual Information (NMI) [Strehl and Ghosh, 2002], Adjusted Rand Index (ARI) [Santos and Embrechts, 2009], F1score (F1), and Dialogue Levenshtein-Distance (DLD) [Jiang et al., 2021]. ARI is the most strict metric that bases the evaluation on pairwise biases, while NMI penalizes more on the cluster-level. F1 is the reconciliation of accuracy and recall and DLD calculates Levenshtein distance of correcting negative disentanglement, which reflects the user satisfaction.

Implementation Details. The entire chat log is first processed. We split a chat log into a set of chat slices if 1) the number of dialogues in the chat exceeds 4, or 2) the time interval between two adjacent utterances exceeds 48 hours. If the number of utterances in a set of chat slices exceeds 100, a new set of chat slices is created. DLD is reported to be a better metrics on dialogue disentanglement against other metrics [Jiang et al., 2021], thus we use DLD for tuning hyperparameters in the experiments. The final set of the hyperparameters is shown as follows. The utterance embedding vector and the heuristic feature vector are 768-dimensional. The LSTM hidden size is 256. When training MUIDIAL, the mini-batch size is set to 16. Adam optimizer [Kingma and Ba, 2015] is used to optimize the parameters with the initial learning rate of 5e-4. The experiment environment is a Windows 10 desktop computer with NVIDIA GeForce RTX 2060 GPU, Intel Core i7 CPU, and 32GB RAM.

1 According to magnitude guidelines, kappa value above 0.81 is considered as almost perfect agreement.

3.2 Experiment Results

Comparison with SOTA

Baselines. 1) U2U baselines: CISIR is a Siamese hierarchical CNN that estimates the similarity between utterances. PtrNet applies the pointer module on calculating heuristic similarity scores between the current utterance and its former utterances. 2) U2T baselines: DialBERT applies BERT to encode the utterance and predicts the belonging relationship with a BiLSTM classifier. SSE2E utilizes the session-state encoder and "build+update" state-transition model to predict the session-based probability. 3) Hybrid baselines: CATD combines a link-based and a cluster-based model together by weighting the two prediction probabilities. DAG-LSTM enriches the utterance feature embedding with thread encoding and predicts the dialogues with N-ary Tree-LSTM.

Results. Table 2 demonstrates the baseline comparison results. When comparing with the best performing baselines, MUIDIAL achieves the best performance on all four metrics on average, improving by 4.07% (NMI), 4.17% (ARI), 4.86% (F1), and 8.34% (DLD). MUIDIAL also outperforms the baselines on the majority of the individual benchmark datasets. We believe that the performance advantage of MUIDIAL mainly comes from two perspectives: 1) MUIDIAL embeds utterances with multiple aspects, such as intent embedding and heuristic features. Thus it is able to accurately capture the semantics of utterances; 2) MUIDIAL incorporates mutual learning to better take the advantages from both U2U and U2T, thus enhances the prediction accuracy.

Case Study. To qualitatively compare MUIDIAL and baselines, we illustrate how they disentangle the example chat slice presented in Figure 1. The disentanglement results are shown in in Table 4. We can see that the U2U baselines (CISIR and PtrNet) are capable of finding utterances with similar semantic features, but are likely to miss some coupling utterances in the scope of the dialogue (e.g., missing u1 and u4 in Dial. 1, and missing u7 and u8 in Dial. 2). U2T baselines (DialBERT and SSE2E) are more fitted to detect comprehensive utterances in the dialogue-thread scope, but are likely to miss some semantically similar utterances posted far from the majority (e.g., missing u9 in Dial. 1). Hybrid baselines (CATD and DAG-LSTM) are likely to make less mistakes than other baselines. Overall, MUIDIAL provides the most accurate disentanglement result.

Component Analysis

Variants. 1) U2UDIAL+U2TDIAL is the MUIDIAL without the intent embedding. 2) Int.+U2UDIAL refers to the models without mutual learning, which use only U2U prediction for disentanglement. 3) Int.+U2TDIAL refers to the models without mutual learning, which use only U2T prediction for disentanglement.

Results. Table 2 presents the performances of MUIDIAL and its variants. We can see that MUIDIAL outperforms all its variants on average. When comparing with MUIDIAL and U2UDIAL+U2TDIAL, removing the intent embedding leads to a moderate decrease of 4.48% (NMI), 2.27% (ARI), 3.47% (F1), and 4.96% (DLD). When comparing MUIDIAL with Int.+U2UDIAL and Int.+U2TDIAL respectively, removing the mutual learning framework leads to a dramatic decrease, where using Int.+U2UDIAL decreases 9.54% (NMI), 7.14% (F1), 6.87% (F1), 12.40% (DLD), and using Int.+U2TDIAL decreases 12.17% (NMI), 11.93% (ARI), 10.62% (F1), 13.83% (DLD). In summary, both the intentbased utterance embedding and mutual-based joint training adopted by MUIDIAL are helpful for dialogue disentanglement, where mutual-based joint training provides the most significant contribution to the effectiveness of MUIDIAL.

Performance of Intent Classification. Figure 4 shows the performance of intent classification during the mutual learning (Epoch 0-30), along with the performance of MUIDIAL. We can see that the intent-F1 scores can reach 75.62%- 77.03% at the end of training on all the four datasets, indicating the intent classification performance is relatively satisfactory. We can also see that MUIDIAL converges faster than intent classification. Since MUIDIAL achieves the highest performance when the intent-F1 scores are around 55%, we believe that the current intent classification is "good enough", and using a better intent classification may not upgrade the MUIDIAL performance any more.

Hyper-parameter Analysis

We conduct experiments to investigate the sensitivity of our method to the three hyper-parameters: intent loss weight α, mutual loss weight β, and number of state sub-vectors K. The results are illustrated in Figure 5. For α and β, we first independently train MUIDIAL by varying α from 0 to 1.0. Our model achieves the highest performance when α = 0.4 on average. Then, we fix α to 0.4 and vary β from 0 to 0.6. The best performance is achieved at β = 0.3 for IM and Gitter, and β = 0.4 for Reddit and IRC. Finally, we vary K from 2 to 7. During the process, we observe that with the increase of K, the value of DLD shows a positive correlation, but the growth is slight.

4 RelatedWork

Dialogue Disentanglement Models. Apart from the SOTA approaches introduced in Section 4.1, there are some other studies on dialogue disentanglement. Shen et al. [2006] proposed Weighted-SP with three variations of a singlepass clustering algorithm to predict whether the utterancepair had any potential semantic relationships. Elsner et al. [2010] chose three heuristic features to represent the utterances and trained the utterance-pair prediction by using maxentropy classifiers. DeepQA[Severyn and Moschitti, 2015] and ABCNN[Yin et al., 2016] adopted neural models to predict the QA structure of utterances and obtained independent dialogues. Mehri et al. [2017] utilized RNN as the classifier to predict the "reply-to" links.

Mutual Learning Application. Although mutual learning has not been commonly facilitated in dialogue disentanglement, it is found successful in several other research areas.

Zhang et al. [2018] proposed a deep mutual learning (DML) approach to learn collaboratively in a group of student models throughout the training process, which improved the image detection task on the CIFAR-100 dataset. Wu et al. [2019] utilized a mutual learning module (MLM) to better leverage the correlation of multiple tasks and significantly improved saliency detection accuracy. Liu et al. [2019] proposed the DAML models, which utilized mutual-based attention to predict user rating for item recommendation. Yang et al. [2020] proposed MutualNet, which integrated a mutual learning scheme with both network width and resolution in order to reduce computation resources of perception tasks.

5 Conclusion and FutureWork

In this paper, we propose a novel intent-based dialogue disentanglement model MUIDIAL using the mutual learning framework, which enriches the utterance embedding with user intents. A new U2U prediction U2UDIAL and a new U2T prediction U2TDIAL are built separately to predict the "reply-to" and "belong-to" probability matrices. The thread-based transformation (TBT) is then introduced to transform the probability matrices to the same dimensions. Mutual learning framework is used to train U2UDIAL and U2TDIAL. The evaluations on four benchmark datasets show that our model outperforms the baselines by 5% on average.

In the future, we plan to conduct ablation studies on MUIDIAL via utilizing other user-intent encoders. Additionally, we look to introduce some keyword analysis on text fragments such as codes and external links into MUIDIAL, which may further improve the performances of our dialogue disentanglement model.

Acknowledgments

This work is supported by the National Key R&D Program of China under Grant No. 2018YFB1403400, the National Science Foundation of China under Grant No. 61802374, 62002348, and 62072442, and Youth Innovation Promotion Association CAS.