A Structure Self-Aware Model for Discourse Parsing on Multi-Party Dialogues

1 Introduction

As a common dialogue scenario, multi-party dialogues have lots of potential applications, attracting increasing research attentions recently. To understand multi-party dialogues, conversational discourse parsing was proposed, which aims at discovering the inter-dependencies between EDUs 1 . In this aspect, most of dominant approaches study dependencybased structures. Figure 1 shows a multi-party dialogue involving five speakers (A, B, C, D, E) and the corresponding discourse relations. We can observe that it effectively includes relations between non-adjacent utterances, such as the "Comment" relation between the first turn and the last turn.

∗Equal contribution

†Corresponding author

1 EDUs are the fundamental discourse units in discourse parsing. Each EDU corresponds to an utterance in a dialogue.

Initial efforts [Muller et al., 2012; Li et al., 2014; Afantenos et al., 2015] for discourse parsing are mainly based on handcrafted features, where the decoding process is modeled in a pipeline manner. In this process, the probability of the discourse relation for each EDU pair is firstly estimated, and then a discourse structure is inferred by a search algorithm such as maximum spanning tree. Inspired by the success of deep learning on other NLP tasks, Shi and Huang [2019] proposed a neural model, i.e. DeepSequential, for discourse parsing on multi-party dialogues. Typically, DeepSequential simultaneously constructs and utilizes the discourse structure for each dialogue: it first extracts features from the already predicted discourse structure, then makes the next prediction before merging it into the partial discourse structure.

Although taking the previously predicted structure can provide richer information, DeepSequential is confronted with severe error propagation. Figure 2 gives the prediction accuracy of DeepSequential and its baseline without historical predictions at different dialogue turns. We can see that utilizing predicted structure has a negative effect on EDUs after the 6th dialogue turn. One likely reason for the severe error propagation is that the current state-of-the-art performance is not accurate enough (less than 60% in accuracy), thus such utilization will introduce more noises than benefits.

In this work, we propose a novel edge-centric Structure Self-Aware Graph Neural Network (SSA-GNN) for discourse parsing of multi-party dialogues. With this model, we explore another direction that learns effective representations without the features from historical actions (thus no error propagation is introduced). Unlike previous work that focuses on learning EDU-specific representations, our model directly uses an edge-specific vector to capture the implicit structural information between each EDU pair. Benefiting from the message passing of Graph Neural Networks [Kipf and Welling, 2017; Marcheggiani and Titov, 2017; Velickovic et al., 2018], edgespecific vectors in SSA-GNN can gradually capture implicit correlation and global information via the semantic interactions with their connected EDU nodes. As a result, our model can learn better representations using implicit structural information instead of explicit historical predictions.

To further enhance representation learning, we introduce two auxiliary loss terms (i.e. relation recognition loss and structure distillation loss) that provide orthogonal training signals into the overall objective function. The first one is calculated by conducting relation recognition at each intermediate layer of SSA-GNN. The second one is an MSE loss function for knowledge distillation [Hinton et al., 2014; Zhang et al., 2019]. It transfers the knowledge of a teacher model that accesses ground-truth discourse relations except for the relation which needs predicting to our model.

To summarize, our contributions in this work mainly include the following three aspects:

• We propose a novel SSA-GNN model for discourse parsing on multi-party dialogues. It directly learns the representation for each EDU pair, yielding stronger performances than previous node-centric GNN models.
• We explore relation recognition and structure distillation to further enhance the robustness of our model for learning better representations.
• Extensive experiments and analysis on two benchmarks demonstrate the effectiveness of our model.

2 RelatedWork

2.1 Discourse Parsing

Most previous studies for discourse parsing are based on Penn Discourse TreeBank (PDTB) [Prasad et al., 2008] or Rhetorical Structure Theory Discourse TreeBank (RST-DT) [Mann and Thompson, 1988]. PDTB mainly focuses on shallow discourse relations while ignoring the overall discourse structure [Yang and Li, 2018]. As for RST, there have been many approaches including transition-based methods [Braud et al., 2017; Wang et al., 2017; Yu et al., 2018], CYK-based approaches [Joty et al., 2015; Li et al., 2016; Liu and Lapata, 2017] and greedy bottom-up approach [Feng and Hirst, 2014]. However, constituency-based RST does not allow structures with crossing dependencies [Afantenos et al., 2015].

To deal with this issue, other approaches [Prasad et al., 2008; Li et al., 2014] take dependency-based structures to represent discourse relations. The dependency-based formalism is especially prevalent on dialogues [Holmer, 2008; Perret et al., 2016], where the non-adjacent can frequently occur. We follow this line of research and propose a novel edge-centric GNN with several auxiliary losses to learn better representations, which alleviates error propagation to make further improvement.

2.2 Edge-centric GNN

As one type of effective approaches for processing structural inputs, GNNs have attracted increasing attentions in recent years. Most previous GNN models [Marcheggiani and Titov, 2017; Beck et al., 2018; Song et al., 2019] mainly resort to learning representations for nodes. Until recently, some studies [Zhu et al., 2019; Yin et al., 2019; Cai and Lam, 2020] introduce edge representations. However, their edge representations are generated only from edge labels and kept constant to serve as additional inputs for enriching node representations.

Compared with previous GNNs, our edge-centric GNN has the following advantages: (i) it directly learns the representation for each edge, thus it can work better for problems that involve a pair of nodes as an input (e.g. discourse parsing); (ii) our GNN iteratively updates the edge hidden states and it allows information exchange both from node states to edge states and vice versa within each iteration. Thus, it can generate more accurate representations for both edges and nodes. To our knowledge, this is the first attempt to apply a GNN model on conversational discourse parsing.

3 Our Model

In this section, we first give a brief description about the task definition before introducing our proposed model in detail.

3.1 ProblemDefinition

Unlike previous work that considers this task as a "resolution" problem, we formulate it as a classification problem for each utterance pair. Given a sequence of EDUs (utterances) x1, x2, ..., xN from a dialogue, we aim to predict all relations {(xj, xi, lji) | i>j} between EDU pairs, where (xj, xi, lji) stands for a discourse link of the relation type lji from xj to xi. 2 Generally, the prediction of each triple (xj, xi, lji) is divided into link prediction P (xj → xi | x0, x1, ..., xi) and relation classification P (lji | xj → xi).

3.2 Structure Self-Aware Graph Neural Network

Figure 3 illustrates the architecture of our model. We first employ a hierarchical GRU consisting of two bidirectional GRU (BiGRU) layers to learn vector representations of EDUs. The bottom layer consumes each EDU xi, where the last hidden states in two directions are concatenated to form the local EDU representation hli , and the top layer acts on hl0 , hl1 , . . . , hln to learn the global context-aware EDU representations Hg = [hg0 , hg1 , . . . , hgn ] in a dialogue.

With the learned EDU representations, we then apply a Structure Self-Aware Graph Neural Network (SSA-GNN) to capture the implicit structural information between EDUs. The input of SSA-GNN is a fully connected graph, where each EDU or each edge connecting two EDUs is represented as a vector. The basic intuition behind our learnable edge representations is to explicitly capture and exploit the implicit structural information within the input dialogue. To initialize hidden states of SSA-GNN, we directly use Hg as the initial node representations u(0) . Besides, we form the initial vector representation r(0) ij for each EDU pair (xj, xi) by concatenating three learnable embeddings: sij indicating whether xi and xj are from the same speaker, tij meaning whether xi and xj are continuous utterances of the same speaker, and dij denoting the relative distance between xi and xj .

Afterwards, inspired by the recent work [Zhu et al., 2019; Cai and Lam, 2020; Wang et al., 2020], we perform Structure-Aware Scaled Dot-Product Attention operation to update node hidden states. With the t-th layer node representations (e.g. u(t) i ) and edge representations (e.g. r(t) ij ), we obtain the node representations u(t+1) at the next layer as follows:

2 Following previous work, we add a dummy root x0 to represent the beginning of a dialogue.

where W ∗ (∗ ∈ {Q,K, V,R, F}) are learnable model parameters 3 , and du is the dimension of the node representations. Meanwhile, we also update edge representations, which enables our model to capture implicit structural information gradually. Specifically, we adopt a GRU-style gating mechanism to update the edge representation r(t) ij :

where ⊙ represents the dot-product operation, and γij and zij are reset gate and update gate, respectively.

We iterate the above hidden state updating process for T times, where the top-layer hidden states are then used for conversational discourse parsing. Concretely, for each EDU xj preceding xi in the dialogue, we adopt ̂Ri,j = [r(T) ij ; r(T) ji ], the concatenated vector of r(T) ij and r(T) ji to conduct link prediction and relation classification.

4 Structure Self-Aware Training

Given the training data D, we train our model according to the following training objective:

3 We use W ∗ to denote learnable model parameters in this work.

where d is a multi-party dialogue from training corpus D, Lce(d; θ) is the standard loss term of conversational discourse parsing based on cross entropy, Lcls(d; θ) and Lskd(d; θ) are two auxiliary loss terms on multiple granularities, both of which are used to further enhance representation learning, and α, β are hyperparameters used to balance the preference among loss terms.

The intuition behind our auxiliary losses is as follows: SSA-GNN adopts a multi-layer architecture and leverages edge hidden states to capture implicit structural information. However, without sufficient supervision information, SSAGNN may heavily rely on its top layer to make the final predictions, resulting in extra challenge for our model to be fully trained. To deal with this issue, we augment the conventional training objective with two loss terms of relation recognition and structure distillation, aiming to guide each SSA-GNN layer effectively learn the implicit structural knowledge. In essence, Lcls(d; θ) provides the coarse-grained structural supervision information at label level, while Lskd(d; θ) exploits the fine-grained structural supervision information at neuron level. Thus, they have the potential to be used together to improve model training.

4.1 Discourse Parsing Loss Lce(d; θ)

Formally, Lce(d; θ) is composed of the loss term Llink(d; θ) for link prediction and the loss term Lrel(d; θ) for relation classification:

where |d| indicates the EDU number of d, x∗i and l∗ ji denote the gold parent and corresponding relation for xi respectively, and ̂Ri,<i = [ ̂Ri,0, ̂Ri,1, ..., ̂Ri,i-1] denotes the relations of xi with its previous EDUs. Note that if EDU xi does not depend on any preceding EDU, x∗i is the added dummy root x0.

4.2 Relation Recognition Loss Lcls(d; θ)

This loss term is calculated by taking the edge hidden states of each intermediate SSA-GNN layer to predict the corresponding discourse relations. By using this loss term, we expect the edge hidden states of every layer can effectively capture all discourse relations. Formally, Lcls(d; θ) is defined as

where T is the layer number of SSA-GNN, lji is the relation type of EDU pair (xj, xi). In particular, if xj → xi does not exist in the discourse structure, we set its target label lji as "None". Comparing with this type of loss, the standard loss Lce(d; θ) has to be propagated from the last layer, and thus the supervision can be weakened throughout this process.

4.3 Structure Distillation Loss Lskd(d; θ)

This loss term is used to exploit the knowledge of a structureaware model (i.e. teacher) for enhancing our model training. Different from our model, the teacher model takes the whole dialogue and all gold relations except for the relation being predicted as additional inputs. Inspired by previous work [Romero et al., 2014], formally, we take the following Mean-Square Error (MSE) loss to reduce the distance of the edge hidden states of each intermediate layer between the teacher and our model:

where r∗,(t) ij denotes the edge hidden state of (xj, xi) from the teacher model.

5 Experiments

5.1 Setup

Datasets. We conduct experiments on two benchmark datasets: (i) Molweni. It is a multi-party dialogue corpus manually annotated based on Ubuntu Chat Corpus [Lowe et al., 2015], which contains 9,000, 500 and 500 instances for training, development and testing, respectively. (ii) STAC. This dataset is collected from an online game. It is much smaller than Molweni and only contains 1,062 and 111 dialogues for training and testing, respectively. We preprocess datasets following Shi and Huang [2019].

Settings. For experiments on STAC, we follow previous work to represent words with 100-dimensional GloVe embeddings [Pennington et al., 2014] that are fine-tuned during training. We adopt a 3-layer SSA-GNN module with 4 heads, where all layers share the same parameters. The dimensions for the edge and the node states in SSA-GNN are 128 and 256, respectively. We set the dropout rate to 0.5 and employ Stochastic Gradient Descent to train all models with batch size and initial learning rate set to 40 and 0.1, respectively. For the Molweni corpus, we use 200-dimensional pretrained GloVe embeddings to initialize word vectors, since it is much larger than STAC. Besides, the batch size is set to 100 for more stable training. For fair comparison, we use the same settings as ours for DeepSequential and its variants on both datasets. Performance of using either Lcls(d; θ) or Lskd(d; θ) with different coefficients are showed in Figure 4. We set our coefficients α and β to 0.2 and 3, respectively. For the experiments using pretrained model, we apply ELECTRA-small [Clark et al., 2020] that has been proved effective on several tasks with a small model size. In this work, micro-averaged F1 score is adopted for evaluation. Our code is available at https://github.com/DeepLearnXMU/Structure-Self-Aware

Baselines. We compare ours with the following baselines:

• DeepSequential [Shi and Huang, 2019]: It adopts an incremental predicting method. Note that it is confronted with error propagation as discussed above.
• DeepSequential(NS): It is a variant of DeepSequential that does not use any features from already predicted discourse structures.

• DeepSequential(Share): It is a variant of DeepSequential that shares the parameters of link prediction and relation classification modules (e.g. embedding layer and hierarchical GRU) except the prediction layer.
• HGRU: It only adopts the hierarchical GRU for representations learning and possesses the same structure as DeepSequential(NS).

Besides, we also conduct experiments with pretrained language model (i.e. ELECTRA-small) to further verify the effectiveness of our model.

5.2 Main Results

Table 1 lists the test results on Molweni and STAC. Here, we can draw the following conclusions. First, DeepSequential which adopts incremental predicting method does not outperform DeepSequential(NS) on Molweni. This indicates that the severe error propagation causally hurts the model performance. Second, different from the discovery by Shi and Huang [2019], we find that DeepSequential(share) using less parameters is comparable with DeepSequential. Third, our model outperforms all baselines on Molweni and STAC (bootstrap test, p < 0.01), demonstrating its effectiveness and robustness. Fourth, with the better local representations generated by the pretrained model, Our+ELECTRA can be further enhanced. The improvement mainly comes from the gains of relation classification accuaries, which shows that tokenlevel information is very important for capturing the knowledge about relation types.

5.3 Ablation Study

To evaluate the effectiveness of different components in our model, we compare ours with the following variants:

• HGRU+SSA(FixEdgeRep): It is a simplified nodecentric SSA-GNN, where edge representations of each layer are always set as the initial vectors r(0) .
• HGRU+SSA(NodeRep): It uses the concatenated node representations to predict discourse structures.
• HGRU+SSA(ShareEdgeRep): In this varient, edge representations rij and rji of each SSA-GNN layer share the same vector. For fair comparison, we extend its edge hidden size to 256.

4 https://github.com/shizhouxing/DialogueDiscourseParsing

• HGRU+SSA+L∗: It denotes a kind of models trained with our proposed auxiliary loss terms. The following variants are considered: HGRU+SSA+Lcls, HGRU+SSA+Lskd and HGRU+SSA+Lcls+Lskd.
• HGRU+SSA(Teacher): It is the teacher model used to guide the training of our model via structure distillation loss term.

Table 2 reports the ablation experimental results on Molweni, where we have the following observations. First, the extra information learned by our SSA-GNN module is very important for context-aware representations learning. Compared with HGRU, HGRU+SSA(FixEdgeRep) achieves better performance, because our SSA-GNN module helps extract better dialogue features. Second, predicting with edge representations directly can reach better overall performance than that with node representations. Comparing HGRU+SSA with HGRU+SSA(NodeRep), both Link and Link&Rel scores are improved, which indicates the gain of using edge-centric representations. Third, both Lcls and Lskd can improve the performance of our model. The results of +Lskd are better than those of +Lcls since the distillation loss (Lskd) provides more fine-grained (neuron-level) supervision than the label-level classification loss (Lcls). Nevertheless, training with Lcls and Lskd can give further improvements, indicating that they can still provide complementary information. Fourth, by comparing HGRU+SSA with HGRU+SSA(ShareEdgeRep), we find that assigning distinct hidden states for opposite directions (e.g. rij and rji) yields better performance than using the same states for both directions. For this result, we speculate that the former learns more flexible representations, and thus it can better fit the asymmetric fact of each utterance pair on this task.

5.4 Case Study

As shown in Figure 5, given a dialogue example from three speakers, different discourse structures are predicted by these models, where we have the following observations. First, DeepSequential is based on RNN, therefore, it is difficult to correctly predict the long-distance dependencies such as u1 → u6. Besides, it tends to be confronted with the problem of error propagation, which results in errors on u7, u8 given u4 → u6. Second, HGRU+SSA allows more efficiently semantic interaction and does not suffer from error propagation. As a result, it correctly predicts u6, but still fails on u7 and u8. It indicates that this task is still very challenging. Third, with both auxillary losses, our model is able to correctly predict both u7 and u8 due to its better representation learning for intermediate layers.

Surprisingly, all models fail to predict the correct relation type of u4 → u5. This is because that u5 seems to be confusing and not related to the given context. In fact, this phenomenon is common in the actual conversation scenario. Therefore, the application of conversational discourse parsing still faces challenges.

5.5 Accuracy at Different Dialogue Turns

Since the prediction of each EDU may depend on any preceding one, the accuracy may decrease as the dialogue turn increases. Hence, we investigate the accuracy of our model and other system at different dialogue turns, which is shown in Figure 6. Overall, all models have a similar downward trend. DeepSequential exhibits the worst performance especially after the 6th dialogue turn because of error propagation. In comparison to HGRU, HGRU+SSA achieves better performance in most cases, demonstrating the advantage of directly learning edge-specific vectors for EDU pairs and utilizing important dialogue features. Particularly, when introducing the two auxiliary loss terms, the accuracy of our model is further improved for later dialogue turns.

6 Conclusions

In this paper, we propose a Structure Self-Aware model for conversational discourse parsing. Particularly, it adopts an edge-centric GNN to directly learn the implicit structural information between each EDU pair. Besides, we explore two effective auxiliary losses for relation recognition and structure distillation to enhance representation learning. Compared with previous models, ours avoids the serious defect of error propagation, but also makes better use of the structural information in training data. In the future, we plan to continuously refine our model by enhancing its robustness for domain transfer.

Acknowledgements

The project was supported by National Natural Science Foundation of China (No. 62036004, No. 61672440), Natural Science Foundation of Fujian Province of China (No. 2020J06001), Youth Innovation Fund of Xiamen (Grant No. 3502Z20206059), and the Fundamental Research Funds for the Central Universities (Grant No. ZK20720200077)