Conversational Semantic Role Labeling with Predicate-Oriented Latent Graph

1 Introduction

Semantic Role Labeling (SRL) as a shallow semantic structure parsing task aims to find all the arguments for a given predicate [Gildea and Jurafsky, 2000; Marcheggiani and Titov, 2017; Strubell et al., 2018; Fei et al., 2020d; Fei et al., 2021b]. Conversational SRL (CSRL) is a newly proposed task by Xu et al. [2021], which extends the regular SRL into multi-turn dialogue scenario. As illustrated in Fig. 1, CSRL is characterized by that, the predicate is given at current utterance, while the correlated arguments are scattered in the history utterances of the dialogue that are generated by two speakers. So far, few attempts have been made for CSRL [Xu et al., 2021; Wu et al., 2021b; Wu et al., 2021a], where, unfortunately, several key CSRL

∗Corresponding author

characteristics are still remained unexploted, which may hamper the further task improvements.

First of all, intuitively SRL structure echoes much with the syntactic dependency structure [Strubell et al., 2018; Marcheggiani and Titov, 2017], and the existing regular SRL works frequently employ external structural information for performance enhancement, i.e., providing additional prior links between predicates and arguments. However, it is quite intractable to directly employ the external syntax knowledge into CSRL for some reasons. For examples, a dependency tree takes one single sentence piece as a unit, while a dialogue could contain multiple utterance sentences; the parse trees from third-party parsers inevitably involve noises; only a small part of the dependency structure can really offer helps, rather than the entire tree [He et al., 2018]. Second, the predicate-argument structures in CSRL are broken down and scattered into different utterances, which makes the detection of the CSRL more challenging. Actually the chances are much higher for the predicate to find its arguments when they are being closer, i.e., near-neighbor influence. In other words, nearer history utterances will show more impacts to the latest utterance. 1 Fig. 1 exemplifies the case.

Based on the above observations, in this paper we present an effective CSRL method with an innovative predicateoriented latent graph (namely, POLar). Unlike the explicit syntactic structures, we make use of a two-parameter HardKuma distribution [Bastings et al., 2019] to automatically induce latent graph from task's need (cf. §4). Particularly, we propose a predicate-centered Gaussian inducer for yielding the latent edges, by which the nearer and informative words to the predicate will be placed with more considerations. The POLar is then dynamically pruned, so that only the taskrelevant structure will be built, while the irrelevant edges are droped. The overall CSRL framework is differentiable and performs predictions end-to-end (cf. Fig. 2).

1 Our data statistics shows that, cross-one-utterance arguments account for 60.3% among all cross-turn arguments; while the ratio decreases to 30.3% for cross-two-utterance arguments.

The BERT [Devlin et al., 2019] pre-trained language model (PLM) is extensively employed in existing works for CSRL performance boosts [Xu et al., 2021; Wu et al., 2021a]. Nevertheless, it could be problematic to directly leverage BERT for CSRL. On the one hand, one entire dialog often consists of far more than two utterance sentences, while the raw BERT restricts the input with at maximum two sentence pieces, which consequently limits the PLM's utility. Therefore, we consider adopting the DiaBERT [Liu and Lapata, 2019; Li et al., 2020], which is designed for well supporting multiple utterance inputs and thus yields better dialogue-level representations. On the other hand, we note that in CSRL both two speakers use the personal pronoun in their own perspective (i.e., 'I', 'you'), and directly concatenating the multiturn utterances into PLMwill unfortunately hurt the speakerrole consistency, i.e., speaker coreference issue. Therefore, we introduce a coreference-consistency-enhanced DiaBERT (namely CoDiaBERT, cf. Fig. 3) that enhances the speakerrole sensitivity of PLM with a pronoun-based speaker prediction (PSP) strategy.

Our system significantly outperforms strong-performing baselines with big margins on three CSRL benchmarks. In particular, over 4% F1 score of improvement is achieved for detecting the cross-utterance type of arguments. Further analyses reveal the usefulness of the proposed latent graph and the dynamic pruning method, as well as the CoDiaBERT PLM. Also we show that our model effectively solves long-range dependence issue. Overall, we make these contributions:

• We for the first time propose to improve the CSRL task by incorporating a novel latent graph structure.
• We construct a predicate-oriented latent graph via a predicate-centered Gaussian inducer. The structure is dynamically pruned and refined for best meeting the task need.
• We introduce a PLM for yielding better dialogue-level text representations, which supports multiple utterance sentences, and is sensitive to the speaker roles.
• Our framework achieves new state-of-the-art CSRL results on three benchmark data.

2 RelatedWork

The SRL task aims at uncovering the shallow semantic structure of text, i.e. 'who did what to whom where and when'. As a fundamental natural language processing (NLP) task, SRL can facilitate a broad range of downstream applications [Shen and Lapata, 2007; Liu and Gildea, 2010; Wang et al., 2015]. By installing the current neural models, the current standard SRL has secured strong task performances [Strubell et al., 2018; Li et al., 2019; Fei et al., 2021c]. Recently, Xu et al. [2021] pioneer the task of CSRL by extending the regular SRL into multi-turn dialogue scenario, in which they provide benchmark datasets and CSRL neural model. Later a limited number of subsequent works have explored this task [Wu et al., 2021b; Wu et al., 2021a], where unfortunately several important features of CSRL are not well considered. In this work, we improve the CSRL by fully uncovering the task characteristics.

This work also closely relate to the line of syntax-driven SRL [Marcheggiani and Titov, 2017; Fei et al., 2020c; Fei et al., 2020b]. For the regular SRL, the external syntactic dependency structure is a highly-frequently equipped feature for performance enhancement, as the SRL shares much underlying structure with syntax [He et al., 2018; Fei et al., 2020a; Fei et al., 2021a]. However, it could be problematic for CSRL to directly benefit from such convient syntactic knowledge, due to the dialogue nature of the text as we revealed earlier. We thus propose to construct a latent structure at dialogue level, so as to facilitate the CSRL task with structural knowledge. In recent years, constructing latent graph for downstream NLP tasks has received certain research attention [Choi et al., 2018]. As an alternative to the pre-defined syntactic dependency structure yielded from third-party parsers, latent structure induced from the task context could effectively reduce noises [Corro and Titov, 2019], and meanwhile enhance the efficacy (i.e., creating task-relevant connections) [Chen et al., 2020]. In this work, we revisit the characteristic of CSRL, and based on the two-parameter Hard-Kuma distribution [Bastings et al., 2019] investigate a predicate-oriented latent graph by proposing a predicate-centered Gaussian inducer.

3 CSRL Framework

Task modeling. Consider a conversation text U={ut}Tt=1 (T is the total utterance number), with each utterance ut={w0,w1, · · · } a sequence of words (w0 is the utterance speaker). In CSRL the predicate prd is labeled as input at the current (lastest) utterance uT . We follow Xu et al. [2021], modeling the task as a sequence labeling problem with a BIO tagset. CSRL system identifies and classifies the arguments of a predicate into semantic roles, such as A0, A1, AM-LOC, etc, where we denote the complete role set as R. Given U

and the predicate prd, the system finally assigns each word w a label ˆ y ∈ Y, where Y=({B, I}×R) ∪ {O}.

Framework overview. Our overall CSRL framework is illustrated in Fig. 2. The dialogue encoder first yields contextual representations for the input dialogue texts. Then, the system generates the predicate-oriented latent graph (i.e., POLar induction), and performs structure pruning. Afterwards, GCN layers encode the POLar into feature representations, based on which the predictions are finally made.

3.1 CoDiaBERT: Dialogue Encoder

Contextualized word representations from BERT have brought great benefits to CSRL [Xu et al., 2021; Wu et al., 2021b; Wu et al., 2021a]. In this work, we follow them by borrowing the advances from PLM as well. However, we notice that the raw BERT limits the input with maximum two sentence pieces, while often a conversation text can comprise far more than two utterance sentences. Directly using BERT can thus lead to discourse information incoherency. We thus leverage a dialogue-level BERT-like PLM DiaBERT [Liu and Lapata, 2019]. Technically, we pack the utterance with its speaker as a group, and concatenate those groups into a whole (separated with SEP tokens), and feed into the PLM encoder.

The speaker coreference issue in conversational context may quite confuse the model. For example, speaker #1 would call speaker #2 'you' in speaker #1's utterance, while both speaker #1 and speaker #2 call themselves with the first-person pronoun 'I'. To strengthen the sensitivity of the speaker role, we further retrofit the DiaBERT so as to enhance the coreference consistency, i.e., CoDiaBERT. Specifically, we based on the well-trained DiaBERT perform a pronounbased speaker prediction (PSP) upon DiaBERT, as shown in Fig. 3. We first concatenate different utterance texts into a whole piece that are separated with <SEP> token. Then we prepare three types of embeddings for each input token: 1) word embedding xw , 2) speaker id embedding xq , and 3) position embedding xp , all of which are fed into PLM for PSP:

Based on the pronoun representation (i.e., the corresponding word is a pronoun), we encourage the PLM to predict the speaker id.

After PSP, the CoDiaBERT could yields better dialogue representations. In our CSRL framework, CoDiaBERT will take as input the conversation texts (including the speaker id) as well as the predicate word annotation:

where xprd is the predicate binary embeddings {0, 1} indicating the presence or absence of the predicate word prd. hi denotes the output representation for the input token wi.

3.2 Latent Graph Encoder

Based on the CoDiaBERT representation 2 we can construct the POLar structure, which we will elaborate in the next section (cf. §4). In the POLar G = (V,E), each edge πi,j ∈ E is a real value that denotes a latent connection between node vi ∈ V to node vj ∈ V with a connecting intensity, i.e., E is a K × K adjacent matrix (|V | = K). 3 Once we obtain the POLar we encode it into feature representations. Specifically, we employ a multi-layer (M) graph convolutional network (GCN) [Marcheggiani and Titov, 2017]. We denote the m-th layer of GCN hidden representation of node vi as rm i :

where ¯ A = E + I (I is a K × K identity matrix), di = ∑Kj=1 Ei,j is for node normalization. Note that the input of the initial layer is the CoDiaBERT representations, i.e., r0 i = hi After total M layers of message propagations, we expect the GCN can sufficiently capture the structural features.

3.3 Decoder and Training

To take the full advantages of the global dialogue contextual features, we create a residual connection from CoDiaBERT to the end of the GCN layer:

where ei is the final feature representation, which fuses both the contextual features and the structure-aware features. gi is a gate mechanism that is learned dynamically:

Based on ei we adopt a Softmax classifier to predict the labels for tokens:

Also the Viterbi algorithm is used to search for the highestscoring tag sequence ˆ Y = {ˆ y1, · · · , ˆ yK}.

Our training objective is to minimize the cross-entropy loss between the predictions ˆ Y and the gold labels Y .

where K is the total sequence length (i.e., |V |).

4 Predicate-Oriented Latent Graph Induction

Since the goal of CSRL is to find the arguments of the predicate, it is crucial to treat the predicate word as the pivot and induce a predicate-oriented latent graph (POLar) to fully consider the near-neighbor influence. Here we demonstrate how to develop the POLar structure. First, we give a description on the theoretical fundamentation of the HardKuma distribution, upon which we build the latent strucutre. Then we introduce the predicate-centered Gaussian inducer. Finally we present the method for dynamically pruning the POLar.

2 We abandon the special sentinel tokens (e.g., <CLZ> and <SEP>), and only make use of the word and speaker tokens.

3 The node could either be a word, or a speaker.

4.1 HardKuma Distribution

HardKuma distribution [Bastings et al., 2019] is derived from the Kumaraswamy distribution (namely Kuma) [Kumaraswamy, 1980], which is a two-parameters distribution over an open interval (0, 1), i.e., t ∼ HardKuma(a, b) where a ∈ R>0 and b ∈ R0 are the parameters controlling the shapes. However, the Kuma distribution does not cover the two discrete points 0 and 1. Thus, the HardKuma distribution adopts a stretch-and-rectify method to support the closed interval of [0, 1]. This feature allows to predict soft connections probabilities between input words, i.e., a latent graph, where the entire process is fully differentiable.

First, we sample a variable from a (0,1) distribution, i.e., U ∼ U(0, 1), based on which we generate another variable from HardKuma's inverse CDF function:

Then we stretch the k into t:

where l < 0 and r > 1 represent an open interval (l,r). 4 A Hard-Sigmoid function rectifies the t into h via

In short, we can summarize the HardKuma distribution as:

For more technical details we refer the readers to the raw papers [Bastings et al., 2019].

4.2 Predicate-centered Gaussian Inducer

By sampling variables from HardKuma distribution with trained parameters a and b, we can generate the latent graph based upon the dialogue. Specifically, we present a predicatecentered Gaussian inducer (PGI), so that the near neighbors to predicate that carry more important information would serve more contributions.

4 Following the standard setup in Bastings et al. [2019], l=-0.1 and r=-1.1.

As depicted in Fig. 4, we first upgrade each token representation into h′i with the prior of predicate word, via a predicate-centered Gaussian operator:

where d = |i - i(prd)| is the edit distance between a token wi and the predicate prd. Here f(d) is a Gaussian distance, i.e., f(d) = exp(-πd2 ). So h′i is reduced into:

Based on h′i , we then create the parameter context representations (i.e., denoted as sa and sb ) via separate feedforward layers (i.e., sa/b i =FNNa/b (h′i )). Then we build the prior parameter representations of the distribution:

Thereafter, we can sample a soft adjacency matrix between tokens, i.e., πi,j ∈ E:

4.3 Dynamic Structural Pruning

There are high chances that the induced POLar structure is dense, which would introduce unnecessary paths that are lessinformative to the task need, i.e., noises. Therefore, we adopt the α-Entrmax [Correia et al., 2019] to prune the POLar. αEntrmax imposes sparsity constraints on the adjacency matrix E, and the pruning process automatically removes irrelevant information according to the contexts dynamically:

where α is a dynamic parameter controlling the sparsity. When α=2 the Entrmax becomes a Sparsemax mapping, while α=1 it degenerates into a Softmax mapping.

5 Experimentation

5.1 Setups

We conduct experiments on three CSRL datasets [Xu et al., 2021], including DuConv, NewsDialog and PersonalDialog, with average 10.1, 5.2 and 6.1 utterances per dialogue, respectively. All the three data is in Chinese language. We take the default data split as in Xu et al. [2021], where DuConv has the 80%/10%/10% ratio of train/dev/test, while NewsDialog and PersonalDialog are taken as out-of-domain test set. Our CoDiaBERT shares the same architecture with the official BERT/DiaBERT (Base version), and is further posttrained on the CSRL data with PSP strategy. GCN hidden size is set as 350. We adopt Adam as the optimizer with an initial learning rate of 5e-4 with weight decay of 1e-5. The initial α value is 1.5. To alleviate overfitting, we use a dropout rate of 0.5 on the input layer and the output layer.

We mainly make comparisons with the existing CSRL baselines, including CSRL [Xu et al., 2021], CSAGN [Wu et al., 2021b] and DAP [Wu et al., 2021a]. Also we implement several representative and strong-performing models designed for regular SRL, including UE2E [Li et al., 2019], LISA [Strubell et al., 2018] and SynGCN [Marcheggiani and Titov, 2017], in which we concatenate the utterances into a long sequence. In particular, LISA and SynGCN use the external syntactic dependency trees. Follow Xu et al. [2021], we compute the F1 score for the detection of intra-/crossutterance arguments (i.e., F1intra and F1cross), and the overall performance (F1).

5.2 Results and Discussions

Main results. Table 1 presents the main performances by different models, from which we gain several observations. First of all, our proposed POLar system significantly outperforms all the baselines by large margins on both the in-domain and out-domain datasets, which demonstrates the efficacy of our method. Specifically, we notice that our model achieves at least 4.07%(=90.75-86.68) and at most 7.71%(=45.47-37.76) F1 improvements on the crossutterance argument detection, over the corresponding best baselines. This significantly proves the superiority of our method on the cross-turn context modeling. Second, by comparing the results with BERT and with CoDiaBERT, we know that our proposed CoDiaBERT PLM is of prominent helpfulness for the task. Third, we see that with the aid of external syntactic dependency structure information, SynGCN and LISA models achieve considerable performance gains over the existing CSRL baselines (i.e., CSAGN, DAP). However, such improvements are limited to the detection of intrautterance arugments, contributing less to the cross-utterance arugments. The possible reason is that, the dependency tree only works at sentence level, which fails to capture the crossuttereance contexts. Fortunately, our proposed latent graph can nicely compensate for this.

Ablation study. In Table 2 we give the model ablation results with respect to the CoDiaBERT PLM and the POLar parts, respectively. We can observe that, by replacing the CoDiaBERT with a vanilla BERT or removing the pronounbased speaker prediction policy (downgraded as DiaBERT), there can be considerable drops. If we strip off the PSP, and instead use the speaker id indicator to label the speaker pronoun (i.e., spk-lb), we also witness the drops.

Further, without the PGI for the latent graph induction, i.e., directly feeding the PLM representations h in Eq. 14 instead of s, we can receive the most significant performance drops among all the other factors, i.e., -5.58%F1 on the crossutterance arguments detection. This also reflects the importance to handle the near-neighbor influence of CSRL. Besides, the graph pruning is quite important to the results of cross-utterance arguments. The gating mechanism takes the positive roles to the system.

Performances on cross-utterance argument detection. In Fig. 5 we study the error rate on the cross-utterance argument detection. We see that with the increase of the crossed utterances, the error for the argument detection grows universally. But in all the cases, our POLar system commits nearly half error rate, comparing to baselines. Also we notice that, both the PGI mechanism and the CoDiaBERT is important to our system, with the former more significant than the latter.

Impacts of utterance numbers. Intuitively the more the utterance in a dialogue the severe complexity of the speaker parties, i.e., due to the speaker coreference issue. Fig. 6 further plots the performances under different numbers of dialogue utterances. It is clear that increasing the utterance number in a dialogue worsens the overall results, especially when the number ≥11. In particular, the removal of PSP in CoDiaBERT shows greater impact to the removal of the PGI mechanism. This indirectly proves that CoDiaBERT can help solve the speaker coreference issue, which gives rise to the performance gains.

Solving long-range dependence issue. Structure information has been shown effective for relieving the long-range dependence issue in SRL [He et al., 2018; Fei et al., 2021a]. Here we explore the performances when the distances between the arguments and the predicates are different in the dialogue. Fig. 7 shows that, notably, our system equipped with the latent graph performs well for those super-long argumentpredicate distances, where the other baselines could fail. Also the ablated POLar system (w/o PGI) reflects the importance of the predicate-certered Gaussian mechanism.

Study of the dynamic pruning for latent graph. Finally, we investigate the process of the dynamic pruning by studying the changing pattern of α-Entrmax (Eq. 16). Fig. 8 plots the learning trajectories of parameter α as well as the variations of the correlated task performances (on three datasets). We see that, along the training process, the α soon decreases to 1.35 from 1.5 at step 1,500, and then grow to 1.9, during which the latent graph becomes dense and then turns sparse gradually. At the meantime, the CSRL performances climb to the top slowly. This suggests that the dynamic pruning process improves the quality of POLar, which helps lead to better task demand of structure.

6 Conclusions

In this work we investigate the integration of a latent graph for conversational semantic role labeling. We construct a predicate-oriented latent graph based on the two-parameter HardKuma distribution, which is induced by a predicatecentered Gaussian mechanism. The structure is dynamically pruned and refined to best meet the task need. Also we introduce a dialogue-level PLM for yielding better conversational text representations, e.g., supporting multiple utterance sentences, and being sensitive to the speaker roles. Our system outperforms best-performing baselines with big margins, especially on the cross-utterance arguments. Further analyses demonstrate the efficacy of the proposed latent graph as well as the dialogue-level PLM, respectively. Automatically inducing task-oriented latent structure features for the structural parsing tasks is promising, which we leave as a future work.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No.61772378, No. 62176187), the National Key Research and Development Program of China (No. 2017YFC1200500), the Research Foundation of Ministry of Education of China (No.18JZD015), the Key Project of State Language Commission of China (No. ZDI135-112) and the Science of Technology Project of GuangZhou (No. 20210202607).