Multi-Grained Role Labeling Based on Multi-Modality Information for Real Customer Service Telephone Conversation

1 Introduction

Call center plays an important role in customer service of many kinds of companies, such as retailer, bank, mobile service and e-commence. There are some self-service platforms, but currently the quality of automatic processing is not able to meet users' complex requirements. In these cases, users still prefer to give a call to the customer service call center for help.

⇤Corresponding author

Customer service calls include many valuable information. We can get the hot topics, the problems about products and other information that customers are concerned about, and they are helpful for improving product quality. On the other hand, call center service satisfaction can be evaluated according to the conversation content. As far as we know, the analysis in customer service is based on human analysis with sampling. However, it is possible to conduct the analyses in an automatical way in the big data era.

Role recognition is a fundamental work of such automatic analysis. While there are still two problems in telephone conversation role recognition: 1) A telephone call is a continuous audio stream which records the mixed information conveyed by users and agents. Therefore, roles have to be separated in conversations. 2) Unlike previous speaker recognition studies, the speakers in this study are not in a pre-defined set. We do not know who will call customer service for help and there may be thousands of agents online to answer the calls. Before analyzing the customer service calls, the two problems have to be addressed to get satisfactory role labeling results.

In this paper, we propose a four-phase framework for role labeling in practical customer service telephone conversation based on acoustic and textual features. Different from most of the previous work which conducted speaker recognition only with acoustic features, we tried to integrate low level acoustic features with high level textual features. Moreover, we designed a text-based post-processing with the help of semantic information in the conversation to reduce the errors accumulated in previous phases. The results indicate that our framework performs better than the single modality work.

We applied this model to two actual customer service telephone conversation role labelings, a mobile service and a bank dialog datasets. The precision of clusters and roles mapping in both datasets in dialog level is over 99.0%. Compared with only using acoustic features, the accuracy of sentence level labeling achieved 90.4%. which increased by 11.9%.

The main contributions of the work are:

• ÷ We propose a uniformed role labeling framework which utilizes both low level acoustic features and high level text based features. Most of the previous work only concentrated on making use of acoustic features.

• ÷ It is a multi-modality role labeling framework including two labeling steps: dialog level role labeling and sentence level correction which are able to reduce the mistakes generated in previous phases.
• ÷ The proposed approach is domain-independent and can be successfully applied to real scenarios as the groundwork of large scale data analysis.

The remainder of this paper is organized as follows: we introduce related work in Section 2. The overview of the proposed four-phase framework is shown in Section 3. We give the detailed description of each phase in Section 4. In Section 5, we introduce experimental settings with real customer service dialog datasets and report the comparative results. The conclusions and the outline of future work are drawn in Section 6.

2 RelatedWork

In previous work, there are several topics which are related to our role labeling work: speaker diarization and role labeling, automatic speech recognition, and multi-modality work.

Speaker Diarization and Role Labeling

Speaker diarization focuses on grouping speech segments according to the speakers in an audio stream [Tranter et al., 2006], which is critical for automatic audio transcription [Tranter et al., 2006], spoken document retrieval [Wang, 2004] and speaker recognition [Zhou et al., 2012]. It has been studied in conversational telephone speech [Zhao and Fan, 2004], broadcast news data [Barras et al., 2006] and other fields [Pardo et al., 2007].

Speaker diarization systems usually include two core parts: speaker segmentation and speaker clustering. Speaker segmentation splits the audio stream into segments. Windowgrowing-based segmentation [Zhou and Hansen, 2005], fixed-size sliding window segmentation [Malegaonkar et al., 2007] and DISTBIC [Delacourt and Wellekens, 2000] are three popular distance-based segmentation approaches. Then, speaker clustering step groups the segments into speaker clusters, and Hierarchical Agglomerative Clustering (HAC) is usually applied to speaker clustering [Barras et al., 2006]. Each cluster contains the speech segments produced by a speaker. Several ∆BIC based speaker diarization methods are proposed in [Cheng et al., 2010]. Some studies, like [Katharina et al., 2005], [Zhang and Tan, 2008] and [Das, 2011] are aimed at speaker recognition based on acoustic feature.

But different from our work, in most of the previous work, the speakers are in a pre-defined set and the speaker role is indistinct in most of previous work. Therefore, the models cannot be applied to role labeling in practical telephone conversations.

Automatic Speech Recognition

The goal of automatic speech recognition is to handle different speaking styles, channels and environmental conditions as effectively as human does. In the past years, Gaussian Mixture Model (GMM) has remained as the state-of-the-art model to compute probabilities of Hidden Markov Models (HMM) in ASR fields. HMM-GMM based ASR models obtain notable performance with some parameters adjusting methods, such as minimum Bayes risk [Gibson and Hain, 2006] and large margin estimation [Li and Jiang, 2006].

With the development of Deep Neural Network (DNN), which can better combine various features, it is used to replace GMM in ASR field. In previous work, it has shown that DNN-HMM based ASR systems outperforms traditional HMM-GMM based systems in phoneme recognition [Mohamed et al., 2012] and large vocabulary continuous speech recognition task [Seide et al., 2011]. Different work is conducted to find a DNN-HMM ASR model which has better performance and faster training speed, like [Zhou et al., 2012] and [Zhou et al., 2014].

The ASR systems are applied to various fields. ASR is also an important phase in our framework for role labeling which facilitates the transformation from audio to text. After that, we can extract textual features from the text.

Multi-modality Work

As mentioned before, our role labeling work is based on combining low level acoustic features with high level textual features, which is a multi-modality work rather than single modality work that is based on acoustic features.

Many multi-modality studies are applied to various fields. There are several applications of multi-modality features in the phoneme sentiment analysis field. An emotion recognition model using acoustic prosodic information and text semantic labels is proposed [Wu and Liang, 2011]. Text tagged data from twitter and acoustic features are applied in some studies in order to get better speech emotion recognition performance [Hines et al., 2015] . The multi-modality features used in emotion analysis field are analyzed [Cambria et al., 2013]. Furthermore, McAuley et al. integrate images and text features in the recommendation system, which performs better than other systems [McAuley et al., 2015].

Most multi-modality work has better performance than single modality based work, which shows that multi-modality can help the models to become more powerful. To the best of our knowledge, multi-modality has not been applied to role labeling.

3 Multi-modality Role Labeling Framework

In this section, we will introduce our four-phase framework for role labeling in real customer service telephone conversation, which integrates low level acoustic features with high level text content features.

As introduced in Section 1, there are two fundamental challenges in this work: 1) A telephone conversation is a continuous audio stream. Speaker diarization is conducted to get the segments of an audio stream. In practice, the audio segments will be translated into text sentences in ASR phase for further analysis. 2) This study aims at real conversation speaker recognition, e.g the speakers are not pre-defined. Mapping and classification methods which can recognize if the speaker of a sentence is an agent or a user are applied to conduct role labeling to deal with this problem. The flow chart of our framework is drawn in Figure 1. The input and output of each phase are presented in Table 1.

The framework contains four-phase: Speaker Diarization, ASR, Dialog Level Role Labeling, and Sentence Level Role Correction. Speaker Diarization is designed based on acoustic features extracted from the audio stream, using Mel Frequency Cepstrum Coefficient (MFCC) and ∆BIC algorithm. Filter bank feature is used in ASR phase for speech recognition in DNN-HMM model. Dialog Level Role Labeling takes both the outputs of Speaker Diarization and textual features extracted from ASR phase into account and get the primary role labeling results. In the last phase, Sentence Level Role Correction, the labeling results are revised in a fine-grained level, which turns out to be helpful in reducing the error accumulated in the previous phases. In next section, we will introduce each phase in detail.

4 Four-phase Model Construction

4.1 Speaker Diarization Based on Acoustic Feature

The first phase designed for role labeling is Speaker Diarization, to split the audio stream into segments clusters. We assume that the feature vectors of each segment arise from some probability distribution, so we will try to decide if the segments are in the same distribution, which means the segments are given by the same speaker.

Usually, there are only two speakers in a telephone conversation. In service dialog, the two speakers are an agent and a user. After splitting the audio stream into segments, we divide the segments into two clusters in this step based on acoustic features, and they are taken as a prior information in the following phases.

Firstly, a telephone conversation audio stream is splitted into audio segments by silent durations. Apparently, that is just a coarse grained segmentation. Then, a fine-grained segmentation is conducted based on ∆BIC-based algorithm [Cheng et al., 2010]. Given two audio segments represented by feature vectors, X = x1↪ x2↪ ▷▷▷↪ xn and Y = y1↪ y2↪ ▷▷▷↪ yn, the following two hypotheses are evaluated:

H0 means that X and Y are derived from the same multivariate Gaussians distribution, while H1 means that they are from different distribution. The ∆BIC value can be calculated as the difference between the BIC value of H0 and H1 as follows:

The larger the value of ∆BIC is, the less similar the two segments will be. Speaker change point can be located in this way. This segmentation method is window-growing based, in which the segments are sequentially input, and all change points are detected via this method.

At last, hierarchical agglomerative clustering is applied to cluster the segments into two clusters. MFCC feature extracted in audio stream is used in this phase.

4.2 Automatic Speech Recognition with DNN-HMM Model

In the last phase, acoustic features are applied in speaker diarization. The text content of audio stream is very useful in role labeling. For example, if a speaker says: Can I help you? Obviously, this is the agent speaking. Therefore, it is necessary to translate the audio stream into text content. Automatic speech recognition is conducted with the help of the ASR algorithm in this phase. Although both Speaker Diarization and ASR will introduce some errors, they are still valuable compared with the information they bring for role labeling. Moreover, the errors will be fixed in later phases.

DNN-HMM model is applied to implement automatic speech recognition step , which is one of the state-of-the-art ASR model [Povey et al., 2011] and [Zhou et al., 2012]. Figure 2 is the framework of the DNN-HMM model. The filter bank feature is extracted from the audio stream, which is the input of DNN.

Further more, as shown in Figure 1, the inputs of this phase are the acoustic features of audio stream segments splitted in Speaker Diarization, instead of the acoustic features of the whole telephone conversation. The outputs of ASR are text segments.

In practice, a GMM model with HMM model is pre-trained using MFCC feature for initialization. Then, DNN model will be used to replace the GMM model, and the input features are changed into filter bank features. After being trained, this DNN-HMM model becomes the final model for automatic speech recognition. The pre-training step is set up for HMM model parameters initialization, which can save the training time of the DNN-HMM model. Admittedly, other ASR models can also be applied here to get the text content, while DNN-HMM model is chosen considering its better performance.

4.3 Dialog Level Role Labeling

Low level acoustic features and high level text content features are integrated to conduct a coarse-grained level role labeling in this phase. The two audio segments are constructed according to acoustic features, and the text features are extracted from ASR phase. We name this phase coarse-grained level role labeling, because the two clusters are mapped to the two speaker roles, an agent and a user, as dialog level role labeling. The algorithm of textual features extraction and cluster classification for mapping is shown in Algorithm 1. The acoustic feature is remained in dialog segments set, and text feature is applied to construct the mapping relationship between the clusters and speakers.

As shown in the algorithm description, there are 5 steps in dialog level role labeling. Firstly, V is initialized with 0. The length of V is determined by the length of F. Secondly, each dimension of V records the difference between cluster X and Y in word frequency. Notice that in this case, the values of some dimensions can be very large, and it is not excepted in classification. Therefore, in next step, we normalize the value into ¶0,1,2♦. Then, we use a classifier to calculate the mapping relationship: X to A, Y to U or X to U, Y to A. According to the mapping relationship, we replace the role label li in D into l0i , and get D0 . D0 is the role labeling result.

Algorithm 1 Dialog Level Role Labeling Algorithm

Definition: D : Dialog segments set; X and Y are the two clusters; A : Agent; U : User; F : A pre-defined feature words set; V : A map records (word, count), saving the vectorized D; R : Classification result, 1 or 0.

Input: D = ¶(t1,l1), (t2,l2), (t3,l3) ... (tn,ln)♦, ti is segment i's text content, li is a cluster label, li = X or Y .

Initialize: Pre-train a binary-classification classifier.

Output: D0 = ¶(t1,l01 ), (t2,l01 ), (t3,l02 ) ... (tn,l01 )♦, l0i = A or U

1: Initial map V with F, the feature words are mapped with 0 and stored in V .

• 2: Traverse the set D and accumulate the frequency of each feature word, and refresh the V . Notice that if the label of the word is Y , we will minus the count of this word in V .
• 3: Normalized the values of each key in V into 0,1,2. Positive number, 0 and negative number will be replaced by 0,1and 2.
• 4: Take V as the input of classifier, get the classification result.
• 5: Refresh the labels in D with label U and A according to classification result, get D0 .

In Section 5, we will show that different classifiers have been used in this phase, and they all have a good performance.

4.4 Sentence Level Labeling Correction

From the 3 phases above, we will get role labeling results, while that is not enough. Even though the mapping relationship between clusters and roles is perfectly constructed in Dialog Level Role Labeling phase, the role labeling results in sentence level might be wrong. Since there are errors made and accumulated in the Speaker Diarization and ASR phases, we need to correct the role labeling results. The two phases are based on acoustic features, meaning that it is hard to correct it with only acoustic features. In this phase, textual features are used to deal with the mistakes accumulated in previous phases.

A basic assumption in this phase is that most of sentences are labeled correctly, and we will modify the sentences that are highly possible to be falsely labeled. The feature words to vectorize the sentences were strictly selected, and a probability algorithm - logistic regression is applied. The text features used here are also bag-of-words. Different from phase 3, the feature words in this phase are selected according to relative entropy, which is computed with the probability of the words in word set, rather than the frequency of them. The high frequency words perform well in vectorization, while they may be unqualified for sentence level role classification, because they are frequently used by both speakers. On the other hand, the high relative entropy words are usually the typical words that are distinguishable.

Basic symbol notations are defined in Table 2, and the formulations to calculate the relative entropy of word x are defined as following:

The probability of each word for relative entropy calculating depends on the train set. And the top ten words in REP↪Q↪x and REQ↪P↪x are used as feature words for sentence vectorization to select the most discriminative features. Some sentences may vectorized into zero vector, and these sentences will not be modified in this phase due to the low correction confidence.

Logistic regression is leveraged as the correction method in this phase. X represents a vectorized sentence s. The probability of s being said by an agent or a user is calculated by the following formulations:

In traditional logistic regression steps, the classification result is determined by whether the possibility is larger than 0.5. But in this phase, only the labeling results which have enough confidence will be revised. The label of a sentence will be modified into "agent" or "user" if and only if the possibility satisfied one of the following two inequations.

These are the steps we designed for correction, and the outputs of this phase are the final role labeling results. The experiments in Section 5 verify that our probability based correction method helps in getting better performance.

5 Experiments

In Section 3 and 4, we introduced our four-phase model for role labeling. In this section, we will report the experiment results which are conducted in real customer service dialog datasets based on our framework .

5.1 Dataset

The datasets used in our work come from two different fields, a mobile service call center and a bank service call center, and both of them are real customer service telephone conversations. All conversations are in Chinese.

Mobile Service Dialog Dataset (MSDataset) , contains 34 telephone conversations with over 2,000 sentences. This is a normal size for acoustic recognition, like [Malegaonkar et al., 2007] and [Cheng et al., 2010]. We also adopt our method to a much larger dataset, Bank Service Dialog Dataset (BKDataset), which contains 85,336 conversations, to test whether our method is effective.

5.2 Evaluation

We chose 3 evaluation metrics to evaluate the labeling results: AccuD: the accuracy of dialog level labeling (mapping relationship between clusters and speakers). AccuS: the accuracy of sentence level labeling result. AccuT: the accuracy of time level labeling result, which considers the percentage of time that the correct labeled sentences occupied. In addition, we used segmentation error rate (SER) to evaluate the speaker diarization result in phase one following previous approaches [Barras et al., 2006], which takes two kinds of errors into account: missed speech (MiS) and false alarm speech (FaS).

We labeled some conversations to evaluate the experiment results. Each sentence in the MSDataset is labeled with speaker role by hand according to audio and text content. Then, we selected 1,000 conversations from the BKDataset based on stratified sampling according to the length of the conversations and labeled the dialog level roles.

5.3 Results

Firstly, compared with human recognition results, the segmentation error rate in Speaker Diarization on dataset is 14.28% (9.53% in FaS and 4.75% in MiS). The error rate can be reduced by using better speaker diarization methods, while we did not concentrate on dealing with it in this study.

Dialog Level Role Labeling

As introduced in Section 5.1, there are two dialog datasets, and the BKDataset is larger than the other. Our dialog level role labeling experiment is conducted on BKDataset at first. The 1,000 sampled conversations are used for classification. Each conversation is labeled by two professional annotators. If their opinions are different, they would discuss carefully until an agreement is reached.

We use bag-of-words features for classification. Notice that not all of the words are used to vectorize dialog D, because the vector will be too long if so. We only select the top 20 frequent words in the two clusters separately to construct the bag-of-words vector. As mentioned in Section 4.3, a binary classifier is used for classification. We have adopted different methods: Decision Tree, SVM, Naive Bayes, and so on. In 5-fold cross validation, the performance of Decision tree is the best which accuD achieves 99.5%. Moreover, other methods also get more than 97.8% in accuD, which indicates that our framework has a steady performance.

We applied the trained Decision Tree classifier to MBDataset. The accuD is 97.1%, only 1 of the 34 telephone conversations get a wrong labeling result, indicating that the dialog level role labeling classifier is domain-independent.

Sentence Level Correction

However, when we concern about sentence level labeling accuracy, the AccuS is 87.1 % and the AccuT is 88.2% in MBDataset (The accuracy reported is calculated in the right recognized sentences). That is due to mistakes made in previous phases. The mistakes cannot be fixed with only dialog level labeling. Logistic regression model is adopted in correction.

The feature extraction steps and correction algorithm are introduced in Section 4.4. The AccuS's variation with the increase of ⌧ (in 5-fold cross validation) is drawn in Figure 3. As illustrated in Figure 3, the blue line is the AccuS before correction, and the red line records the AccuS after correction with ⌧. The corrected results have worse performance than before because when ⌧ is low, some modifications are not with enough confidence. There are some sentences even modified into wrong label. With the increase of ⌧, AccuS increases. Specially, at the same time, fewer sentences are checked and modified, for the reason that both possibilities are lower than 0▷5+⌧ . So the AccuS drops when ⌧ is over 0.4. The accuracy of logistic classification result continuously increases when we use larger ⌧, and the classification result achieves 100% when ⌧ is larger than 0.47. The best performance of AccuS achieves 90.5% when ⌧ equals to 0.39.

In Figure 4, AccuT shows similar results with AccuS with the increase of ⌧. AccuT is sensitive with the length of correctly labeled segments. The accuracy before correction is higher than AccuS, showing that it is easier to label the long segments correctly. And the accuracy dropping less than AccuT means that most modified labeling segments are short.

Comparison with Other Methods

In this part, we compared our framework with several other methods. To the best of our knowledge, there is not an appropriate solution. Therefore, all baselines are based on the fourphase framework: 1) Keywords based dialog labeling with acoustic features. 2) Labeling without segment clustering based on text features. 3) Role labeling without correction. The experiment results are presented in Table 3. It is obvious that our acoustic and textual features based framework performs better than other methods that use a single type of feature do. The correction step helps improve the final results.

6 Conclusions and FutureWork

In this paper, we present our work on role labeling in real customer service telephone conversation. This multi-modality work is based on both acoustic and textual features. Differing from previous speaker recognition work, our goal is not mapping the speaker into a pre-defined people group.

We propose a four-phase framework for role labeling: Speaker Diarization, ASR, Dialog Level Role Labeling, and Sentence Level Role Correction. Speaker Diarization and ASR are based on acoustic feature of a telephone conversation, which are the basic steps of role labeling. The clustering results in Speaker Diarization and text features extracted in the output of ASR are used for Dialog Level Role Labeling. With the help of Decision Tree, role mapping accuracy is over 99.0%. In the last phase, logistic regression is applied to labeling result correction based on text features, which improves the performance. The final accuracies in sentence level and time level achieve 90.4% and 89.6%.

To the best of our knowledge, this is the first work that takes both acoustic and textual features into consideration in role labeling. Our multi-grained framework performs better than other methods in realistic datasets based experiments.

Our future work includes two parts: 1) We will try to use better speaker diarization method to minimize the mistakes in segments clustering, which will be helpful in improving the final performance. 2) As mentioned before, this is the fundamental work of customer satisfaction evaluation and customer service evaluation, and we would like to go further in customer service telephone conversations analysis.

Acknowledgments

We thank Ji Cao, Libo Yang for their insightful discussions and help. This work was supported by National Key Basic Research Program (2015CB358700), Natural Science Foundation (61472206, 61073071) of China and joint project of Beijing Sino Voice Technology, Co., Ltd.