Discriminatively Trained Recurrent Neural Networks for Continuous Dimensional Emotion Recognition from Audio

1 Introduction

Continuous dimensional emotion recognition from audio is a sequential learning problem that has attracted increasing attention in the past few years [Coutinho and Cangelosi, 2010; Schmidt and Kim, 2011; Metallinou et al., 2013; Soleymani et al., 2013; Wang et al., 2015]. There, sequences of acoustic features have to be mapped to emotion contours in several dimensions that represent the emotion communicated by means of audio, e.g., speech utterances or excerpts of music. Typical emotion dimensions comprise arousal and valence [Russell, 1980], as explored in this study, although other dimensions such as dominance and expectation can be added [Eyben et al., 2012]. Defining the target labels as real-valued mappings from time instants to targets helps capturing the temporal dynamics of emotion, which cannot be assumed to be constant over time [Schmidt and Kim, 2011]. To learn such mappings, deep recurrent neural networks are a promising model [Coutinho and Cangelosi, 2010], as they take into account temporal dependencies in inputs and outputs and can handle correlated features.

Continuous emotion recognition is typically evaluated in terms of the correlation between the learner's outputs and the target values (such as by the correlation or determination coefficient), as well as the average deviation of outputs and targets, such as by the mean linear or mean squared error (MLE/MSE) [Schuller et al., 2012; Jenke et al., 2013]. Since neural networks are usually trained using criteria such as the (root) MSE, this only takes into account the latter while neglecting the former. Further, although it is well known that the minimization of the MSE and the maximization of the (Pearson) correlation coefficient (CC) are equivalent if the outputs and targets are standardized, such standardization cannot be assumed in emotion regression, as the emotional intensity, and hence the mean and variance, is of crucial importance.

Moreover, the CC is insensitive to scaling and shifting, which is problematic for training neural networks with this metric. Imposing a cost function based on CC may lead to an infinite number of local minima with different prediction behavior. In fact, a neural network trained on CC cannot learn the correct scales and offsets from the target values ('goldstandard') because the CC is not sensitive to such variations. We illustrate those issues on different variants of the same time-series in Figure 1. Because CC is insensitive to scaling and shifting, it always provides the same perfect prediction score (CC = 1▷00) on different versions (shifted and/or scaled) of the same time-series, although such variations are actually far from a perfect prediction. As a result, the CC is not sufficient as evaluation criterion in practice, and additional measures such as mean squared error need to be taken into account.

From the above considerations, we can conclude that firstly, the usual objective functions for neural network regression do not fully match the evaluation criteria used for continuous dimensional emotion recognition, and secondly, the use of CC as objective function cannot lead to satisfying results. To alleviate these problems, we propose to use the concordance correlation coefficient (CCC) [Lin, 1989] as a differentiable objective function that unites both correlation and mean squared error, and can be thought of as a CC that enforces the correct scale and offset of the outputs. As a result, CCC takes into account the effects of shifting and scaling the prediction when computing the performance (cf. Fig. 1), and can thus be used as an objective function to train neural networks for time-continuous prediction tasks.

In the following, we will show that the choice of objective function (sum of CCCs per sequence, overall CCC, or MSE) for network training significantly influences the evaluation outcome on standard corpora for continuous dimensional emotion recognition from music and speech.

The remainder of this paper is as follows: we describe some related work on task-specific discriminative objective functions for neural network training in section 2, we introduce different discriminative objectives for emotion regression in section 3, and the optimization of CCC by stochastic gradient descent in section 4, we evaluate the performance in dimensional emotion recognition tasks (arousal and valence) for different objective functions on two different corpora (music and speech) in section 5, and provide a conclusion in section 6.

2 RelatedWork

Task-specific discriminative objective functions for neural network training are well known. For example, in training networks for automatic speech recognition, the minimum phoneme error or minimum Bayes risk objectives are used in place of the standard cross entropy objective [Vesel´y et al., 2013; Kingsbury et al., 2012]. In [Weninger et al., 2014], it is proposed to optimize the prediction of time-frequency masks for acoustic source separation based on local signalto-noise ratio rather than MSE. Joint optimization of masking functions and deep recurrent neural networks has been investigated for monaural source separation tasks by [Huang et al., 2015], with a discriminative criterion designed to enhance the source to interference ratio.

Regarding neural network approaches for emotion recognition, most of the existing work has been focused on classification tasks. Deep Neural Network Hidden Markov Models (DNN-HMMs) were investigated with discriminative pretraining and restricted Boltzmann Machine (RBM) based unsupervised pre-training by [Li et al., 2013]. Experimental results have shown the superiority of the hybrid DNNHMMs with discriminative pre-training in comparison with other models, such as GMM-HMMs and MLP-HMMs. Another hybrid architecture combining DNN with an Extreme Learning Machine (ELM) was also successfully utilized for classifying emotion from utterances in [Han et al., 2014].

Continuous prediction of dimensional emotion was investigated with a Deep Belief Network in [Schmidt et al., 2012]. [Eyben et al., 2012] dealt with multi-task recurrent neural network based speech emotion regression. The winning team of the last edition of the Audio-Visual Emotion recognition Challenge [Ringeval et al., 2015b] employed bidirectional long short-term memory recurrent neural networks (DBLSTM-RNN) to perform unimodal emotion recognition and multimodal fusion [He et al., 2015]. While these works use a similar learning framework as in our paper, none of them uses the discriminative objective based on correlation coefficients as introduced below.

3 Discriminative objectives for emotion regression

In the following, we will introduce two different objectives, one based on the CCC for each sequence, and one based on the overall CCC. We will denote by yif the regression outputs for sequence i and target variable f (in case of neural networks, the sequences of activations of unit f of the output layer), while yif ⇤ denotes the corresponding training targets (i.e., gold-standard). The standard sum of squared errors (SSE) training objective for a mini-batch B is given by

where F is the set of target variables (e.g., arousal, valence, etc. in the case of emotion recognition) and t denotes the index of a time step at which the target variable is annotated.

While the above objective is discriminative, it is not discriminative on the sequence level. Let us thus introduce the proposed objective function based on the CCC per training sequence. The total cost function O is:

This objective will be denoted by ⌃CCC below, as it is computed on all training sequences i available in the mini-batch B. The CCC per sequence i and target f is defined in accordance with [Lin, 1989] as:

where E, Var, and Cov denote sample mean, variance, and covariance, respectively.

Let us consider the mean squared error, E¶(yif - yif ⇤ )2 ♦, which is equivalent to Var¶yif - yif ⇤ ♦ + E¶yif - yif ⇤ ♦2 = Var¶yif ♦ + Var¶yif ⇤ ♦ + E¶yif - yif ⇤ ♦2 - 2Cov¶yif ↪ yif ⇤ ♦. Based on this observation, we can rewrite the CCC as:

defining N i as the length of sequence i, the covariancerelated quantity as:

and the sum of squared errors (SSE) related quantity as:

An alternative objective to maximize (denoted simply by CCC below) is the 'total' CCC on the training set. This can be achieved by simply considering the entire training set as a single sequence i in (3). As shown in Fig. 2, the ⌃CCC objective differs from the CCC objective in that it necessarily enforces accurate prediction of the target contour within each sequence, while the CCC objective could assign a good score to over-smoothed regression outputs that only predict the average label right. Conversely, if the target label has low variance within the sequences, the ⌃CCC objective is hard to optimize and might emphasize on noise in the 'gold-standard', which is often given in emotion recognition. Thus, which objective is preferable certainly depends on the application.

Note that since the CCC on two partitions of the training set is not equivalent to the sum (or average) of the CCCs on these two partitions, it is not directly possible to optimize the total CCC on the training set, unless the mini-batch size comprises the total training set, which might be impractical. This is in contrast to SSE training and the ⌃CCC maximization, where batch learning can be implemented by summing up the gradients from the mini-batches.

In the case of mini-batch learning with ♣B♣ = 1 (one sequence per mini-batch), the optimization of ⌃CCC and CCC is equivalent. However, in case of recurrent neural network training as considered here, ♣B♣ ≫ 1 is required for efficiency [Weninger et al., 2015].

Further, we could also define an objective in analogy to (2) yet based on the Pearson's CC,

where σ denotes the standard deviations of outputs and targets in analogy to the above. However, since the Pearson CC is invariant w.r.t. the scale of the network outputs (in contrast to the CCC, cf. Fig. 1), this function has infinitely many minima, making its minimization hardly feasible.

4 Training algorithm

In this study, optimization of the discriminative objectives is performed by stochastic gradient descent. For the proposed CCC objective, we compute the gradient ryO = (@O◁@yif↪t )i↪f↪t. Using the quotient rule, the partial derivative for a sequence i, target f and time step t is computed as:

with the partial derivatives

With this, the desired derivative @O◁@yif↪t is obtained as

Having obtained the output gradient as above, the gradient w.r.t. the weights, @O◁@w is determined by backpropagation through time as usual.

Discriminative training is implemented on top of the open source, GPU-enabled neural network training software CURRENNT [Weninger et al., 2015], which supports deep feedforward and recurrent neural networks. The additional code which minimizes the proposed objectives will be made available upon publication of this manuscript.

During the forward pass, we compute and store the target means E¶yif ⇤ ♦, as well as Sif and Qif in ♣F♣ ⇥ ♣B♣ matrices. The summations are computed using an outer loop over time steps, and using matrix addition for all i and f in parallel. For sequence lengths which are large in comparison to the number of target features and the batch size, this might still get inefficient. However, in our experiments, the speed of network training using the ⌃CCC, CCC, or the standard SSE cost function was in the same ballpark. The backward pass is similar to the calculation of the SSE backward pass. The output gradient can be computed for all i, f, and t in parallel.

In case of optimizing CCC rather than ⌃CCC, sequence boundaries need not be taken into account (i = 1). Consequently, the computation of the quantities S1 f and Q1f for each f is much simpler and similar to the SSE calculation.

5 Experiments and Results

We present in this section the results of time-continuous dimensional emotion (arousal and valence) prediction tasks on two different corpora from different domains (speech and music). The objective of those experiments is to empirically demonstrate the benefits of using CCC as cost function for network training, in comparison to the traditional SSE, for emotion recognition from speech and music.

5.1 Emotions frommusic: MediaEval

Experiments on emotion recognition from music are done on the 'Emotion in Music Database' which was used in the MediaEval 2013 evaluation campaign [Soleymani et al., 2013]. The task is to recognize time-varying emotion contours in the arousal and valence dimensions at a rate of 1Hz from music signals. The data set includes excerpts of 45 seconds randomly selected (uniform distribution) from 744 songs taken from the online library Free Music Archive 1 , and split between a development set (619 songs) and an evaluation set (125 songs). Ratings of emotion were performed on a crowdsourcing platform (MTurk) by a pool of 100 selected workers (57 male, 43 female, mean age is 32 years and standard deviation of 10 years) from different countries (72% from the USA, 18% from India and 10% from the rest of the world).

Both features extraction and machine learning steps are based on the setup reported in [Weninger et al., 2013]. The 6 373-dimensional ComParE set of generic affective features, and the Long Short-Term Memory (LSTM) [Gers et al., 2000] architecture for deep recurrent neural networks (DRNNs) are used. LSTM networks have two hidden layers with 192 or 256 hidden units. The ComParE set consists of supra-segmental acoustic features, i.e., summarization of frame-level features over segments of constant length. In the present study, supra-segmental features are extracted from non-overlapping segments of 1 second length, in accordance with the time resolution of the emotion contours.

The training parameters are preserved from [Weninger et al., 2013]. Input noise with σ = 0▷6 is added to help generalization, and an early stopping strategy is used to alleviate overfitting. Stochastic gradient descent with a batch size of 25 sequences is used in all experiments. The learning rate ⌘ is determined in a preliminary cross-validation experiment for each objective function. Note that the objective functions are on different scales and hence the optimal step size varies between various objectives.

Both the sum of CCC and total CCC objectives are investigated. As baseline, standard SSE training is used. In accordance with the MediaEval challenge, the evaluation metrics comprise the overall Pearson's correlation coefficient (CC) 2 as well as the average Kendall's rank correlation coefficient per sequence (E¶⌧♦), which is related to our ⌃CCC objective function but not differentiable. Furthermore, we report the average CCC (E¶(CCC)♦) per sequence, which directly corresponds to the ⌃CCC objective.

1 http://www.freemusicarchive.org

2 Note that MediaEval uses the determination coefficient, which is the square of the CC, but we report CC as it is in the same order of magnitude as the CCC, which is the focus of our evaluation.

5.2 Emotions from speech: RECOLA

Time-continuous prediction of emotional dimensions (arousal and valence) has also been investigated on speech data by using the RECOLA database [Ringeval et al., 2013] 3 ; the full dataset was used for the purpose of this study, which corresponds to speech recordings from 46 French-speaking participants (27 female, 19 male, mean age is 22 years and standard deviation of 3 years) with 5minutes for each. Ratings of emotion were obtained by 6 French-speaking research assistants (3 female, 3 female) using the ANNEMO annotation toolkit. Traces were then interpolated at a 40ms frame rate and averaged as a gold-standard, using the same procedure as described in [Ringeval et al., 2015a]. The dataset was split equally in three partitions - train (16 subjects), development (15 subjects) and test (15 subjects) - by stratifying (i.e., balancing) the gender, the age and the nationality of the speakers. Details on those partitions are provided in Table 1. The same procedure as the one used in the latest edition of the Audio-Visual Emotion Recognition Challenge (AV+EC 2015) [Ringeval et al., 2015b] has been used to extract acoustic features from the speech recordings: the extended Geneva minimalistic acoustic feature set (eGeMAPS - 102 features) [Eyben et al., 2015] has been applied at a rate of 40 ms (to match the sampling frequency of the gold-standard) using overlapping windows of 3 seconds length.

For the prediction task, we used LSTM-DRNNs with three hidden layers with 128 units each. Input noise with σ = 0▷1 is added and early stopping is also used to prevent overfitting. The networks were trained with stochastic gradient descent on a batch size of 5 sequences with a fixed momentum of 0.9, at different values of learning rate ⌘ = ¶10-2 ↪ 10-3 ↪ ▷ ▷ ▷ ↪ 10-7 ♦. An optimal learning rate ⌘ was chosen based on the CCC on the development set for each emotional dimension and objective function. The CCC metric was computed on the gold-standard and prediction values concatenated over all recordings, in accordance with the AV+EC challenge. In addition, we also report the average CCC (E¶CCC♦) per sequence in analogy to the experiments on music.

For all the networks (regardless of the training objective), a chain of post-processing was applied to the predictions obtained on the development set: (i) median filtering (with size of window ranging from 0.4 second to 20 seconds) [Ringeval et al., 2015b], (ii) centring (by computing the bias between gold-standard and prediction) [K¨achele et al., 2015], (iii) scaling (using the ratio of standard-deviation of gold standard and prediction as scaling factor) and (iv) time-shifting (by shifting the prediction forward in time with values ranging from 0.04 second to 10 seconds), to compensate for delays in the ratings [Mariooryad and Busso, 2015]. Any of these post-processing steps was kept when an improvement was observed on the CCC of the development partition, and applied then with the same configuration on the test partition.

3 http://diuf.unifr.ch/diva/recola/

5.3 Results

Table 2 shows the results on the MediaEval 2013 test set (music). We can observe that the evaluation metrics exactly reflect the choice of the objective function: SSE training works best for minimizing the MLE, while CCC based training yields the best CCC on the test set.

The official Challenge evaluation metric, E¶⌧♦, is significantly (according to a z-test, ↵ = ▷05) improved by using the ⌃CCC objective function (.221 ! .251) for arousal but only slightly (.189 ! .199) for valence. Generally, it is observed that the larger network with 256 hidden units performs worse on the test set, which can be attributed to the relatively small data set which causes over-fitting. The discrepancy between E¶CCC♦ and CCC on this data set is astonishing; we found that for some test sequences, the variance in the annotated emotion contours is very low, which makes it hard to achieve good CC on these. One may further notice that the best performance measured as CC on valence is obtained with the CCC objective. The improvement over the SSE objective is significant (.637 ! .653). Regarding the optimization of the network, results show that each objective function requires a specific learning rate to perform best.

Next, in Table 3 we report the metrics on the RECOLA database (speech). Here, we observe a significant improvement in the CC, CCC and E¶CCC♦ metrics by using the ⌃CCC objective function, particularly on the test set, where SSE training does not deliver useful results in the arousal dimension: CCC = .097 with SSE training and .350 with ⌃CCC training. Since this difference is less pronounced on the development set, for which the network is tuned, we have some evidence that the ⌃CCC objective function leads to better generalization. In fact, when training using the SSE criterion, we observed a bias of the network towards predicting the mean annotation on the training set, which leads to good RMSE but low correlation; conversely, the RMSE is significantly increased by using the CCC-based criteria. This result can also be observed on the CC evaluation metric, where a significant improvement over the SSE objective function is obtained when using PCCC for both arousal and valence. One may also note that the same results hold for the optimization of the network: each objective function requires a specific learning rate in order to provide the best performance.

6 Conclusions

We have demonstrated that the SSE objective in neural network regression can be effectively replaced by a criterion derived from the CCC, which has a significant impact on performance in continuous dimensional emotion recognition of arousal and valence from speech and music. Indeed, the CCC is an elegant solution to the issue of scaling and shifting timecontinuous predictions, as it is sensitive to both of these variations and thus alleviates the problem of local minima in neural network training. Still, the observed increase in MSErelated criteria indicates that further investigations need to be performed in order to find an appropriate trade-off between MSE- and CC-like criteria.

Furthermore, note that the proposed approach based on CCC optimization can be applied to any sequence regression task where the correlation between the regression outputs and the ground truth should be maximized. There are no assumptions made on the underlying problem, other than that there be one or more continuous-valued target labels and that the regression model can be effectively trained by a first-order method such as stochastic gradient descent. Thus, we will verify its efficiency on other recognition tasks involving timecontinuous measurements.

Finally, there are many more areas in emotion recognition and related fields where usage of the CCC can be explored, including the definition of the 'gold-standard' by computing the agreement of the raters, estimating an annotation delay, or the selection of features by using CCC instead of CC.

Acknowledgments

The research leading to these results has received funding from the European Commission's Seventh Framework Programme through the ERC Starting Grant No. 338164 (iHEARu), and the European Union's Horizon 2020 Programme through the Innovative Action No. 644632 (MixedEmotions), No. 645094 (SEWA) and the Research Innovative Action No. 645378 (ARIA-VALUSPA).