Structured Probabilistic End-to-End Learning from Crowds

1 Introduction

The success of deep learning in many vision and language tasks heavily relies on the quantity and quality of labeled training data [Zhang et al., 2016]. Crowdsourced data annotation offers a cost-effective means to acquire a large set of labeled data. However, crowdsourced labels are often of limited quality, which has become a key concern for training deep neural networks and other types of classifiers [Yang et al., 2019].

To train classifiers from noisy crowdsourced labels, existing methods generally treat true labels as unknown variables, and infer the value by modeling a probabilistic relationship between true labels and the annotations (e.g., the confusion matrix) [Zheng et al., 2017]. Key considerations are factors in the annotation process such as annotator reliability and instance (data instance) difficulty, which are represented as parameters of the probabilistic relationship. Depending on how the inference is integrated with the training of classifier, existing methods fall into two broad categories:

∗Equal Contribution

†Corresponding Author

• The two-stage approach [Dawid and Skene, 1979; Whitehill et al., 2009; Welinder et al., 2010; Han et al., 2016] infers the true label for each instance and afterward, it trains classifiers using the inferred labels.
• The joint approach, also referred to as the learningfrom-crowds approach [Raykar et al., 2010; Kajino et al., 2012; Yan et al., 2012; Rodrigues et al., 2013; Bi et al., 2014; Albarqouni et al., 2016; Rodrigues and Pereira, 2018], simultaneously infers the true labels while training the classifier, allowing the two processes to benefit from each other; consequently, this approach generally results in better performance.

Most of the methods in the joint approach, however, are computationally expensive when the classifier is a neural network, as the learning with generally carried out with expectationmaximization (EM) algorithm that infers the true labels and learns parameters of the classifier (especially neural networks) in an iterative manner [Rodrigues and Pereira, 2018].

Recently, Rodrigues and Pereira [2018] introduce an endto-end learning-from-crowds approach where the relationship between true labels and annotations is modeled by a specific type of neural layers, i.e., the crowd layer. Parameters of the crowd layer can be trained together with the parameters of the rest of the neural network using backpropagation, thus largely accelerating the learning process. Despite that, parameters of the crowd layer can hardly be interpreted as annotator reliability, which comes in contrast to the traditional probabilistic approach that always gives us effective and principled solutions. The lack of probabilistic interpretation also makes it challenging to extend the approach to account for other important factors of the annotation process, e.g., instance difficulty, hindering the potential improvement of the learning process.

In this paper, we demonstrate that we can get the best of both worlds by presenting SpeeLFC (Structured Probabilistic end-to-end Learning From Crowds), a structured probabilistic model (i.e., probabilistic graphical models) to end-to-end learning from noisy crowd annotations. SpeeLFC enforces the parameters of the crowd layer to satisfy the constraints of probability axioms, thereby explicitly modeling annotator reliability in the network. To learn the parameters with such constraints, we introduce a reparameterization trick that allows the network to be trained with the standard gradientbased optimization algorithm. Our approach, therefore, benefits both from the expressiveness and reasonableness of probabilistic approaches for representing annotator reliability and from the effectiveness of the end-to-end approach for training neural networks. In addition, our approach can be easily extended to model other influential factors of the annotation process. Moreover, we propose SpeeLFC-D, which extends SpeeLFC to model instance difficulty as an additional set of parameters of the crowd layer.

In summary, we make the following key contributions:

• We propose SpeeLFC, a novel structured probabilistic model that learns interpretable parameters of the crowd layer in end-to-end learning from crowds;
• We propose SpeeLFC-D, an extension of SpeeLFC that further models instance difficulty in end-to-end learning from crowds;
• We conducted evaluation of our proposed approach on two real-world datasets, showing that both SpeeLFC and SpeeLFC-D are able to outperform the state-of-the-art models.

To the best of our knowledge, we are the first to consider the probabilistic interpretation of neural network parameters and the first to model instance difficulty in end-to-end learning from crowds. Experimental results show that the probabilistic constraints alone in SpeeLFC not only make our approach more interpretable and expressive, but also contribute to learning classifier with higher performance.

2 Preliminaries

To help motivate the two proposed models, here we first introduce the far-reaching approach Crowd-Layer [Rodrigues and Pereira, 2018], and then introduce the simple yet profound idea that underlies both of our proposed models.

LFC problem formulation. Let D = {x(i) ,Y(i) }Ii=1 be an i.i.d. dataset, where for each instance x(i) ∈ RD we are given a set of noisy crowdsourced annotations Y(i) = {y(i,j) }j∈J (i) . J (i) represents all the annotators who annotated the ith instance, and y(i,j) represents annotation by the j th annotator (a total of J annotators) on instance x(i) , which is a 1 - of - K encoded K dimensional vector. Each instance x(i) has its corresponding unobserved ground truth t(i) . The goal of LFC is to train an accurate classifier for predicting t given new unknown instances x by using noisy data {x(i) , Y(i) }Ii=1 .

Crowd-Layer

The Crowd-Layer constructs a probabilistic discriminative model and optimizes the cross-entropy loss on crowdsourced annotations, which is shown in Figure 1 (a). Particularly, it adds functions fj(t(i) ) = Π(j) t(i) to the ground truth t(i) after the classifier to obtain a new vector a(i,j) , where Π(j) is an annotator-specific matrix. The elements in the matrix are the parameters of neural network to be trained, whose value range is (-∞,+∞). Then it uses a softmax function to map vector a(i,j) to the new vector y(i,j) which can satisfy a distribution form. Note that there are other variants in Rodrigues et al. [2018] , but the model presented here is the main model when faced with classification problems.

The Basic Idea in Our Models

Different from Crowd-Layer, in our two proposed models, the distribution of crowdsourced annotation y(i,j) is obtained by the linear transformation of ground truth distribution t(i) through the annotator transition matrix Π(j) we construct. That is, assume in a binary classification scenario:

where element π(j) mn represents the probability that the j th annotator will annotate m given the ground truth is n. Note that the transition matrix we construct here is the transpose of the confusion matrix generally used in machine learning. Our annotator transition matrix Π(j) can therefore express a probabilistic relationship between the annotation y(i,j) and ground truth t(i) , and its meaning can thus be interpreted as worker reliability. We illustrate the relationship among the three variables as "triangular association" in Figure 1.

Recall that Crowd-Layer, like ours, uses a specific matrix to model an annotator. However, the linear transformation a(i,j) = Π(j) t(i) without any probabilistic meaning and the nonlinear transformation of softmax in the model break the delicate "triangular association".

3 Method

In this section, we formally introduce two models, SpeeLFC and SpeeLFC-D, which exploit the power of structured probabilistic models for end-to-end learning from crowds.

3.1 SpeeLFC

Model

Here we introduce the probabilistic generative process we construct from instance features to noisy crowdsourced annotations. First, as shown in Figure 2, for each instance x(i) ,

where the distribution of its unobserved ground truth t(i) comes from a conditional categorical distribution Cat(t(i) |fΘNN (x(i) )), which can be a flexible neural network model parametrized by ΘNN .

Then, we directly model each annotator with an annotatorspecific transition matrix Π(j)′ as mentioned in Section 2, which represents her performance pattern on ground truth. Thus, the distribution of annotation y(ij) is determined by:

where π(j)′ mn represents the probability that the j th annotator will annotate m given the ground truth is n, and ind(·) is to take the location index of the value 1 in the one-hot vector (K denotes the number of categories). Here the generation process from ground truth to crowdsourced annotation corresponds to the basic idea introduced in Section 2.

Based on the probabilistic model constructed above, we unfortunately find an optimization dilemma when we try to optimize the log-conditional likelihood log p(Y|X;ΘNN, {Π(j)′ }Jj=1 ) using gradient-based optimization algorithms. Because there are inevitable constraints of the parameters {Π(j)′ }Jj=1 :

and the updates brought to the parameters by the backpropagation cannot make the parameters automatically obey the constraints of Eq. 5 and Eq. 6. Last, we break out of this optimization dilemma just by invoking the reparameterization technique, which also used in the "Mixture Density Network" in Bishop [2006] and variational autoencoder (VAE) [Kingma and Welling, 2013].

The reparameterization. The essence of the reparameterization is quite simple, i.e., we posit that each column vector of each annotator's true-transition matrix π(j)′ :,n is derived from the corresponding column vector of its ancestral-transition matrix π(j) :,n through softmax:

Thus, after reparameterization, the whole network is parameterized by Θ = {ΘNN , Π(1) , . . . , Π(J) } on which the Adam can be smoothly performed. Furthermore, the {Π(j)′ }Jj=1 can perfectly play the roles of annotator transition matrices which strictly satisfy Eq. 5 and Eq. 6.

Objective and Optimization

Based on the model we have constructed, our optimization objective is to maximize the log-conditional likelihood log p(Y|X;Θ) of the observed crowdsourced annotations w.r.t. Θ = {ΘNN,Π(1) , . . . ,Π(J) }:

Based on Eq. 3-8, the U(Θ) provides a unified objective function for optimization in SpeeLFC, which can be done with standard stochastic optimization techniques, such as SGD or Adam [Kingma and Ba, 2014]. In fact, according to Eq. 9, we only need to use Π(i,j) t(i) to obtain the distribution y(i,j) as mentioned in Section 2, and then maximize the log-likelihood of the corresponding observed annotation.

3.2 SpeeLFC-D

Overall, the probabilistic generative process of crowdsourced annotations constructed in SpeeLFC-D is similar to SpeeLFC. As shown in Figure 3, for each instance x(i) , its ground truth t(i) is generated by:

Then, we construct the instance difficulty β(i)′ , representing how easily the instance can be annotated correctly by annotators. For images, it may be related to the resolution of the image. We assume that the instance difficulty is obtained by mapping the instance features through the instance difficulty mapping function fΘNN2 (x(i) ) and a subsequent auxiliary function. That is:

where fΘNN2 (x(i) ) is a neural network function parametrized by ΘNN2, and λ1, λ2 ∈ (0,+∞) are hyper-parameters. Eq. 12 helps us map the original β(i) ∈ (-∞, +∞) to a positive range (λ1, λ1 + λ2).

Last, we model each annotator ability with an annotatorspecific scalar α(j) ∈ (-∞,+∞), and we assume that the higher the annotator ability α(j) and the less instance difficulty (meaning bigger β(i)′ in our model), the greater the probability that the annotator will annotate the instance correctly, and vice versa. Thus, we have the following construction:

where ind(·) is to obtain the location index of the value 1 in the one-hot vector (ind(·) ∈ {1, 2, . . . ,K}). Correspondingly, the probability that the j th annotator with α(j) will annotate the instance x(i) with β(i)′ incorrectly is as follows:

In fact, for each annotator, the above construction will separately models a transition matrix Π(i,j) for her performance pattern on each instance x(i) . Thus, the diagonal elements of the matrix Π(i,j) are obtained from Eq. 13, while the elements on other positions are obtained from Eq. 15.

Based on the model we construct, the objective and optimization in SpeeLFC-D are the same as in SpeeLFC.

4 Experiments

4.1 Settings

Datasets and Compared Methods

We performed experiments on two real-world datasets labeled from Amazon Mechanical Turk (AMT), i.e. the Sentiment Polarity Classification (SPC) dataset [Rodrigues et al., 2013] and the Music Genre Classification (MGC) dataset [Rodrigues et al., 2013]. The SPC dataset contains 5000 sentences (with crowdsourced annotations) from movie reviews extracted from the website RottenTomatoes.com and their sentiment polarity was classified as positive or negative, while the MGC dataset contains 700 samples (with crowdsourced annotations) of songs with 30 seconds in length and were divided into 10 different music genres (e.g., classical, country, disco). The two datasets received a total of 27747 and 2946 annotations from 203 and 44 distinct annotators on AMT, respectively. For both tasks, 5429 and 300 instances are provided as test sets respectively. Figure 4 shows the distributions of the number of annotations provided by each annotator and their accuracies.

SpeeLFC and SpeeLFC-D are compared with: MV + MLP (Majority Voting + Multilayer Perceptron), first the ground truth was estimated by MV, and then a general MLP was trained using the estimated ground truth; GLAD + MLP, similar to MV + MLP, except that the truth inference algorithm is GLAD [Whitehill et al., 2009]; Crowd-Layer [Rodrigues and Pereira, 2018]; Raykar and Raykar(w/prior) [Raykar et al., 2010], the latter is the Raykar model with the prior for the annotator reliability; MA-LR [Rodrigues et al., 2013].

Implementation Details

For the SPC dataset, we set the classifier in both SpeeLFC and SpeeLFC-D as an MLP with one hidden layer (with 1200 units, ReLU activations), using 50% dropout and Adam stochastic optimization [Kingma and Ba, 2014]. The learning rate is 0.0001, batch-size is 64, and epoch number is 200. In addition, the function fΘNN2 (x(i) ) in SpeeLFC-D is also an MLP with one hidden layer (with 128 units, ReLU activations, 50% dropout). And the values of the hyperparameters λ1 and λ2 in SpeeLFC-D are 0.001 and 100, respectively. In SpeeLFC, the values on the diagonal elements of Π(j) (j = 1, . . . , J) were initially set to 1.4, and the other values were set to 1. α(j) (j = 1, . . . , J) in SpeeLFC-D was initially set to 0.028. For the MGC dataset, Rodrigues and Pereira [2013] uses deep learning representation methods to extract 124 features for each instance. We directly set the classifier in both SpeeLFC and SpeeLFC-D as an multiclass Logic Regression just like Rodrigues and Pereira [2013], that is, the MLP without hidden layer. The learning rate is 0.001, and epoch number is 5000. The function fΘNN2 (x(i) ) in SpeeLFC-D is an MLP with one hidden layer (with 32 units, ReLU activations, 50% dropout, the learning rate on θNN2 is 0.0001). In SpeeLFC, the values on the diagonal elements of Π(j) were initially set to 4.7, and the other values are 1. In addition, the other settings are the same as those in the SPC dataset. The settings of the compared methods are the same as ours.

4.2 Results

The accuracies of the classifiers on the two datasets are shown in Table 1 and Table 2, respectively. Among them, the results of Semi-LFC [Atarashi et al., 2018], PC [Kajino et al., 2012], PC-GP [Kajino et al., 2012], Raykar [Raykar et al., 2010], Raykar (w/prior) [Raykar et al., 2010], and MA-LR [Rodrigues et al., 2013], are taken fromAtarashi et al. [2018] and Rodrigues and Pereira [2013]. We observe that the proposed SpeeLFC and SpeeLFC-D achieve better results on both datasets as compared to the baselines and state-of-theart models. On the SPC dataset, because of the inherent task difficulty and the relatively more number of annotations received for each instance (5.55 on average), most methods exhibit similarly high accuracies. In particular, the highlight we can find is that our proposed models even surpass the recently released semi-supervised learning-from-crowds model called Semi-LFC [Atarashi et al., 2018], even though the Semi-LFC uses more information (i.e., unlabeled data) than ours.

In the case of the MGC dataset for the ten-class classification scenario, the performance of these models shows a larger difference. We observe that the proposed SpeeLFC and SpeeLFC-D show more obvious advantages over the compared methods; in particular, both of them significantly outperform Crowd-Layer by a large margin. We conducted an additional experiment to further optimize the network architecture of Crowd-Layer, by adding a hidden layer containing 128 units (50% dropout), and still found the performance (accuracy: 0.5850) to be largely below our proposed models. Those results clearly demonstrate the importance of probabilistically interpretable parameters in end-toend learning from crowds. Additionally, we further observe in Table 2 that SpeeLFC outperforms SpeeLFC-D. Recall that SpeeLFC models each annotator with K × K parameters while SpeeLFC-D only models each annotator with one parameter. Such a comparison result shows that SpeeLFC is more robust for the ten-class classification task with more parameters to be learned.

We now analyze the annotator transition matrices estimated by the proposed SpeeLFC. Similar to Atarashi et al. [2018] and Rodrigues et al. [2017], the six annotators with the largest number of annotations were selected. The results are shown in Figure 5 and Figure 6. Note that the corresponding real annotator transition matrices are calculated based on their annotations and the ground truth. The accurate estimation of annotator performance pattern verifies the validity of the whole network we build and demonstrates that the structured probabilistic models provide us with an explainable and principled solution based on an end-to-end learning manner using pure backpropagation. For comparison, we also show in Figure 5 the corresponding matrices estimated with CrowdLayer: among the six matrices, five contain negative numbers, thus cannot express the probability relationship between the annotation and ground truth.

5 Conclusion

This paper presents two novel models for end-to-end learning from crowds. We present SpeeLFC, a structured probabilistic model that benefits both from the expressiveness and reasonableness of probabilistic approaches for representing annotator reliability and from the effectiveness of the endto-end approach for training neural networks. Moreover, we present SpeeLFC-D, which further models instance difficulty in end-to-end learning from crowds. Evaluation of our proposed methods on real-world datasets shows that both methods improve the state-of-the-art.

Acknowledgments

This work was supported partly by National Key Research and Development Program of China under Grant No.2019YFB1705902, partly by National Natural Science Foundation under Grant No.(61932007, 61972013, 61421003). We thank the authors of Rodrigues et al. [2018] for the sharing of the two datasets that greatly support our experiments.