Adaptive Deep Learning from Crowds

1 Introduction

Recent years have witnessed the remarkable success of deep neural network training on large-scale datasets [LeCun et al., 2015; Vaswani et al., 2017]. Crowdsourcing is a useful way to collect labeled data from human workers for these data-hungry deep models [Han et al., 2025]. Task owner pays rewards to workers for their annotations in crowdsourcing platforms, and every annotation carries a cost, e.g., 100 annotations of a text classification task cost 2 Yuan in a Chinese platform maintained by Netease Fuxi 1 . However, the crowd-supervised dataset inevitably contains incorrect, noisy, and redundant labels, which causes the task budget to increase [Nguyen et al., 2024; Chen et al., 2022]. The simplest solution is to conduct an admission test to filter out insufficient workers. Since this in-advance test introduces additional test costs and wastes workers' time, many efforts have been made to reduce crowdsourcing costs according to knowledge from data [Wang and Zhou, 2016; Fang et al., 2018; Yang et al., 2018; Miao et al., 2023].

∗Corresponding author

1 https://zb.163.com/mark/task

A popular model training approach in crowdsourcing is the co-training of target models and label correction mechanisms in an end-to-end fashion. The basic paradigm is connecting the crowd layer, i.e., transition matrices of workers, behind the original classifier [Rodrigues and Pereira, 2018]. This end-to-end training enables any improvement of deep learning to be applied in crowdsourcing and shows flexibility and practicality. Existing works in cost saving focus on modeling cost complexity under quality requirements or dynamically determining the price for each worker. However, these theoretical results under label aggregation, such as majority voting, are hard to apply in end-to-end deep learning from crowds, where the hypothesis space and sample complexity are hard to analyze under the Probably Approximately Correct framework [Haussler and Warmuth, 1995].

To this end, we aim to save costs with end-to-end deep learning. Inspired by computerized adaptive testing (CAT) [Ghosh and Lan, 2021], the main idea is to estimate the ability of human workers using as few unlabeled instances as possible. CAT is widely used in education assessments such as GMAT [Rudner, 2010] and Duolingo [Brenzel and Settles, 2017], where the question is adaptively selected based on students' responses to estimate their ability efficiently. The selection criterion is usually informativeness, e.g., Fisher information, where the maximum value means the difficulty of the exercise equals the ability of the student, and the correct probability is 50%. However, in crowdsourcing, the labels of instances are unknown initially; therefore, the informativeness cannot be measured directly.

To tackle the above challenge, we propose an adaptive learning from crowds method, called AdaCrowd, to efficiently estimate worker parameters and train the target model using as few annotations as possible. Similar to CAT, the core goal is to choose the most informative instance from the candidate set for workers to label. When the worker labels the selected instance, the model parameter updates, and the new instance is selected for the worker until convergence. Note that in our method, there is no need to label some instances in advance to test the worker's ability, although this test is adopted on many platforms. From this perspective, AdaCrowd can be considered an "unsupervised test".

Specifically, we first introduce an annotation probabilistic model to simulate the annotation procedure. Then, the informativeness of instances is assessed based on the uncertainty in the prediction model's outputs. This approach enables a more strategic allocation of workers to instances, ensuring that their expertise is leveraged most effectively. The adaptive learning process in AdaCrowd is fine-tuned by maximizing data likelihood, making it compatible with existing crowdsourcing methodologies.

Through empirical evaluation and visualization on realworld datasets, including LabelMe and CIFAR-10H, we showcase the effectiveness of the AdaCrowd. The results affirm that AdaCrowd can reduce the number of necessary training annotations without decreasing model performance.

The contributions of this research are summarized as follows:

• We propose AdaCrowd, a cost-efficient solution for crowdsourcing. To the best of our knowledge, AdaCrowd is the first research work on cost-saving in end-to-end deep learning from crowds.
• To adaptively choose an instance for each worker to annotate in every epoch, we incorporate the idea similar to computerized adaptive testing. We believe our work can help improve related CAT methods.
• The proposed adaptive crowdsourcing coincides with the out-of-distribution via Evidence Deep Learning. Our task is another application of the Theory of Evidence.

2 RelatedWorks

Learning from Crowds. A fundamental model in crowdsourcing is the Dawid-Skene (D&S) model [Dawid and Skene, 1979]. The D&S model assumes that each worker possesses a confusion matrix, which outlines the probabilities of their annotations matching the true labels. Recently, the end-to-end models that jointly learn the deep neural network and worker parameters directly arose as an EM-free approach. The first work in end-to-end models is CrowdLayer [Rodrigues and Pereira, 2018], which applies the learnable crowd layer after the classifier for confusion modeling. After that, TraceReg [Tanno et al., 2019] introduces a regularization term in mapping the classifier output onto the workerspecified output. Besides, CoNAL [Chu et al., 2021] goes further by distinguishing a common confusion from the individual confusion of each worker. Coupled Confusion Correction [Zhang et al., 2024] simultaneously trains two models to correct the transition matrices learned from each other.

Active learning has also been introduced in crowdsourcing with different models such as logistic regression [Yan et al., 2011], Bayesian networks [Zhao et al., 2014], and active learning with SVM [Zhong et al., 2015]. Our work is nascent and differs from these works since, in end-to-end learning, existing methods are hard to scale in deep networks with confusion layers. The proposed approach, adaptively assigning the best instance to workers, can be seen as a prepositive work that could serve as a plug-in to enhance existing works.

Crowdsourcing Cost. The pioneering work in crowdsourcing cost-saving is the adaptive task-assigning with gold standard tasks for classification [Ho et al., 2013]. This work builds the relationship between the target error rate and the times of querying humans. After this, SmartCrowd formats the problem of worker-to-task assignment in knowledgeintensive crowdsourcing, e.g., Wiki writing, as an optimization problem [Basu Roy et al., 2015]. Probably approximately correct (PAC) is also used to study the cost-saving effect of crowdsourcing learning theoretically, and an upper bound for the minimally sufficient number of crowd labels can be given [Wang and Zhou, 2016]. Then, the cost complexity, also based on PAC, is proposed to model the trade-off between costs and quality [Fang et al., 2018].

3 Background

3.1 End-to-end Learning fromCrowds

Notation. Suppose that there are R workers labeling N instances as belonging to K possible classes. xi ∈ X refers to the i-th instance, and yri ∈ Y refers to the one-hot label from the r-th worker on the i-th instance, where X is the instance space depending on task and Y is the label space, i.e., a K-1 probability simplex is considered:

We denote the instance set as X = {xi}Ni=1 , the annotation set as Y = {yir}, where yir = [yri1, yri2, . . . , yriK], and the unknown instance truth set as Z = {zi}Ni=1 . Let us represent the classifier as f : X → Y, and the workers' transition matrices as {Ar}Rr=1 , where Ar satisfies that its columns are conditional probability distributions.

End-to-End Training. In the general end-to-end training paradigm, classifier f and worker parameters {Ar}Rr=1 are connected to a deeper network. Note that the classifier is sequentially combined with a feature extractor and a linear layer. The objective function is expressed as follows:

After proper network initialization, optimal parameters of crowdlayer {Ar}Rr=1 and classifier f can be estimated with stochastic gradient descent. Assuming that the ground truth confusion matrices are {A♮r }Rr=1 , the difference between the learned parameters {Ar}Rr=1 and {A♮r }Rr=1 are bounded [Ibrahim et al., 2023].

3.2 Problem Setting

We focus on cost-saving by reducing the number of annotations in this research work. The intuitive way is improving the task assignment mechanism, similar to the CAT [Zhuang et al., 2022]. The correspondence between CAT and crowdsourcing task assignment is shown in Table 1. The core concept is instance selection, which makes a decision on which instance and possible annotation can best help learn the target model. Here, we give the setting of instance selection.

Instance Selection. Given instance pool X and worker pool Q, our task is to design a strategy π to select an instance, i.e., x(t) r ∼ π(t) (X, r), in each step t according to current model parameters f (t) and {A(t) r }Rr=1 . The selection strategy should balance two goals: importance of each instance for estimating the worker parameter and coverage of all selected instances for training the target model.

With the instance selection, adaptive crowdsourcing is described as follows.

Adaptive Crowdsourcing. At step t (1 ≤ t ≤ T), there is one instance x(t) r sampled from distribution π(t) (X, r) for worker r. Then, the worker labels this instance, i.e., gives a one-hot observation. Both the classifier and worker parameters in step (t + 1) are updated with this new labeled sample

(x(t) r , y(t) r ). The number of steps T may vary with workers. For simplicity and comparability, we set T to be numberfixed or proportion-fixed as a hyperparameter. After learning from selected annotations sequentially after T steps, we get the target model f (T) . The performance of the target model is measured by computing accuracy in an unseen test set.

4 Method

4.1 Probabilistic Model

To measure the uncertainty of an instance on the current model state, we model the annotation process as a probabilistic model, as shown in Figure 2. Instead of treating f(xi) as the softmax output, we take the output f(xi) as the parameter of the Dirichlet distribution αi about the multinomial probability of this instance. Therefore, Ar is not the confusion matrix with the sum of each row equaling 1, but a transformation matrix without simplex constraint. After labeling by worker r, the parameter of Dirichlet distribution changes to αri = Arαi. Then, the annotation can be seen as a sample from posterior Dirichlet, i.e., pri ∼ Dir(αri). Therefore, denoting the annotation set as Y , the likelihood of data is:

where prik = [pri]k.

The likelihood is the function of parameters f, {Ar}Rr=1 , therefore, the maximum likelihood estimation (MLE) of this probability model is computed by minimizing the negative of log-likelihood, i.e.,

4.2 Overview of AdaCrowd

The model architecture is illustrated in Figure 3. The main components are the backbone network, the evidential learning module, and the instance selection module. The backbone network is a task-specific pre-trained model, and the setting of the backbone is shown in the experiments. We describe the overview of the modules as follows.

• Evidential Learning Module: We introduce evidential learning to measure the uncertainty. By modifying the traditional CrowdLayer, evidential learning can be applied to existing end-to-end crowdsourcing models. The modified network is optimized directly by log-likelihood without sampling.
• Instance Selection Module: We design a mechanism with uncertainty and evidence for selecting the most suitable ones for workers. The importance and overall coverage are both considered for making the best of both worker parameter estimation and target model training.

4.3 Evidential Learning

Evidential deep learning is currently a popular technique in out-of-distribution detection, aiming to find samples that never appear in the training set [Sensoy et al., 2018]. Here we show how to compute belief and uncertainty for an instance x ∈ X.

With backbone network f, the last softmax layer is replaced with a linear layer, and the output shape is the number of classes K. Therefore, the output of the backbone is:

For each possible class k, the belief mass is denoted as bk, and the overall uncertainty mass is denoted as u. These mass values are non-negative, and their summary is 1, i.e., u + ∑Kk=1 bk = 1. To construct these mass values, the e is normalized to (u, b), i.e.,

where ek is obtained from Eq. (5) and referred to as evidence because it quantifies the level of support gathered from data for classifying a sample into a specific class.

Therefore, the above value can be used to define a Dirichlet distribution with parameter α, where αk = ek + 1. The density of the Dirichlet distribution is:

where p is a probability satisfying simplex constraint, αk = ek +1 = [f(x)]k +1, and the length of Dirichlet distribution is S in Eq. (6).

In the crowdsourcing setting, we assume the instance is xi, and the worker's parameter is Ar. After inference of the backbone, the CrowdLayer is used to integrate the worker's characteristics into the instance's final annotation, i.e.,

where 1 is the fulling one array.

According to Eq. (4), the loss function is as follows. For simplicity, we denote L(xri, yri; f , Ar) as L(r, i).

whereDir(p;α) = 1 B(αri) ∏Kk=1 pαrik-1 rik , and Sk is the open set of K - 1 simplex.

Similar to the study in [Sensoy et al., 2018], the loss function is derived as:

where ψ(z) ≃ ln z - 1 2z is the digamma function.

To reduce the covariance evidence and probability, the Kullback-Leibler (KL) divergence [Kullback and Leibler, 1951] is used to regularize the learning process, i.e.,

The final loss function is:

where λ is a trade-off parameter.

4.4 Instance Selection

After training, our end-to-end model can infer the belief and uncertainty mass of possible annotations of instances before worker labeling. In short, the importance module finds instances with maximum uncertainty. However, only considering uncertainty will decrease the diversity of instances. The coverage module leverages belief to cover more instances in the overall workers' tasks.

Importance Module. Denote the instance candidates set as X = {xi}Ni=1 . For worker r, in each step t, all instances are inferred to generate αri. The uncertainty mass is:

where K is the number of classes.

The top-L most uncertain instances are selected as candidates for the coverage module, i.e., X(t) r = Top(X, uri,L).

With the mild assumptions that (1) annotations are i.i.d. variables, (2) ground-truth f ♮ exists and is bounded with the hypothesis set, (3) the near-class specialist exists, and (4) the near-anchor point exists, we have the following theorem.

Theorem 1. [Ibrahim and Fu, 2021] The optimal solution of deep learning from crowds exists, and the distance between this solution f and ground-truth f ♮ is bounded.

However, adaptive training may violate the fourth assumption. For class k, a near-anchor point matching ground-truth predictor f ♮ may not exist. Therefore, we propose the coverage module to alleviate it.

Coverage Module. After obtaining candidates X(t) r , we exploit the global belief information to extend the coverage. The global belief for xi is computed by αi = f (t) (xi) instead of αri. Then, the belief for class k is:

This coverage module bridges workers' selected instances. From step 1 to t-1, all labeled instances is saved as X(1:t-1) with size N ′ , and after selecting the new instance x(t) r , the union is {x(t) r } ∪X(1:t-1) . The accumulation of belief mass is b′ = ∑N ′ i=1 bi.

To improve the class-level coverage, the instance is selected to minimize the variance of b(t) , i.e.,

The pseudo-code of the overall approach is shown in Algorithm 1. It is worth mentioning that after each step, all collected annotations are trained in E epochs to avoid underfitting. The collection procedure is asynchronous and parallel in the real-world scenario. In our experiments, this sample selection is implemented as training on the whole dataset with a mask, and the mask is updated after data collection.

Different from the existing evidential learning and active learning methods, some important features of the proposed method are discussed in the following perspectives.

Algorithm 1 Pseudo-code of AdaCrowd

Input: Instance pool X.

Output: Target classifiers f (T) , and the workers' transition matrices {Ar}Rr=1

• 1: Initialize classifiers f (0) , the workers' transition matrices {Ar}Ri=1 with identity weights.
• 2: for t = 1, ..., T do
• 3: for parallel worker r do
• 4: Inference all instances with the evidential model.
• 5: Compute the uncertainty by Eq. (14).
• 6: Compute the accumulated belief by Eq. (15).
• 7: Select instance by Eq. (16).
• 8: Obtain the annotation yri.
• 9: end for
• 10: for E epochs do
• 11: Update parameters of classifier and worker matrices by Eq. (13).
• 12: end for
• 13: end for
• By utilizing the implicit uncertainty in data between multiple workers and instances, AdaCrowd circumvents golden labels for explicitly testing workers, instead estimating worker characteristics in the labeling process.
• Compared with coreset mining, such as learning with redundant and noisy data, AdaCrowd is deployed throughout the whole crowdsourcing method rather than only on clean data after label collection.
• Compared with evidential deep learning, the goal of AdaCrowd is to measure the information of annotation instead of out-of-distribution detection.

4.5 Theoretical Analysis

The convergence analysis is provided here. The distance between parameter f (t) ,A(t) r in step t and truth f ♮ ,A♮r is bounded. In crowdsourcing learning, the empirical risk minimization (ERM) is directly about Arf, therefore the bound of E[∥f (t)A(t) r -f ♮A♮r ∥] is trivial as common stochastic gradient descent (SGD) convergence analysis. The key challenge is analyze E[∥f (t) -f ♮ ∥], E[∥A(t) r -A♮r ∥]. Here we give the convergence analysis of E[∥A(t) r -A♮r ∥]. The adaptive learning is assumed to select a representative subset, known as the coreset of the whole dataset [Mirzasoleiman et al., 2020]. With this coreset assumption, the theorem is as follows.

Theorem 2 (Expected Estimation Error Bound). Assume that the loss function L, i.e., cross-entropy in classification, is µ-strongly convex, and D is the selected annotation subset with size n that approximates the gradient of full annotations by the error at most ϵ. Then with learning rate η, it holds:

where C is an upper-bound on the norm of the gradient of the loss function, H is a constant value determined by the initial state, and d0 = E[∥A(0) r -A♮r ∥2 ] is the initial distance to the optimumA♮r . Moreover, if learning rate is η = 1µ , it gives:

The proof is given in the supplementary material.

5 Experiments

In this section, we conduct experiments to answer the following research questions:

• RQ1: Can AdaCrowd perform better with existing crowdsourcing models than random selection with fewer annotations?
• RQ2: How does AdaCrowd perform in choosing important data while keeping instance coverage compared to baseline methods?
• RQ3: How does AdaCrowd improve accuracy in fixed training steps, i.e., saving crowdsourcing cost, with the adaptive instance selection reasonably?

5.1 Dataset Description and Analysis

The experiments are conducted on crowdsourcing datasets of image classification: LabelMe and CIFAR-10H, which can be found: LabelMe 2 , CIFAR-10H annotation 3 , image 4 .

LabelMe [Russell et al., 2008; Rodrigues and Pereira, 2018] is an open-source dataset collected from Amazon Mechanical Turk. There are 59 workers, 1,000 instances, and 8 classes: highway, inside city, building, street, forest, coast, mountain, and open country.

CIFAR-10H [Battleday et al., 2020] is an subset of wellknown CIFAR-10 dataset. There are 2,571 workers, 10,000 instances, and 10 different classes in the raw version of CIFAR-10H: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. Every worker has labeled 200 images. To keep things practical, our experiments select the top 100 lowest-accuracy workers. After that, there are 100 workers and 8,621 instances in the CIFAR-10H.

The statistics for the datasets are shown in Table 2. The correct rate of workers is counted as shown in Figure 4. According to Table 2 and Figure 4, we find that the datasets have the following properties:

• Accuracy: The average accuracy of workers in CIFAR10H 95% is much higher than LabelMe 69.20%. In CIFAR-10H, only the top 100 workers with the lowest

2 http://labelme.csail.mit.edu/

3 https://github.com/jcpeterson/cifar-10h

4 https://www.cs.toronto.edu/∼kriz/cifar.html

accuracy are chosen in experiments. In Figure 4, the top-100 workers are the blue part. After that, the average accuracy is 77.50%.

• Annotate Times: In LabelMe, all instances were annotated no more than three times, while the times are inconsistent. In CIFAR-10H, all instances were annotated about 50 times. Therefore, in experiments, we set the number of steps T in LabelMe varying and in CIFAR10H fixed across workers.

The above analysis shows that crowdsourcing usually assigns the same instance to multiple workers, and the workers give correct annotations in most cases. As a result, our proposed approach is suitable for the cost-saving effort in crowdsourcing.

5.2 Experimental Setup

Model Setting. Our models are implemented with the PyTorch library, and the codes are released on our repository 5 . Following the previous work, the pre-trained VGG-16 network is used as the backbone of the classifier for the LabelMe and the CIFAR-10H dataset. Similarly, we follow the arrangement in the previous work that splits and saves the training set, validation set, and test set. In the LabelMe dataset, the augmented training set contains 10,000 images, the validation set contains 500 images, and the testing set contains 1,188 images. In the CIFAR-10H dataset, the training and validation sets contain 10,000 images from the original CIFAR-10H test set, and 2,000 unseen images from the CIFAR-10 dataset are sampled for evaluation.

Hyperparameter Setting. The trade-off parameter λ is increase with training step: λ(t) = min(1, t/Tw). The learning rate λ is selected from [0.0005, 0.001, 0.005, 0.01]. The weight decay is selected from [0, 0.003, 0.009, 0.01]. The annealing step Tw is selected from [5, 10, 15, 20]. The epoch in each step E is selected from the range [1, 5]. According to the validation set, λ is set to 0.001, the weight decay is set to 0, E is set to 2, and Tw is set to 5.

5 https://github.com/MISE-MUST/AdaCrowd

5.3 Performance Comparison (RQ1)

To evaluate the performance of the proposed AdaCrowd, four well-known crowdsourcing approaches are chosen as baselines: CrowdLayer [Rodrigues and Pereira, 2018], TraceReg [Tanno et al., 2019], CoNAL [Chu et al., 2021], and GeoCrowdNet [Ibrahim et al., 2023]. And the instance selection methods Random, Softmax, MaxGrad, and MaxInf are chosen for comparison. The softmax means selecting an instance with the highest logits across classes but minimal probability across instances. MaxGrad and MaxInf select instances by the expected change of parameters, where MaxGrad leverages the gradient, and MaxInf leverages the influence function. Meanwhile, we ablate the importance and coverage modules to show their improvement. The test accuracy and Area Under Curve (AUC) are used as metrics.

The results in LabelMe and CIFAR-10H datasets are shown in Table 3 and Figure 5. Further, we observe that:

1. The full-data training usually shows the highest performance across all methods. The AdaCrowd method performs better than other adaptive methods. The MaxGrad slightly fails the MaxInf because the influence function performs better in estimating the change of parameters.
2. Softmax underperforms relative to AdaCrowd; the lower performance could stem from overconfidence in out-

put probabilities, which can be misleading when dealing with noisy crowdsourced labels.

1. AdaCrowd outperforms AdaCrowd w/o imp and AdaCrowd w/o cov in most cases, while it performs close to the top for GeoCrowdNet, slightly surpassed by AdaCrowd w/o cov, suggesting the improvements in importance and coverage modules, while impact varies by crowdsourcing method.
2. The variability, indicated by the standard deviation, is generally low, showing that these adaptive training strategies are consistent across crowdsourcing methods.

5.4 Coverage in Instance Selection (RQ2)

To gain a better insight into instance selection, we take a close look at the breadth of instances selected by the algorithm, and the instance coverages of different compared methods are measured. The intuition here is that under constant annotations from constant steps, more instances from more classes should be included for training models. In other words, if many annotations about the same instance are collected, the generalization of the model will degrade.

For each step, the number of instances of each class is first recorded. Then, the statistical value of all classes about coverage can be obtained. We focus on the mean value, the lower quartile, and the upper quartile of the first 50 training steps, which are shown as the curve and the shadow in Figure 6.

As portrayed in Figure 6, we can find that:

1. The curve of the MaxGrad is usually below, which shows that the MaxGrad method tends to select the replicated

instances compared with other methods.

1. The shadow area of the Random is the smallest, which shows that the distribution is very concentrated, and the Random method uniformly samples instances from all classes.
2. Compared with the above methods, the proposed AdaCrowd can select different instances with different emphasis between classes when sampling, which guarantees the diversity of chosen annotations and instances.
3. The ablation method performs similarly to the original method, but the mean value is slightly lower, which shows that the coverage module can help to reduce replication.

5.5 Case Study (RQ3)

The No.74 worker in the CIFAR-10H is chosen as a case, and the result is shown in Figure 7. After following the training process, we have the following observations.

The left frame is the training steps from 50 to 56. We find that these selected instances are difficult to label correctly. Besides, the annotations are closely related to the confusion matrix, as shown in the middle frame.

One of the main goals of AdaCrowd is to estimate the transition matrix of workers with fewer annotations. To measure the performance, the transition matrices are softmaxnormalized. For Ar, the softmax-normalized matrix is: [Softmax(Ar)]ij = exp[Ar]ij ∑Kk=1 exp[Ar]ik . Let us denote the ground truth by A∗r , and the normalized transition matrix by Ar, and the error matrix by Er = Softmax(Ar) - Softmax(A∗r ). Then, the Frobenius norm of the error matrix is considered as the measurement. For this K×K matrix Er, the norm is ||Er||F = √∑Ki=1 ∑Kj=1 [Er]2ij .

The reduction of the Frobenius norm starting from step 50 is recorded in training steps as shown in the right frame. According to the curves in the figure, we can find that the proposed AdaCrowd performs better on this metric compared with other methods, especially in the long-term training steps. Although some compared methods, such as MaxGrad, are better in the first few steps, the convergence value is far worse than that of the proposed method.

6 Conclusion

In this research, we propose AdaCrowd, an adaptive learning method to efficiently utilize crowdsourced datasets by leveraging worker characteristics early in the data collection process. AdaCrowd optimizes the use of workers' abilities by dynamically assessing the informativeness of instances based on prediction uncertainties. This method not only enhances the quality of the training data but also reduces the number of necessary annotations without compromising model performance. Our experiments on datasets such as LabelMe and CIFAR-10H validate the effectiveness of AdaCrowd, demonstrating its potential as a cost-effective solution for training models in the crowdsourcing approach.

Acknowledgements

This work is supported by the Science and Technology Development Fund (FDCT) under Grant number 0029/2023/RIA1.