Statistical Quality Control for Human Computation and Crowdsourcing

1 Introduction

Human computation is a method for solving difficult problems by combining humans and computers. By querying human intelligence from computer systems, human computation solves problems that cannot be solved by either humans or computers individually. An example application of human computation is reCAPTCHA, which is designed to digitize a large number of books by involving humans in character recognition [von Ahn et al., 2008]. Another example is VizWiz, a mobile application for assisting blind people [Bigham et al., 2010]. When a user takes a picture and asks a question, this application returns answers provided by humans.

To achieve a high throughput, human computation requires the participation of a large number of people. Crowdsourcing marketplaces such as Amazon Mechanical Turk or CrowdFlower provide an access to human intellect; by using crowdsourcing, one can therefore easily outsource tasks to crowds of people through the Internet. The expansion of crowdsourcing marketplaces has thus played an important role for evolving research on human computation.

One major challenge in human computation is how to handle uncertainty as well as the variety of crowds. Quality control is a particularly critical issue because all people make mistakes and have different levels of reliability. A solution to this issue is to leverage the power of the "wisdom of crowds"; for example, we can aggregate the outputs from multiple workers (i.e., participants in human computation) or ask a worker to check the output of another worker to improve its quality. In this paper, we review several statistical approaches for controlling the quality of outputs from crowds.

2 Quality Control for Simple Tasks

2.1 Multiple-choice Questions

A multiple-choice question is a typical format for querying workers. Reliable answers can be obtained by asking the same question to multiple workers and applying majority voting. In addition to this simple aggregation method, various sophisticated statistical methods have been proposed [Dawid and Skene, 1979; Whitehill et al., 2009; Welinder et al., 2010; Bachrach et al., 2012; Venanzi et al., 2014]. These methods estimate the reliability of each worker and aggregate the results. Specifically, they assume that workers who provide majority answers are reliable workers (i.e., the opinions of the majority are strengthened). Consequently, most such approaches often fail when the majority of workers provide wrong answers. These cases occur when tasks require highly specialized knowledge and the reliability of a large majority of the workers is inadequate.

Targeting Experts with Hyper Questions

In [Li et al., 2017], we proposed a method for targeting reliable workers who are overwhelmed by the large number of unreliable workers. Our key idea is to focus on sets of questions (which we call hyper questions) instead of individual questions. Let us assume that two types of workers exist, experts and non-experts; experts are likely to provide correct answers and non-experts are likely to guess randomly. When we consider multiple questions, non-experts are thus less likely to reach a consensus than experts. This fact enables us to let experts win in taking the majority voting on multiple questions.

Our method, called Hyper-MV, takes the majority voting on hyper questions. A hyper question consists of a subset of the original single questions. We specifically consider a khyper question, which contains k single questions. For example, when we have a set of four questions, {1, 2, 3, 4}, {1, 2, 3}, {1, 2, 4}, {1, 3, 4}, and {2, 3, 4} are 3-hyper questions. Given k, our method first enumerates all possible khyper questions and then transforms answers to single questions into answers to the hyper questions. Each answer to a hyper question is represented by the concatenation of the answers to the single questions included in the hyper question. For example, when a worker answers questions 1, 2, and 3 with "A," "B," and "C," respectively, the answer to the hyper question {1, 2, 3} is "ABC." Hyper-MV then performs the majority voting on each hyper question, which results in an answer to the hyper question. The aggregated results of the hyper questions are next decoded into answers to the single questions. Finally, another round of majority voting is carried out for each single question to obtain the final aggregated answer to each one. It is worth noticing that the first round of majority voting on hyper questions can be replaced with other aggregation methods, such as GLAD [Whitehill et al., 2009] or DARE [Bachrach et al., 2012].

Targeting Experts by Confidence Reports

Another solution for strengthening the answers provided by experts is to ask each worker for his/her level of confidence in his/her responses. This confidence information is useful for finding reliable workers from crowds; however, some workers can be overconfident and report a high level of confidence even though they provide incorrect answers. Additionally, some workers are underconfident and report a low level of confidence even when their answers are correct. To leverage this confidence information for quality control purposes, we proposed a statistical method for handling the characteristics of the confidence report of each worker [Oyama et al., 2013].

Our method is based on two generative models, the answer model and the confidence model. For simplicity, we consider binary questions and we denote the true answer to the i-th question by ti ∈ {0, 1} and the answer of the j-th worker to the i-th question by yij ∈ {0, 1}. Similar to the DawidSkene model [Dawid and Skene, 1979], we assume that each worker has a reliability parameter α(j) = {α(j) 0 , α(j) 1 }, where α(j) 0 and α(j) 1 are the probabilities that the j-th worker answers yij = 1 when the true answer is ti = 0 and ti = 1, respectively. The answer model is formalized as follows:

A worker chooses his/her confidence level (i.e., high or low) depending on his/her answer and the true answer, and the confidence level is denoted by cij ∈ {0, 1} where cij = 1 indicates that the j-th worker reports a high confidence in the i-th question. We introduce a confidence parameter for each worker β(j) = {β(j) 00 , β(j) 01 , β(j) 10 , β(j) 11 }. For example, β(j) 01 is the probability that the j-th worker reports high confidence (i.e., cij = 1) when the true answer is ti = 0 and the answer from the worker is yij = 1. Our confidence model is given as follows:

The probabilities Pr[cij | ti = 0, yij = 0], Pr[cij | ti = 1, yij = 0], and Pr[cij | ti = 1, yij = 1] are given in the same manner. This confidence model enables us to capture worker characteristics in the confidence report and thus strengthen reliable workers in the aggregated answers.

2.2 Other Types of Questions

In addition to multiple-choice questions, several question formats can be used to query workers, and statistical quality control methods have been proposed for them, such as pairwise comparison tasks [Chen et al., 2013], numerical response tasks [Lin et al., 2012], ordinal response tasks [Raykar and Yu, 2011], and sequence labeling tasks [Wu et al., 2012].

In [Matsui et al., 2014], we proposed an aggregation method for ranking tasks, in which workers are asked to arrange given objects in the correct order. We assume that there is the (latent) true order πi for the i-th task and that each worker has a reliability parameter λj. We model the generative process of the worker's answer as follows:

where πij is the answer of the j-th worker for the i-th task, d (·, ·) denotes the distance between the two rank vectors, and Z (λ) is a normalizing constant given as Z (λ) = ∑π′ exp (-λd (π′ ,π)). Specifically, we employ the Euclidean distance (also referred to as the Spearman distance in the ranking model literature). Given the answers from workers, we estimate the true order of each task. We also developed statistical quality control methods for other types of tasks such as hierarchical labeling [Otani et al., 2015] and collecting points of interest [Kajimura et al., 2015].

3 Quality Control for Complex Tasks

3.1 General Tasks

Statistical quality control approaches for simple response tasks aggregate workers' answers. Hence, such aggregation approaches are unsuitable for complex response tasks, such as providing translations and designing logos. An alternative approach is to choose the highest quality output among those given by multiple workers for the same tasks.

Two-stage Workflow with Grading

To estimate the quality of each output, we proposed a twostage workflow [Baba and Kashima, 2013]. In the first stage, each worker (called the author) creates an output for the task. Then, authors' outputs are processed in the second stage, where each output is graded by another set of workers (called reviewers); each reviewer chooses a grade from the given set (e.g., a five-point scale). Although it is inevitably difficult to directly estimate the quality of the output, this two-stage workflow enables us to indirectly estimate the quality from the grades and distinguish high quality outputs from the others.

Because there is no guarantee that all reviewers have sufficient reliability, we introduce statistical models for estimating the characteristics of both reviewers and authors. We assume that an author with higher ability creates higher quality outputs on average; hence, each author has the ability µj. We also assume that a reliable author shows stable performance for different tasks. The reliability of the j-th author is represented by λj. Our generative model for the first stage formalizes the generative process of the quality of an output as follows:

where qij is the (latent) true quality of the output created by the j-th author for the i-th task.

In the review stage, we assume that each reviewer has a bias ηk ∈ R, assuming that a reviewer with a lower (higher) bias tends to give lower (higher) grades to the given output. We also incorporate the reliability of the k-th reviewer, τk. If reviewers have low reliability, their grades are likely to contain noise. In our model, when a reviewer grades an output, the reviewer first estimates the (latent) quality score of the output, sijk ∈ R. The generative model for the quality score is represented as follows:

Then the reviewer returns the final grade label gijk ∈ {1, 2, . . . , n}. Since the final grade label is a discrete value depending on the quality score, we apply Pr [gijk = d | sijk] which is the conditional probability of selecting d given the quality score sijk. For modeling this probability, we employ the graded response model [Samejima, 1969]. In this model, the conditional probability of a graded response is decomposed by using n - 1 binary response models, namely,

where Pr[gijk > 0 | sijk] = 1 and Pr[gijk > n | sijk] = 0. The Rasch model [Rasch, 1960] is applied as a binary response model as follows:

where {bd}d are threshold parameters. For simplicity, we set the thresholds (b1, b2, · · · , bn-1) = (1, 2, · · · , n - 1). Given the grades from the reviewers, we can estimate the true quality of each output as well as the other model parameters.

Two-stage Workflow with Pairwise Comparison

We proposed another method for the two-stage workflow in which a reviewer compares a pair of outputs and votes for one of them [Sunahase et al., 2017]. Pairwise comparison has several advantages over rating single outputs; for example. it can capture small differences in quality between outputs, and reviewers do not need to calibrate their standards over time. Our proposed method is an extension of the HITS algorithm [Kleinberg, 1999], and thus we call it two-stage pairwise HITS. Analogous to the hubs and authorities in the HITS algorithm, we assume that a good reviewer votes for many good outputs and that a good output is voted for by many good reviewers. We modify the HITS algorithm so that it is applicable to pairwise comparison.

Two-stage pairwise HITS aims to estimate the true quality of each output, qi. We assume that each reviewer has a reliability parameter rk, and the difference between the quality of the two outputs is calculated as the difference between the sum of reviewers' reliability voting for them in their pairwise comparison:

where Vi,i′ = {k | i ≻k i′ } is the set of reviewers who prefer the i-th output to the i′ -th output.

Meanwhile, we define the reliability of the k-th reviewer, rk, as the proportion of his/her correct decisions out of all his/her comparisons:

where Vk = {(i, i′ ) | i ≻k j ′ } is the set of pairwise comparisons that the k-th reviewer makes.

In addition, we incorporate the author's ability, cj, into the output quality as follows:

where q∗ i is the quality of the i-th output that is the solution of Eq. (9), ji is the index of the author of the i-th output, and λ is the tradeoff parameter, where 0 < λ < 1. We define the ability of an author as the average of the quality of his/her outputs; that is, the ability ck of the j-th author is given as

where Wj is the set of outputs created by the j-th author. Our two-stage pairwise HITS algorithm iteratively calculates qi, cj, and rk to estimate the true quality of each output.

3.2 Translation Tasks

We also studied a quality control method for language translation tasks in [Ehara et al., 2016]. A typical method of estimating translation ability involves translation tests, in which a translator is asked to translate given sentences and professional reviewers grade the translation results. By contrast, our convenient method uses vocabulary tests to assess translation ability, meaning that we do not need to rely on professional reviewers. In the tests we designed, translators both translate several sentences as well as tell if they know the meaning of each word in the source language sentences. We then use the results of the tests to estimate the translator's ability.

We are given the contents of the test. In other words, we have each sentence in the translation test, s = {ws1,ws2, . . .}, where wsi is a word in the sentence; each sentence s and word w are associated with a feature vector s and w, respectively. We are also given the results of the vocabulary test {ykw}; ykw = 1 if translator k answers that he/she knows the meaning of the word and ykw = 0 otherwise. In addition, some of the translation results can be evaluated by the professional reviewers: zks = 1 if the translation by translator k for sentence s is graded as acceptable and zks = 0 otherwise. We use the information to estimate the translator's ability.

Our method is based on two generative models: the translation ability model and vocabulary ability model. To build the former, we assume that translation quality varies according to the translation ability of the translator as well as the translation difficulty of the source language sentence. We model the translation ability of the translator k as ψk and the translation difficulty of sentence s as νs. By using the Rasch model [Rasch, 1960], the translation ability model thus represents the probability that translator k acceptably translates sentence s as follows:

We further model sentence difficulty νs as the inner product of the sentence feature vector s and a weight parameter τ , that is, νs = τ ⊤s. Incorporating sentence feature vectors allows our model to estimate the probability of a translator providing an acceptable translation for a new sentence.

We next present the vocabulary ability model. We assume that the probability of a translator knowing the meaning of a word is dependent on his/her vocabulary ability and the difficulty of the word. We model the vocabulary ability of translator k as θk and the difficulty of word w as µw. Because we apply the self-report vocabulary test, there may be concerns that overconfident or unreliable translators are likely to answer that they know the meaning of most words. Thus, we introduce a parameter φk to model the unreliability of translator k. We assume that if translator k provides an acceptable translation of sentence s, the probability of translator k answering that he/she knows the meaning of word w ∈ s depends on vocabulary ability θk. Otherwise, we model the probability by using φk. We apply the Rasch model and represent the probabilities, as follows:

We also incorporate the word feature vectors into the vocabulary ability model by modeling the difficulty of word w as µw = π⊤w, where π denotes a weight parameter. We further introduce a prior for ψk ∼ N (ψk | θk, λ-1 ), which enforces the assumption that translation ability correlates with vocabulary ability.

4 Conclusion and Future Directions

In this paper, we reviewed statistical quality control methods for human computation and crowdsourcing. The presented methods implicitly assume that reliable workers agree on the correct answer; however, the correct answer may not be easily determined in practical situations such as when discussions between experts are required. In addition, there may be multiple correct answers owing to the variety of criteria. The presented quality control methods are not designed for such cases.

A promising solution is to ask workers to decide the correct answer(s) for themselves. For example, an approach has been proposed that workers could inspect the answers of others to make it easy for them to agree [Lorenz et al., 2011]. Organizing the answers based on different criteria is also beneficial to support group decision making. For the two-stage workflow (see Section 3.1), we propose a method for visualizing the clusters of the outputs and the suitability of each output [Li et al., 2018]. This method asks workers in the second stage (i.e., reviewers) to judge the similarity of a pair of outputs as well as the preference, and uses these two types of responses for the visualization.

Machine learning is a promising domain for applying human computation approaches. 1 In machine learning projects, human computation is typically used only for collecting training labels; however, it can contribute in various phases. For example, we applied human computation to feature engineering [Takahama et al., 2018]; this method adaptively chooses pairs of positive and negative instances and asks workers to define features that are informative for classifying them. The features generated by humans are naturally interpretable, and we observed that they improve the classification accuracy in some tasks. Additionally, human computation is applicable for creating machine learning models. It can be used for model investigation [Ribeiro et al., 2016], or for modeling itself by aggregating the models created by the crowd of data scientists [Baba et al., 2014]. Human computation approaches can thus contribute towards improving the productivity of machine learning engineers and advancing new fields of machine learning applications.

Acknowledgments

This work was supported by JSPS KAKENHI Grant Number 18K18105.