Efficient Techniques for Crowdsourced Top-k Lists

Efficient Techniques for Crowdsourced Top-k Lists

Luca de Alfaro, Vassilis Polychronopoulos, Neoklis Polyzotis

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
Best Sister Conferences. Pages 4801-4805. https://doi.org/10.24963/ijcai.2017/670

We focus on the problem of obtaining top-k lists of items from larger itemsets, using human workers for doing comparisons among items.An example application is short-listing a large set of college applications using advanced students as workers. We describe novel efficient techniques and explore their tolerance to adversarial behavior and the tradeoffs among different measures of performance (latency, expense and quality of results). We empirically evaluate the proposed techniques against prior art using simulations as well as real crowds in Amazon Mechanical Turk. A randomized variant of the proposed algorithms achieves significant budget saves, especially for very large itemsets and large top-k lists, with negligible risk of lowering the quality of the output.
Keywords:
Artificial Intelligence: human computer interaction
Artificial Intelligence: other