MiSC: Mixed Strategies Crowdsourcing

MiSC: Mixed Strategies Crowdsourcing

Ching Yun Ko, Rui Lin, Shu Li, Ngai Wong

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence
Main track. Pages 1394-1400. https://doi.org/10.24963/ijcai.2019/193

Popular crowdsourcing techniques mostly focus on evaluating workers' labeling quality before adjusting their weights during label aggregation. Recently, another cohort of models regard crowdsourced annotations as incomplete tensors and recover unfilled labels by tensor completion. However, mixed strategies of the two methodologies have never been comprehensively investigated, leaving them as rather independent approaches. In this work, we propose MiSC ( Mixed Strategies Crowdsourcing), a versatile framework integrating arbitrary conventional crowdsourcing and tensor completion techniques. In particular, we propose a novel iterative Tucker label aggregation algorithm that outperforms state-of-the-art methods in extensive experiments.
Keywords:
Humans and AI: Human Computation and Crowdsourcing
Humans and AI: Human-AI Collaboration