Improving Learning-from-Crowds through Expert Validation

Improving Learning-from-Crowds through Expert Validation

Mengchen Liu, Liu Jiang, Junlin Liu, Xiting Wang, Jun Zhu, Shixia Liu

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
Main track. Pages 2329-2336. https://doi.org/10.24963/ijcai.2017/324

Although several effective learning-from-crowd methods have been developed to infer correct labels from noisy crowdsourced labels, a method for post-processed expert validation is still needed. This paper introduces a semi-supervised learning algorithm that is capable of selecting the most informative instances and maximizing the influence of expert labels. Specifically, we have developed a complete uncertainty assessment to facilitate the selection of the most informative instances. The expert labels are then propagated to similar instances via regularized Bayesian inference. Experiments on both real-world and simulated datasets indicate that given a specific accuracy goal (e.g., 95%) our method reduces expert effort from 39% to 60% compared with the state-of-the-art method.
Keywords:
Machine Learning: Machine Learning
Machine Learning: Semi-Supervised Learning