Proceedings

TDG4Crowd:Test Data Generation for Evaluation of Aggregation Algorithms in Crowdsourcing

Yili Fang, Chaojie Shen, Huamao Gu, Tao Han, Xinyi Ding

Download BibTeX Cite

Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. Main Track. Pages 2984-2992.

https://doi.org/10.24963/ijcai.2023/333

Keywords

  • Machine Learning: ML: Cost-sensitive learning
  • Machine Learning: ML: Autoencoders
  • Humans and AI: HAI: Human computation and crowdsourcing

Abstract

In crowdsourcing, existing efforts mainly use real datasets collected from crowdsourcing as test datasets to evaluate the effectiveness of aggregation algorithms. However, these work ignore the fact that the datasets obtained by crowdsourcing are usually sparse and imbalanced due to limited budget. As a result, applying the same aggregation algorithm on different datasets often show contradicting conclusions. For example, on the RTE dataset, Dawid and Skene model performs significantly better than Majority Voting, while on the LableMe dataset, the experiments give the opposite conclusion. It is challenging to obtain comprehensive and balanced datasets at a low cost. To our best knowledge, little effort have been made to the fair evaluation of aggregation algorithms. To fill in this gap, we propose a novel method named TDG4Crowd that can automatically generate comprehensive and balanced datasets. Using Kullback Leibler divergence and Kolmogorov–Smirnov test, the experiment results show the superior of our method compared with others. Aggregation algorithms also perform more consistently on the synthetic datasets generated using our method.

  1. [Albarqouni et al., 2016] Shadi Albarqouni, Christoph Baur, Felix Achilles, Vasileios Belagiannis, Stefanie Demirci, and Nassir Navab. Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE transactions on medical imaging, pages 1313-1321, 2016.
  2. [Baker et al., 2017] Frank B Baker, Seock-Ho Kim, et al. The basics of item response theory using R. Springer, 2017.
  3. [Dawid and Skene, 1979] Alexander Philip Dawid and Al- lan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), pages 20- 28, 1979.
  4. [Fang et al., 2018] Yili Fang, Hailong Sun, Pengpeng Chen, and Jinpeng Huai. On the cost complexity of crowd- sourcing. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), pages 1531- 1537, 2018. IJCAI
  5. [Galal and El-Sharkawi, 2019] Sh Galal and Mohamed E El- Sharkawi. Trr: Reducing crowdsourcing task redundancy. In Database and Expert Systems Applications -30th Inter- national Conference (DEXA), pages 102-117, 2019.
  6. [Gao et al., 2016] Chao Gao, Yu Lu, and Dengyong Zhou. Exact exponent in optimal rates for crowdsourcing. In Pro- ceedings of the 33nd International Conference on Machine Learning (ICML), pages 603-611, 2016.
  7. [Heinecke and Reyzin, 2019] Shelby Heinecke and Lev Reyzin. Crowdsourced pac learning under classification noise. In Proceedings of the 7th AAAI Conference on Hu- man Computation and Crowdsourcing (HCOMP), pages 41-49, 2019.
  8. [Ho et al., 2013] Chien-Ju Ho, Shahin Jabbari, and Jen- nifer Wortman Vaughan. Adaptive task assignment for crowdsourced classification. In Proceedings of the 30th International Conference on Machine Learning (ICML), pages 534-542, 2013.
  9. [Imamura et al., 2018] Hideaki Imamura, Issei Sato, and Masashi Sugiyama. Analysis of minimax error rate for crowdsourcing and its application to worker clustering model. In Proceedings of the 35th International Con- ference on Machine Learning (ICML), pages 2152-2161, 2018.
  10. [Jing et al., 2018] Jing, Zhang, Victor, S, Sheng, Tao, Li, Xindong, and Wu. Improving crowdsourced label quality using noise correction. IEEE Transactions on Neural Networks Learning Systems, pages 1675-1688, 2018.
  11. [Kawase et al., 2019] Yasushi Kawase, Yuko Kuroki, and Atsushi Miyauchi. Graph mining meets crowdsourcing: Extracting experts for answer aggregation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), pages 1272-1279, 2019. IJCAI
  12. [Li and Yu, 2014] Hongwei Li and Bin Yu. Error rate bounds and iterative weighted majority voting for crowdsourcing. CoRR, abs/1411.4086, 2014.
  13. [Li et al., 2013] Hongwei Li, Bin Yu, and Dengyong Zhou. Error rate bounds in crowdsourcing models. CoRR, abs/1307.2674, 2013.
  14. [Liu et al., 2012] Qiang Liu, Jian Peng, and Alexander Ihler. Variational inference for crowdsourcing. In Proceedings of the 26th Annual Conference on Neural Information Processing Systems 2012 (NIPS), pages 701-709, 2012.
  15. [Long and Hua, 2016] Chengjiang Long and Gang Hua. Multi-class multi-annotator active learning with robust gaussian process for visual recognition. International Conference on Computer Vision (ICCV), 2016.
  16. [Mescheder et al., 2017] Lars M. Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In Proceedings of the 34th Inter- national Conference on Machine Learning (ICML), pages 2391-2400, 2017.
  17. [Pan et al., 2023] Yigong Pan, Ke Tang, and Guangzhong Sun. Theoretical guarantee for crowdsourcing learning with unsure option. Pattern Recognition, page 109316, 2023.
  18. [Rodrigues and Pereira, 2018] Filipe Rodrigues and Fran- cisco C. Pereira. Deep learning from crowds. In AAAI conference on artificial intelligence, pages 1611-1618, 2018.
  19. [Roy et al., 2015] Senjuti Basu Roy, Ioanna Lykourentzou, Saravanan Thirumuruganathan, Sihem Amer-Yahia, and Gautam Das. Task assignment optimization in knowledge- intensive crowdsourcing. The VLDB Journal, pages 467- 491, 2015.
  20. [Russell et al., 2008] Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. Labelme: A database and web-based tool for image annotation. Inter- national journal of computer vision, pages 157-173, 2008.
  21. [Sheng et al., 2008] Victor S. Sheng, Foster J. Provost, and Panagiotis G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD), pages 614-622, 2008.
  22. [Sheng, 2011] Victor S Sheng. Simple multiple noisy label utilization strategies. In Proceedings of the 11th IEEE International Conference on Data Mining (ICDM), pages 635-644, 2011.
  23. [Sinha et al., 2018] Vaibhav B Sinha, Sukrut Rao, and Vi- neeth N Balasubramanian. Fast dawid-skene: A fast vote aggregation scheme for sentiment classification. arXiv preprint arXiv:1803.02781, 2018.
  24. [Snow et al., 2008] Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast -but is it good? evaluating non-expert annotations for natural lan- guage tasks. In Proceedings of the 2008 conference on em- pirical methods in natural language processing (EMNLP), pages 254-263, 2008.
  25. [Wang and Zhou, 2015] Wei Wang and Zhi-Hua Zhou. Crowdsourcing label quality: a theoretical analysis. Sci- ence China Information Sciences, pages 1-12, 2015.
  26. [Wang et al., 2013] Jiannan Wang, Guoliang Li, Tim Kraska, Michael J. Franklin, and Jianhua Feng. Leveraging tran- sitive relations for crowdsourced joins. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pages 229-240, 2013.
  27. [Wauthier and Jordan, 2011] Fabian L. Wauthier and Michael I. Jordan. Bayesian bias mitigation for crowdsourcing. In Proceedings of the 25th Annual Conference on Neural Information Processing Systems 2011 (NIPS), pages 1800-1808, 2011.
  28. [Whitehill et al., 2009] Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier R. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of the 23rd Annual Conference on Neural Information Processing Systems 2009 (NIPS), pages 2035-2043, 2009.
  29. [Yu et al., 2017] Han Yu, Chunyan Miao, Yiqiang Chen, Si- mon Fauvel, Xiaoming Li, and Victor R Lesser. Algorith- mic management for improving collective productivity in crowdsourcing. Scientific reports, page 12541, 2017.
  30. [Zhang et al., 2016a] Jing Zhang, Xindong Wu, and Vic- tor S. Sheng. Learning from crowdsourced labeled data: a survey. Artificial Intelligence Review, pages 543-576, 2016.
  31. [Zhang et al., 2016b] Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael I. Jordan. Spectral methods meet em: A provably optimal algorithm for crowdsourcing. Journal of Machine Learning Research, pages 1-44, 2016.
  32. [Zhang et al., 2019] Hao Zhang, Liangxiao Jiang, and Wen- qiang Xu. Multiple noisy label distribution propagation for crowdsourcing. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), pages 1473-1479, 2019. IJCAI

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards