Structured Probabilistic End-to-End Learning from Crowds
Structured Probabilistic End-to-End Learning from Crowds
Zhijun Chen, Huimin Wang, Hailong Sun, Pengpeng Chen, Tao Han, Xudong Liu, Jie Yang
Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
Main track. Pages 1512-1518.
https://doi.org/10.24963/ijcai.2020/210
End-to-end learning from crowds has recently been introduced as an EM-free approach to training deep neural networks directly from noisy crowdsourced annotations. It models the relationship between true labels and annotations with a specific type of neural layer, termed as the crowd layer, which can be trained using pure backpropagation. Parameters of the crowd layer, however, can hardly be interpreted as annotator reliability, as compared with the more principled probabilistic approach. The lack of probabilistic interpretation further prevents extensions of the approach to account for important factors of annotation processes, e.g., instance difficulty. This paper presents SpeeLFC, a structured probabilistic model that incorporates the constraints of probability axioms for parameters of the crowd layer, which allows to explicitly model annotator reliability while benefiting from the end-to-end training of neural networks. Moreover, we propose SpeeLFC-D, which further takes into account instance difficulty. Extensive validation on real-world datasets shows that our methods improve the state-of-the-art.
Keywords:
Humans and AI: Human Computation and Crowdsourcing
Humans and AI: Human-AI Collaboration
Machine Learning: Probabilistic Machine Learning