Impact of Consuming Suggested Items on the Assessment of Recommendations in User Studies on Recommender Systems

Impact of Consuming Suggested Items on the Assessment of Recommendations in User Studies on Recommender Systems

Benedikt Loepp, Tim Donkers, Timm Kleemann, Jürgen Ziegler

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence
Best Sister Conferences. Pages 6201-6205. https://doi.org/10.24963/ijcai.2019/863

User studies are increasingly considered important in research on recommender systems. Although participants typically cannot consume any of the recommended items, they are often asked to assess the quality of recommendations and of other aspects related to user experience by means of questionnaires. Not being able to listen to recommended songs or to watch suggested movies, might however limit the validity of the obtained results. Consequently, we have investigated the effect of consuming suggested items. In two user studies conducted in different domains, we showed that consumption may lead to differences in the assessment of recommendations and in questionnaire answers. Apparently, adequately measuring user experience is in some cases not possible without allowing users to consume items. On the other hand, participants sometimes seem to approximate the actual value of recommendations reasonably well depending on domain and provided information.
Keywords:
Multidisciplinary Topics and Applications: Recommender Systems
Machine Learning: Experimental Methodology;Replicability
Humans and AI: Human-Computer Interaction
Humans and AI: Personalization and User Modeling