Do Users Appreciate Explanations of Recommendations? An Analysis in the Movie Domain Thi Ngoc Trang Tran, Viet Man Le, Müslüm Atas, Alexander Felfernig Martin Stettinger, Andrei Popescu {ttrang,vietman.le,muesluem.atas,alexander.felfernig,martin.stettinger,andrei.popescu}@ist.tugraz.at Institute of Software Technology, Graz University of Technology Graz, Styria, Austria ABSTRACT In this paper, we provide insights into users’ needs regarding the inclusion of explanations in a movie recommender system. We have developed diferent variants of a movie recommender system prototype corresponding to diferent types of explanations and conducted an online user study to evaluate related explanations. The experimental results show that users do not always appreciate explanations. They want to see explanations when they are not sat- isfed with the recommended items. They expect to see explanations showing how well the recommended item meets their preferences. Moreover, explanation goals are interdependent and afect the over- all satisfaction of users with the recommender system. CCS CONCEPTS · Information systems → Information systems applications; Decision support systems; Recommender systems; · Human- centered computing → User models; User studies. KEYWORDS Movie Recommender Systems, Explanations, Collaborative Filtering- based Explanation, Content-based Explanation, Feature-based Ex- planation, Demographic-based Explanation, Knowledge-based Ex- planation. ACM Reference Format: Thi Ngoc Trang Tran, Viet Man Le, Müslüm Atas, Alexander Felfernig, Martin Stettinger, Andrei Popescu. 2021. Do Users Appreciate Explanations of Recommendations? An Analysis in the Movie Domain. In Fifteenth ACM Conference on Recommender Systems (RecSys ’21), September 27-October 1, 2021, Amsterdam, Netherlands. ACM, New York, NY, USA, 6 pages. https: //doi.org/10.1145/3460231.3478859 1 INTRODUCTION Explaining recommendations has emerged as an essential research task in recommender systems [24]. This task helps to clarify the underlying recommendation mechanism and the reasons for se- lecting a specifc item as a recommendation. Diferent goals have been proposed to evaluate explanations [17, 18, 24]. Some examples Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). RecSys ’21, September 27-October 1, 2021, Amsterdam, Netherlands © 2021 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-8458-2/21/09. https://doi.org/10.1145/3460231.3478859 thereof are transparency which explains how the recommender sys- tem works, trust which increases a user’s confdence in the system, efectiveness which helps a user make good decisions, persuasiveness which convinces a user to try or buy recommended items, efciency which helps a user make a decision faster, and satisfaction which increases the ease of use or enjoyment [18]. In this line of research, the existing literature has witnessed a sig- nifcant amount of contributions that focus on proposing efective methods to generate explanation approaches based on recommenda- tion algorithms [4, 6, 7, 20, 24, 26]. Also, plenty of studies have been conducted to propose ways to visualize explanations and evaluate them based on the aforementioned goals [1, 7, 9, 19, 20]. However, no single study provides an in-depth analysis of users’ needs for including explanations in recommender systems. One emerging question is łDo users always appreciate explanations of recommen- dations?ž. We assume a user’s satisfaction with the recommended item could play a role in this context. A low satisfaction level with recommended items could trigger a higher probability of requesting explanations. Besides, although various explanation approaches have been proposed, it is still unclear łwhich explanation approach is of interest for users?ž. Moreover, an explanation can link to diferent goals, which triggers other questions such as łAre there any correla- tions among explanation goals?ž and łDo explanation goals afect the overall satisfaction of users with the recommender system?ž. Inves- tigating the mentioned correlations is an essential task that helps to predict the overall satisfaction of a user with the recommender system based on his/her evaluation of the provided explanation. In this paper, we have conducted a user study and answered the mentioned questions in the movie domain. The contributions of our paper are three-fold: (1) discussing a scenario where users want to see explanations of recommendations, (2) fnding the favorite explanation approach from users’ point of view, and (3) discovering correlations between explanation goals and the overall satisfaction of a user with the recommender system. The remainder of the paper is organized as follows. In Section 2, we summarize related work regarding explanation generation and explanation evaluation in recommender systems. In Section 3, we introduce four types of explanations and diferent display styles to visualize them in our prototype system. In Section 4, we defne research questions and present essential steps of our user study. The analysis results and discussions regarding the research questions are presented in Section 5. Finally, we conclude the paper and discuss open issues for future work in Section 6. 645