Combining interviews and scales in the multidimen- sional evaluation of User Experience. A case study in 3D games Cédric Bach IRIT, Université de Toulouse 118, route de Narbonne, 3162 Toulouse, France cedric.bach@irit.fr Nadia Gauducheau ICD-TechCICO, Université de Technologie de Troyes 12, rue Marie Curie, BP 2060 10010 Troyes, France nadia. gauducheau@utt.fr Pascal Salembier ICD-TechCICO, Université de Technologie de Troyes 12, rue Marie Curie, BP 2060 10010 Troyes, France pascal.salembier@utt.fr ABSTRACT Motivation – To study the convergence between differ- ent methods evaluating two dimensions of User Experi- ence: usability and emotions. Research approach – A prospective experimental approach was used, in which 20 participants were asked to express their experience of two 3D applications of board sports. Scales, questionnaires and self- confrontation interviews are compared. Findings – The results show that participants’ responses for the different scales broadly converge. However, regarding the users’ emotions, the convergence between scales responses and statements in the interviews is less obvious. Research limitations/Implications – The sample size (twenty university students) restricts the possible gener- alization of the findings. Originality/Value – The research makes a contribution to study the effective gain of UX assessment approaches combining different methods. Take away message – Methods based on self-reports of experience are critical as questionnaires did not allow to make a clear distinction between UX of two applica- tions. Keywords User Experience, usability, emotions, multidimensional assessment methods INTRODUCTION The emphasis put during the last years on the notion of user experience (UX) can be seen as an attempt to broaden the traditional framework of usability primarily centered on task-related aspects. UX agenda intends to address the need to account for dimensions such as emotional and aesthetic responses to the use of an inter- active device. A large span of methods has been devel- oped to cover the richness and complexity of experienc- ing interactive products. In this context, the question of multi-method approaches and articulation of heterogen- eous data has become a critical research issue in the field. In this paper we report some preliminary findings drawn from a multicriteria evaluation of two different 3D interactive games. UX FRAMEWORK The dimensions of UX Many authors (Hassenzahl, 2003; Mahlke & Thüring, 2007) describe the UX as a multidimensional set of users’ judgments. The judgement of the pragmatic qualities of the system (usability) is part of the ap- proach. The shift consists in including judgements re- lated to motivational, social and affective processes, especially judgements on the non-instrumental qualities (visual aesthetics and symbolic value) of the system. Affective responses arising from these judgements are especially considered (Mahlke & Thüring 2007). The limits of the concept of UX Several limits of UX have been stressed in the UX re- search literature: its lack of theoretical foundations (Kuutti, 2010); the real nature of its added-value com- pared to other existing concepts (Barcenilla & Bastien, 2009); its individual-centric character (Battarbee, 2003). Another limit is the weakness of the articulation be- tween the different dimensions of UX. In particular, several models point out that users’ emotional reactions are relatively independent from their judgement of the instrumental qualities of the system. In fact, usability judgement is only one dimension among the multiple factors influencing user’ emotions (non-instrumental dimensions like aesthetics, users characteristics, etc.) (Hassenzahl, 2003; Mahlke & Thüring, 2007). Never- theless, some empirical evidences show the correlation between affects and subjective usability (e.g., Mahlke & Lindgaard, 2007). The link between users’ emotional reactions and their judgement of the instrumental quali- ties is therefore a critical issue. In this paper we assume that interviews can be a valuable method to help clarify the nature of this link and establish in detail what users think about the situation and their emotional experience, and thus to understand why/when experienced emotions are (or not) linked to the instrumental qualities.