1 Cultural Dialects of Real and Synthetic Facial Expressions Zsófia Ruttkay HMI, Department of Computer Science University of Twente, The Netherlands zsofi@cs.utwente.nl ABSTRACT In this paper we discuss the aspects of designing facial expressions for Virtual Humans with a specific culture. We review psychological experiments on cross-cultural perception of emotional facial expressions and look at case studies with Virtual Humans. By identifying the culturally critical issues of data collection and interpretation with both real and virtual humans, we aim at providing a methodological reference and inspiration for further research. Author Keywords Facial expressions, embodied agents, cultural difference, adaptation. ACM Classification Keywords H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION Towards culturally adaptive Virtual Humans One of the major motivations for developing Virtual Human (VHs) is their potential as the most natural and easy to use interfaces to computer services, making these accessible for a broad range of users. In our connected and globalized world, the majority of public application would or could attract a culturally diverse user group. For instance, a holiday booking assistant, or a coach to help the user to stop smoking has to deal with very different clients. In everyday life the success of people in such roles very much depends on how well they ‘speak the language, or find the words’ of their client. By speaking the language we refer to all aspects of communication beyond the literal language usage (which, of course, should be familiar to the client). They span from finding out the values and goals of the client in order to frame the task at hand (what to say), the subtleties of language usage (colloquial or not), to accommodating the control of conversation, the adjustment of speech tempo and maybe dialect, and the facial and bodily expressions. On the other hand, people, even if trained in communication, are limited in their degree of adaptation, as constrained by their bodily and facial features, their personality and their cognitive resources (such as the languages they speak). When developing a VH for an application, we could, in principle, accommodate a design not only to match, but even to outperform real people in terms of adaptation capabilities. All that is needed: • assessment of the culture of the user; • ‘instantiation’ a Virtual Human with the bodily, cognitive and communicative capabilities best for the user (and of course, the task) at hand, • to let this VH appear for the user and carry out the tasks further (including possibly learning about the user, and further adaptation to him/her). This scenario is far from feasible at the moment. On the one hand, necessary technological components – image, speech and language processing, handling of huge common sense knowledge bases – are not yet powerful enough. On the other hand, we are lacking the design principles and evaluation methodology to find out what we would go for, had we these technological components at our service. It is evident that to answer such a question, (further) research from social and behavioral psychology, cultural anthropology, and psycholinguistics is needed. At the same time, the perspective of seeing a VH in a cultural context casts new light on some of the existing results, and makes it possible to set a framework to systematically investigate and design for cultural differences. In this paper we concentrate on the issue of emotional facial expressions and culture. The choice of our focus is twofold. On the one hand, one cannot design a virtual human without some emotional facial expressions. Which expressions should be considered, from a semantic point of