Gesture Activated Mobile Edutainment (GAME) Intercultural Training of Nonverbal Behavior with Mobile Phones Matthias Rehm Department of Architecture, Design, and Media Technology (CREATE), Aalborg University Niels Jernes Vej 14 9220 Aalborg, Denmark matthias@create.aau.dk Karin Leichtenstern, Jörg Plomer, Christian Wiedemann Human Centered Multimedia Universitätsstr. 6a 86150 Augsburg, Germany leichtenstern@informatik.uni- augsburg.de ABSTRACT An approach to intercultural training of nonverbal behav- ior is presented that draws from research on role-plays with virtual agents and ideas from situated learning. To this end, a mobile serious game is realized where the user ac- quires knowledge about German emblematic gestures and tries them out in role-plays with virtual agents. Gesture performance is evaluated making use of build-in acceleration sensors of smart phones. After an account of the theoretical background covering diverse areas like virtual agents, situ- ated learning and intercultural training, the paper presents the GAME approach along with details on the gesture recog- nition and content authoring. By its experience-based role- plays with virtual characters, GAME brings together ideas from situated learning and intercultural training in an in- tegrated approach and paves the way for new m-learning concepts. Categories and Subject Descriptors H.1.2 [Models and Principles]: User/Machine Systems— human factors, human information processing ; H.5.1 [Infor- mation Interfaces and Presentation]: Multimedia In- formation Systems—artificial, augmented, and virtual reali- ties ; H.5.2 [Information Interfaces and Presentation]: User Interfaces—haptic I/O, input devices and strategies, interaction styles General Terms Human Factors Keywords Virtual Agents, Mobile Edutainment, Gesture Recognition, Intelligent Tutoring System Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MUM ’10, December 1-3, 2010, Limassol, Cyprus. Copyright 20XX ACM 978-1-4503-0424-5/10/12 $10.00. 1. INTRODUCTION Learning a foreign language often concentrates on the ver- bal aspect of the learning endeavor. But communication is not only concerned with verbal proficiency but it is inher- ently multimodal ranging from facial expressions over ges- tures to spatial behavior, which often follow culturally deter- mined heuristics. As an example consider a dinner table dis- cussion. The structure of such a multiparty conversation can vary from a very orderly turn after turn sequence to a very lively situation where several interactions and discussions take place at the same time between different participants. Often such nonverbal aspects of communication give rise to severe misunderstandings [25]. For instance, the first group in our example might classify the second one as chaotic and unfocused whereas the second group might think of the first one as restrained, distant and cold. Another well studied example is the use of space in interpersonal encounters [7]. While for instance in Northern Europe a certain distance between interlocutors is generally acceptable, in an Arabic context, this distance should not be to far in order to allow for touching between interlocutors. Again the interpreta- tion of the other group’s behavior is bound to differ, often resulting in the first group finding the second group invasive or pushy and the second thinking about the first as distant and cold. This is due to the fact that behavior is interpreted based on unconscious cultural heuristics that are formed by our personal interaction histories in the cultural groups to which we belong. The aim of intercultural training is thus two-fold [10]. First, the trainee has to realize that a com- municative situation is ambiguous and can be interpreted in different ways (awareness) before he can try to adjust to different perspectives (knowledge and skills). In this paper we present our ideas on assisting this en- deavor by a technical solution. Thereby we draw motivation from two different areas. First, virtual agents, e.g. in the form of embodied conversational characters [3], offer natural interaction possiblities because of their potential to emulate verbal and nonverbal human behavior. In general, nonverbal interaction comprises facial expressions, gaze behavior, ges- tures, and body posture, which all play sometimes distinct, sometimes redundant roles in face to face communication. Virtual characters have also been show to be engaging tools for tutoring systems (see Section 2), and thus present a good starting point for exemplifying different perspectives in in- tercultural training and thus presents our first motivation in developing GAME.