From Mediated Human-Human Interactions to Human-Virtual Character Interactions: Selection and Categorization of Expressive Gestures Aurélie Zara 1,2 , Valérie Maffiolo 1 , Jean Michel Adam 2 , Michel Dubois 3 1 France Telecom Orange Labs, 2 av. P. Marzin, 22300 Lannion, France 2 Laboratoire d'Informatique de Grenoble, 385 rue de la Bibliothèque BP53, 38041 Grenoble cedex 9, France 3 Laboratoire des Sciences de l’Education, Université de Grenoble 2, BP 47 - 38040 GRENOBLE Cedex 9 – France {aurelie.zara, valerie.maffiolo}@orange-ftgroup.com ; Jean-Michel.adam@imag.fr;Michel.Dubois@upmf-grenoble.fr Keywords: expressive gestures, mediatised interaction, game, emotion. Introduction. In order to design affective interactive systems, experimental grounding is required for studying multimodal expressions of emotion during interaction. Several studies have shown that expressions of anthropomorphic characteristics, such as cognitive states and processes (e.g. doubt, interest or thinking), are also necessary to enhance the believability of a virtual character [1][2]. Nowadays, a minority of these agents is able to express these characteristics and only through few modalities (e.g. gaze and facial expressions) [2]. Our aim is to select and categorize human gestures and facial displays, expressing emotions, cognitive states and processes, and several communicative acts (e.g. a nod expressing "yes"). The selected behaviours will be replayed by a virtual character to elicit fun and commitment of users implied in an interaction with the character. In this framework, we chose a corpus-based approach of human-human interactions in a game context. Procedure. We defined a protocol called EmoTABOO2, which is an adaptation of the game Taboo. One of the two players had to guess a word described by the other player using his own speech and gestures, without uttering five forbidden words. Different strategies were used to elicit various emotional and expressive behaviours. A previous face to face version of this protocol is described in [3]. To be as closed as possible to a human-virtual character interaction, the recorded human-human interactions were mediated via webcams and microphones. One of the two players could see the upper body of his/her partner on a small screen. The latter was equipped with coloured markers placed on arm articulations, in order to generate the animation of the virtual character's gestures via a motion capture technique. We selected two subjects for their extraversion and expressiveness, as "model" for the virtual character. Both of them had to interact with the same 4 interlocutors. The 8 pairs of players were recorded during one hour each, from three views: the face of the two players and the upper body of the model. A stereoscopic camera was also used in