A. Camurri, C. Costa, and G. Volpe (Eds.): INTETAIN 2011, LNICST 78, pp. 160–169, 2012. © Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2012 Towards Mimicry Recognition during Human Interactions: Automatic Feature Selection and Representation Xiaofan Sun 1 , Anton Nijholt 1 , and Maja Pantic 1,2 1 Human Media Interaction, University of Twente PO Box 217, 7500 AE Enschede, The Netherlands 2 Department of Computing, Imperial College 180 Queen’s Gate, London SW7 2AZ, UK {x.f.sun,a.nijholt}@ewi.utwente.nl. m.pantic@imperial.ac.uk Abstract. During face-to-face interpersonal interaction people have a tendency to mimic each other, that is, they change their own behaviors to adjust to the behavior expressed by a partner. In this paper we describe how behavioral information expressed between two interlocutors can be used to detect and identify mimicry and improve recognition of interrelationship and affect between them in a conversation. To automatically analyze how to extract and integrate this behavioral information into a mimicry detection framework for improving affective computing, this paper addresses the main challenge: mimicry representation in terms of optimal behavioral feature extraction and automatic integration. Keywords: mimicry representation, human-human interaction, human behavior analysis, motion energy. 1 Introduction Mimicry plays an important role in human-human interaction. Mimicry refers to the coordination of movements in both timing and form during interpersonal communication. Behavior matching, synchronized changes in behavior and facial expressions, matching in posture and mannerisms are examples of mimicry. But there can also be vocalic mimicry and matching of verbal style. Mimicry is ubiquitous in daily interpersonal interaction. For example, when two interactants are facing each other and one of them takes on a certain posture such as moving sideways or leaning forward, then the partner may take on a congruent posture [1], [2], [12], and when one takes on certain mannerism such as rubbing the face, shaking the legs, or foot tapping, the partner may take on a congruent mannerism [2]. Another example, if one is crossing his legs with the left leg on top of the right, the other may also cross his legs with the right leg on top of the left leg (called “mirroring”) or with the left leg on top of the right leg (called “postural sharing”).