Avatar Puppetry Using Real-Time Audio and Video Analysis Sylvain Le Gallou 1, 2 , Gaspard Breton 1 , Renaud S´ eguier 2 , and Christophe Garcia 1 1 France Telecom, TECH/IRIS Team, rue du Clos Courtel, Cesson-S´ evign´ e, France firstname.lastname@orange-ftgroup.com 2 Sup´ elec/IETR, SCEE Team, avenue de la Boulaie, Cesson-S´ evign´ e, France firstname.lastname@supelec.fr Abstract. We present a system which consists of a lifelike agent ani- mated in real-time using video and audio analysis from the user. This kind of system could be used for Instant Messaging where an avatar controlled like a puppet is displayed instead of the webcam flow. The overall system is made of video analysis based on Active Appearance Models and audio analysis based on Hidden Markov Model. The pa- rameters from these two modules are sent to a control system driving the animation engine. The video analysis extracts the head orientation and the audio analysis provides the phonetic string used to move the lips. 1 System Overview Fig. 1. Overview of the pupettry system C. Pelachaud et al. (Eds.): IVA 2007, LNAI 4722, pp. 391–392, 2007. c  Springer-Verlag Berlin Heidelberg 2007