ELENA ERBICEANU, DANIEL MAPES, & CHARLES E. HUGHES 269 16 SYNTHESIZING VIRTUAL CHARACTER BEHAVIORS FROM INTERACTIVE DIGITAL PUPPETRY By Elena Erbiceanu, Daniel Mapes, and Charles E. Hughes 1. INTRODUCTION A major challenge in creating interactive virtual environments is getting the user engaged and interested in the experience, providing the vital connection to the story that is needed to achieve a level of immersion that triggers afect responses. Although verbal communication is straightforward in conveying information, it is the nonverbal behaviors that create the symphony of cues that humans are inherently trained to recognize in person-to-person interaction. For instance, observing a character that is verbally communicating the same story while exhibiting diferent gaze behaviors (e.g., fast gaze shifts with no sustained eye contact versus prolonged, lingering eye contact) can trigger completely diferent afective responses, varying from untrustworthy / hiding something, to casual communication, and all the way to triggering an empathetic response and identifying with the character’s emotions. For example, a character telling a secret is far more convincing when it leans across the table to whisper to you. ELENA ERBICEANU ON HER METHODS Our frst stage of designing a role playing experience in Geppetto involves observation of the real characters to be modeled. Tis is typically an on-site study, e.g., with middle school children, and involves collaboration with members of the subject population. When this collaboration is not possible due to practical, ethical or epistemological concerns, the process is done via subject- matter experts. Our actors and our subject population collaborators are then asked to act out various emotions for each virtual character, according to his/her persona, while being recorded. Te videos are then analyzed to extract the most expressive key frames, which are then used to mimic the behaviors previously recorded to model the virtual character. We do this iteratively until everyone is happy with the articulation of the various emotions. (continued on next page)