Communication and automatic interpretation of affect from facial expressions Albert Ali Salah University of Amsterdam, the Netherlands Nicu Sebe University of Trento, Italy Theo Gevers University of Amsterdam, the Netherlands ABSTRACT The objective of this chapter is to introduce the reader to the recent advances in computer processing of facial expressions and communicated affect. Human facial expressions have evolved in tandem with human face recognition abilities, and show remarkable consistency across cultures. Consequently, it is rewarding to review the main traits of face recognition in humans, as well as consolidated research on the categorization of facial expressions. The bulk of the chapter focuses on the main trends in computer analysis of facial expressions, sketching out the main algorithms and exposing computational considerations for different settings. We then look at some recent applications and promising new projects to give the reader a realistic view of what to expect from this technology now and in near future. INTRODUCTION In June 2009, Microsoft released a trailer of its latest project for Xbox 360 gaming console, called Project Natal. The video, an instant Facebook epidemic and a YouTube favourite, featured Peter Molyneux, the creative director of Microsoft Game Studios Europe, demonstrating a virtual agent called Milo. Using the sensing and processing capabilities of its hardware, the virtual agent communicated with the user as if the boundary of the screen is just a window, recognizing identity and speech, but also emotions, which enabled it to respond to the user with an impressive range of realistic behaviours. The innovation of the project was in its ambitious scope: creating a virtual agent that truly communicates with the user. The key to life–like communication was recognizing emotions of the user, and in return, generating states that carry affect information for the agent in human–readable form, i.e. in the body posture, vocal intonation, and most importantly, facial expression. The recently flourishing field of social signal processing (Vinciarelli et al., 2009) targets a greater contextual awareness for computer systems and human–machine interaction, and drawing on cognitive psychology, places great emphasis on automatically understanding facial expressions. The human face is a window that allows peeking into diverse patterns of emotions that manifest themselves voluntarily and involuntarily, communicating affect or projected displays of personality. Even dissociated from gesture and voice (as in still face pictures), facial expressions convey complex, layered, and vital information. Consequently, it is a great challenge to create computer systems that can automatically analyse images to reveal the sometimes obvious and sometimes subtle messages engraved in faces. In this chapter we aim to provide the reader with a broad overview of how computer scientists have risen to this challenge, starting from relevant taxonomies and guidelines, briefly touching upon the cognitive aspects of affect and face recognition, summarizing recent advances in algorithmic aspects of the problem, giving pointers and tools for the initiate, and finally, discussing applications and the future of facial expression recognition. CATEGORIZATION OF FACIAL EXPRESSIONS