Body Movements for Affective Expression: A Survey of Automatic Recognition and Generation Michelle Karg, Ali-Akbar Samadani, Rob Gorbet, Kolja K€ uhnlenz, Jesse Hoey, and Dana Kuli  c Abstract—Body movements communicate affective expressions and, in recent years, computational models have been developed to recognize affective expressions from body movements or to generate movements for virtual agents or robots which convey affective expressions. This survey summarizes the state of the art on automatic recognition and generation of such movements. For both automatic recognition and generation, important aspects such as the movements analyzed, the affective state representation used, and the use of notation systems is discussed. The survey concludes with an outline of open problems and directions for future work. Index Terms—Movement analysis, recognition of affective expressions, generation of affective expressions Ç 1 INTRODUCTION A FFECTIVE computing aims to enhance human-computer interaction (HCI) and human-robot interaction (HRI) through affective communication to create a more intuitive, engaging, and entertaining interaction. During the interac- tion, affective states can be expressed and recognized through facial expressions, speech, body movements, and physiological parameters. Automatic recognition of human affective expressions and generation of expressive behavior for virtual avatars and robots are key challenges in this research area. Several surveys address detection of affective states in general [1], [2], [3], [4], [5], through facial or/and audio expressions [6], [7] and generation of affective expres- sions [4], [8]. As facial expressions and speech dominate during face-to-face interaction, these are the modalities that have been predominantly studied in communication of non- verbal behavior, psychology, and computer science to date [2], [9], [10], [11]. Yet, there exists evidence from communi- cation of nonverbal behavior and psychology research that body movements also convey affective expressions, e.g., [12], [13], [14], [15]. Considering body movement as a modality for affective computing is particularly suitable in situations where the affective state is estimated from a dis- tance [16], to retrieve expressions which are less susceptible to social editing [17], and to communicate affective states which are easier conveyed through movement [18]. A recent survey [19] reviews the literature on affect rec- ognition from body posture and movement, and discusses the main challenges in affect recognition from body pos- ture and movement, including inter-individual differences, impact of culture and multi-modal recognition, and the challenges in collecting appropriate data sets and ground truth labeling. Computational models have been devel- oped for both automatic recognition and generation of affect- expressive movements. 1 A large body of work has emerged in recent years developing these computational models; this survey is intended to synthesize the findings of these studies, and to identify key contributions and open research questions. Two significant characteristics of these computational models are 1) the representation of movements in physical space and time, and 2) the repre- sentation of affect. To provide a comprehensive overview of affect- expressive movements studied to date in HCI/HRI, we introduce a suitable movement categorization and sum- marize works studying similar movements. We discuss the use of movement notation systems in automatic rec- ognition and generation of affect-expressive movements. Movement notation systems, commonly used in the dance community, can provide a systematic approach for the choice of movement descriptors and facilitate knowledge transfer between communication, psychology, and computer science. This elaboration on the move- ments studied to date and the use of movement notation systems for both automatic recognition and generation provides complementary information to the previous survey discussing the importance of postural and dynamic features for automatic recognition [19]. A categorical or dimensional approach can be used for representing affective states. For both automatic recognition and generation studies, we analyze the set of considered affective states and their representation. We report on the common results regarding the expressiveness of affective  M. Karg, A.-A. Samadani, and D. Kuli c are with the Department of Elec- trical and Computer Engineering, University of Waterloo, Canada, 200 University Avenue West, Waterloo, ON N2L 3G1, Canada.  R. Gorbert is with the Centre for Knowledge Integration, University of Waterloo, 200 University Avenue West, Waterloo, Ontario, N2L 3G1, Canada.  K. K€ uhnlenz is with the Dept. of EE and CS, Coburg University of Applied Sciences and Arts, Germany.  J. Hoey is with the David R. Cheriton School of Computer Science, Univer- sity of Waterloo, 200 University Avenue West, Waterloo, ON N2L 3G1, Canada. Manuscript received 21 June 2013; revised 23 Oct. 2013; accepted 28 Oct. 2013.; date of publication 11 Nov. 2013; date of current version 13 Mar. 2014. Recommended for acceptance by A. Batliner. For information on obtaining reprints of this article, please send e-mail to: reprints@ieee.org, and reference the Digital Object Identifier below. Digital Object Identifier no. 10.1109/T-AFFC.2013.29 1. In this work, we introduce the term affect-expressive movement to mean that subset of expressive movements whose purpose is to convey affect. IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, VOL. 4, NO. 4, OCTOBER-DECEMBER 2013 341 1949-3045 ß 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.