A review of motion analysis methods for human Nonverbal Communication Computing ☆ Dimitris Metaxas ⁎, Shaoting Zhang Center for Computational Biomedicine Imaging and Modeling (CBIM), Department of Computer Science, Rutgers University, Piscataway, NJ, USA abstract article info Article history: Received 6 December 2012 Received in revised form 14 March 2013 Accepted 29 March 2013 Keywords: Nonverbal Communication Computing Motion analysis Face tracking Facial expression recognition Gesture recognition Group activity analysis Human Nonverbal Communication Computing aims to investigate how people exploit nonverbal aspects of their communication to coordinate their activities and social relationships. Nonverbal behavior plays impor- tant roles in message production and processing, relational communication, social interaction and networks, deception and impression management, and emotional expression. This is a fundamental yet challenging re- search topic. To effectively analyze Nonverbal Communication Computing, motion analysis methods have been widely investigated and employed. In this paper, we introduce the concept and applications of Nonver- bal Communication Computing and also review some of the motion analysis methods employed in this area. They include face tracking, expression recognition, body reconstruction, and group activity analysis. In addi- tion, we also discuss some open problems and the future directions of this area. © 2013 Published by Elsevier B.V. 1. Introduction Understanding how people exploit nonverbal aspects of their communication to coordinate their activities and social relationships is a fundamental scientific challenge. Deeper insights into nonverbal communication can have a profound impact on how we link theories of perception, learning, cognition and action to models of interactions and groups at the social level. Models of nonverbal behaviors in inter- action are essential for collaboration tools, human–computer and vir- tual interaction and other assistive technologies designed to support people in real-world activities. This knowledge is also useful to devel- op models of the deficits of specific populations, such as autistic chil- dren, and interventions that bring them into fuller participation in communities. In general, nonverbal communication research offers high-level principles that might explain how people organize, display, adapt and understand such behaviors for communicative purposes and social goals. However, the specifics are generally not fully under- stood, nor is the way to translate these principles into algorithms and computer-aided communication technologies such as intelligent agents. To model such complex dynamic processes effectively, novel com- puter vision and learning algorithms are needed that take into ac- count both the heterogeneity and the dynamicity intrinsic to behavior data. As one of the most active research areas in computer vision, human motion analysis has become a widely-used tool in this area. It uses image sequences to detect and track people, and also to interpret human activities. Emerging automated methods for analyzing motion [1] have been studied and developed to enable tracking diverse human movements precisely and robustly as well as correlating multiple people's movements in interaction. Some of the applications of using motion analysis methods for Nonverbal Communication Computing include deception detection, expression recognition, sign language recognition, behavior analysis, and group activity recognition. In the following we illustrate several examples of Nonverbal Communication Computing. Fig. 1 shows an example of deception detection during interactions using an automated motion analysis system [2]. This work investigates how degree of the interactional synchrony can signal whether an inter- actant is truthful or deceptive. This automated, data-driven and unob- trusive framework consists of several motion analysis methods such as face tracking, gesture detection, facial expression recognition and in- teractional synchrony estimation. It is able to automatically track ges- tures and analyze expressions of both the target interviewee and the interviewer, extract normalized meaningful synchrony features and learn classification models for deception detection. The analysis results show that these features reliably capture simultaneous synchrony. The relationship between synchrony and deception is shown to be correlat- ed and complex. The second example is to use an automated motion analysis system to recognize facial expressions of emotions and fatigue from sleep loss in spaceflight [3]. Specifically, this research project aims to develop non-obtrusive objective means of detecting and mitigating cognitive performance deficits, stress, fatigue, anxiety and depression for the op- erational setting of spaceflight. To do so, a computational model-based tracker and an emotion recognizer of the human face have been Image and Vision Computing 31 (2013) 421–433 ☆ This paper has been recommended for acceptance by Matti Pietikainen. ⁎ Corresponding author. 0262-8856/$ – see front matter © 2013 Published by Elsevier B.V. http://dx.doi.org/10.1016/j.imavis.2013.03.005 Contents lists available at SciVerse ScienceDirect Image and Vision Computing journal homepage: www.elsevier.com/locate/imavis