ARTICLE IN PRESS JID: ESWA [m5G;March 2, 2016;12:57] Expert Systems With Applications xxx (2016) xxx–xxx Contents lists available at ScienceDirect Expert Systems With Applications journal homepage: www.elsevier.com/locate/eswa Gesture phase segmentation using support vector machines Renata Cristina Barros Madeo, Sarajane Marques Peres ∗ , Q1 Clodoaldo Aparecido de Moraes Lima School of Arts, Sciences and Humanities, University of São Paulo, R. Arlindo Béttio, 1000, São Paulo, SP 03828-000, Brazil a r t i c l e i n f o Keywords: Gesture analysis Gesture phase segmentation Machine learning Support vector machine a b s t r a c t An interaction between humans or between a human and a machine will be more effective if it is sup- ported by gestures. In different levels of complexity, the communication system used in human interac- tion includes the use of gesture. In natural conversation, for instance, speakers use gestures for both to enhance the expressiveness of their speech and to support their own linguistic reasoning. The audience absorbs the content being transmitted also based on the speakers’ gesticulation. Thus, an analysis of ges- tures should add value to the purpose of the interaction. One of the concerns in the analysis of gestures is the problem arising from the segmentation of phases of a gesture (rest position, preparation, stroke, hold and retraction), which, from the standpoint of Gesture Theory, may reveal information on prosody and semantics of what is being said in a discourse. Finding an automation solution to this problem involves enabling the development of theoretical and application areas that are based on the analysis of human behavior and on the interpretation and generation of natural language. In this study, the problem of ges- ture phase segmentation is modeled as a problem of classification, and then support vector machine is employed to design a model able to learn the patterns of gesture that are inherent to each phase. This work presents two main highlights. The first is to address the limitations of the segmentation approach through the study of its performance in different scenarios that represent the complexity of analyzing patterns of human behavior. In this study, we reached an F-score around 0.9 for rest position and around 0.8 for stroke and preparation as segmentation results in the best cases. Moreover, it was possible to in- vestigate how classification models are influenced by human behavior. The second highlight refers to the conduction of an analysis by considering the standpoint of specialists concerned with gesture phase seg- mentation in the area of Linguistics and Psycholinguistics, through which we obtained impressive results. Thus, in regard to the suitability of our approach, it is a feasible means of supporting the development of the Gesture Theory as well as the Computational Linguistics and Human Machine Interaction fields. © 2016 Elsevier Ltd. All rights reserved. 1. Introduction 1 In the last few years, there has been an increase of studies on 2 gesture analysis in Computer Science (Chen, Jafari, & Kehtarbavaz, 3 2015; Han, Shao, Xu, & Shotton, 2013; Madeo, Wagner, & Peres, 4 2013b; Mitra & Acharya, 2007). Currently, both the academic world 5 and industry are interested in gesture analysis, mainly with the 6 aim of developing new methods of Human–Computer (or Machine) 7 Interaction. These methods have been developed considering dif- 8 ferent aspects, such as: different ways of interacting (Turk, 2014); 9 execution of gestures with body parts that are little explored in 10 gesture analysis (Tran, Doshi, & Trivedi, 2012); multiple users inter- 11 ∗ Corresponding author. Tel.: +5511987110994. E-mail addresses: renata.si@usp.br (R.C.B. Madeo), sarajane@usp.br, smperesbr@hotmail.com (S.M. Peres), c.lima@usp.br (C.A.d.M. Lima). acting at the same time with one or more devices (Baraldi, Bimbo, 12 & Landucci, 2008); and interaction with avatars (Kopp, Sowa, & 13 Wachsmuth, 2004) or robots (Salem, S. Kopp, Wachsmuth, Rohlf- 14 ing, & Joublin, 2012). 15 The increasing interest in gesture analysis is partly due to the 16 development of low-cost sensors, for which there are Software 17 Development Kits that provide the means to create applications 18 with different levels of complexity (Guna, Jakus, Poganik, Tomai, & 19 Sodnik, 2014; Lun & Zhao, 2015; Tashev, 2013). These sensors are 20 able to acquire information that can describe human movements 21 at different levels of detail. For example, they have been used to 22 build large-scale benchmark datasets which allow more extensive 23 research to be carried out in the fields of gesture segmentation 24 and gesture recognition (Escalera et al., 2013). Although these sen- 25 sors have become increasingly popular, video cameras (RGB and 26 RGB-D) and webcams are still the main sources of data for hu- 27 man movements. The methods regarding the use of different types 28 http://dx.doi.org/10.1016/j.eswa.2016.02.021 0957-4174/© 2016 Elsevier Ltd. All rights reserved. Please cite this article as: R.C.B. Madeo et al., Gesture phase segmentation using support vector machines, Expert Systems With Appli- cations (2016), http://dx.doi.org/10.1016/j.eswa.2016.02.021