Tracking hand and finger movements for behaviour analysis Enrica Dente b, * , Anil Anthony Bharath b , Jeffrey Ng b , Aldert Vrij a , Samantha Mann a , Anthony Bull b a University of Portsmouth, United Kingdom b Faculty of Engineering, Department of Bioengineering, Imperial College London, Vision Research Group, Exhibition Road, London SW7 2AZ, United Kingdom Abstract In this paper, we describe ongoing work into methods for the automated tracking of hand and finger movements in interview situ- ations. The aim of this work is to aid visual behaviour analysis in studies of deception detection. Existing techniques for tracking hand and finger movements are reviewed to place current and future work into context. Posterior probability maps of skin tone, based on Parzen colour space probability density estimates, are used for initial hand segmentation. Blob features are then used to produce a markup of hand-states. A complex wavelet decomposition, coupled to weightings provided by the posterior probability map, is applied to detect small hand and finger movements. We discuss our hand tracking algorithm based on blob feature extraction and the results obtained from motion and orientation parameters in a ‘‘high-stakes experiment’’, designed around a real-life situation. We suggest the role of kinematic models of upper body, limb and finger motion for future work. Ó 2006 Elsevier B.V. All rights reserved. Keywords: Deception; Behaviour analysis; Complex wavelets; Posterior probability maps; Computer vision 1. Introduction Detecting psychological states in interview situations is not only technically challenging, but is also hindered by unsolved issues in behaviour analysis. Recently, non-verbal methods of detecting deception have appeared to be promising. Trained observers are better than non-trained observers at discriminating liars from truth tellers by means of visual cues (Frank and Feeley, 2003; Vrij and Mann, 2004). This has led to the hypothesis that the inte- gration of improved human analysis with automated tech- niques for detecting and tracking cues associated with psychological states (Vrij and Mann, 2004; Burgoon et al., 2005) can help create more reproducible means for detecting human deception. Lu et al., in order to automatically detect deceptive behaviour, have used multiple cues to track changes in the head and hand movements of a subject whose body postures are associated with specific psychological states, i.e. relaxed, agitated and over controlled (Lu et al., 2005). However, their findings were not based on controlled ‘‘high-stakes experiments’’. We define a ‘‘high-stakes exper- iment’’ as a deception situation based on a real-life episode of a subject where the subject being referred to has some- thing very significant to lose (e.g. a job or a relationship with a family member) by being judged deceptive when lying. A situation designed as such is likely to elicit strong emotions that provide cues that can be captured visually (Frank and Ekman, 1997). Laboratory based experiments conducted to-date are low-stakes (Vrij, 2004), because in most cases they are not based on real-life episodes. This reduces their validity, as high-stakes situations are more likely to yield non-verbal cues of deception (Frank and Ekman, 1997). Therefore, it is unlikely that progress into quantifying psychological states, which may be indicative of deception, can be made without designing high-stakes experiments based on real-life situations. However, design- ing high-stakes experiments is problematic because it is dif- 0167-8655/$ - see front matter Ó 2006 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2006.02.010 * Corresponding author. Tel.: +44 207 594 5169; fax: +44 20 75846897. E-mail address: enrica.dente@imperial.ac.uk (E. Dente). www.elsevier.com/locate/patrec Pattern Recognition Letters xxx (2006) xxx–xxx ARTICLE IN PRESS