Detection of stereotyped hand flapping movements in Autistic Children using the Kinect Sensor: a Case Study Nuno Gonçalves, Sandra Costa, José Rodrigues R&D Centre Algoritmi University of Minho Campus de Azurém, 4800-058 Guimarães, Portugal Filomena Soares R&D Centre Algoritmi University of Minho Campus de Azurém, 4800-058 Guimarães, Portugal filomena.soares@algoritmi.uminho.pt Abstract— This paper presents a case study on the use of the Microsoft Kinect sensor and the gesture recognition algorithm, Dynamic Time Warping (DTW to automatically detect stereotyped motor behaviours in children with Autism Spectrum Disorder (ASD). The proposed system was initially tested in a laboratory environment, and after the encouraging results obtained, the system was tested during the sessions with children with ASD in two schools with Special Needs Units. Video analysis was used to validate the results obtained using the Kinect application. This study provides a valuable tool to monitor stereotypes in order to understand and to cope with this problematic. In the end, it facilitates the identification of relevant behavioural patterns when studying interaction skills in children with ASD. Keywords— Stereotypical Motor Movements, Kinect Sensor, Gesture Recognition, Autism Spectrum Disorders. I. INTRODUCTION Autism is a developmental disability that typically appears during the first three years of life, being defined as a global development disorder [1]-[4]. The symptoms and characteristics of autism spectrum disorders (ASD) are a result of a neurological disorder that affects the normal development of the brain in the areas of social interaction, communication skills, and emotion recognition. Children and adults with ASD have difficulties in verbal and non-verbal communication in social interactions [5], [6]. Typically, children with ASD manifest stereotyped behaviours, especially when there is a change in their daily routine or when the environment is full of new inputs and as a result, they tend to block the excessive feelings, manifesting stereotyped behaviours. Given that children with ASD have difficulties in communicating, establishing relationships and responding appropriately to the environmental stimulus, failing to develop their full potential, the Robótica-Autismo Project studies whether a robotic mechanism, previously programmed is able to help to promote the referred skills in these children. The work presented in this paper is part of a collaborative project [7]-[10] between the University of Minho, APPACDM (a Portuguese association for mentally disabled people) of Braga and a group of schools, where the main goal is to improve the social life of children with ASD, in particular to promote their social interaction, by using a robot as a mediator or reward. The intervention sessions between the child and the robot were recorded on video and pre-defined behaviour indicators were quantified. One of these indicators was the manifestation of stereotyped movements. Therapists measured this behaviour using traditional methods such as paper-pencil rating scales, direct observation or video-based methods that normally are time consuming. Thus, the main objective of this work is to understand if the Kinect sensor can be used as a time-efficient tool to automatically detect in real-time stereotyped behaviours in children with ASD. This will allow characterizing children’s behaviour during the sessions, when they are exposed to new situations, and in the future to use this input as a form to adapt the behaviour of the robot to the state of mind of the child. This paper is divided in six sections: Introduction, where it is presented the paper focus and motivation. Section II is dedicated to the related work and the overall methodology is presented in Section III. Section IV presents the stereotyped behaviour detection based on the Kinect sensor, as well as the experimental approach, and the experimental methodology is explained in Section V. The results obtained in class environment with Autistic children are presented in Section VI, and the conclusions and further work are detailed in the last section. II. RELATED WORK Recently, systems have been developed to automatically detect stereotyped behaviours using accelerometers and pattern recognition algorithms with promising results. The work presented by [11] and [12] uses a three-axis accelerometer to acquired data. The accelerometer is placed on each wrist and chest of the child to detect hand flapping and body rocking movement associated to these problematic. The pattern recognition algorithm used is based on the acquisition 2014 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC) May 14-15, Espinho, Portugal 978-1-4799-4254-1/14/$31.00 ©2014 IEEE 212