Sci.Int.(Lahore),29(3),515-519,2017 ISSN: 1013-5316; CODEN: SINTE 8 515 May-June A HYBRID METHOD USING KINECT DEPTH AND COLOR DATA STREAM FOR HAND BLOBS SEGMENTATION Mostafa Karbasi, Zulkefli Muhammad, Ahmad Waqas, Zeeshan Bhatti, Asadullah Shah, M.Y.Koondhar, Imtiaz Ali Brohi Department of Computer Science, Kulliyyah of Information and Communication Technology, International Islamic University Malaysia For Correspondence; Mostafa.karbasi@live.iium.edu.my, Ahmad.waqas@live.iium.edu.my, Zeeshan.bhatti@live.iium.edu.my, Asadullah@iium.edu.my, yaqoobkoondhar@gmail.com, brohiimtiaz@hotmail.com ABSTRACT: The recently developed depth sensors such as Kinect have provided new potentials for human-computer interaction (HCI) and hand gestures are one of main parts in recent researches. Hand segmentation procedure is performed to acquire hand gesture from a captured image. In this paper, a method is proposed to segment hand blobs using both depth and color data frames. This method applies a body segmentation and an image thresholding techniques to depth data frame using skeleton data and concurrently it uses SLIC super-pixel segmentation method to extract hand blobs from color data frame with the help of skeleton data. Finally, two segmented blobs are combined to improve the final result by assuming that hands are located in front of the body. The proposed method has low computation time and shows significant result when the basic assumptions are fulfilled. Keywords: hand gesture recognition, human computer interaction, simple linear iterative clustering (SLIC), hand detection, posture recognition. 1. INTRODUCTION Vision based hand gesture needed in hand detection and it could be the most vital things in hand recognition. In some condition hand detection can apply on human body and it works as a boundary and limitation compare to any other things. Now, here mention some obstetrical in this area: 1- Human body has a 3D shape so hand shape as well would give very different lay out according time and position. 2- Light and background make a lot of problem in hand detection. Skin color and background color next to it, also similar color in background n skin color is used as information. Also depth image is another way to concentrate and find out the fine and right information in RGB image. everything in the background can be discarded with a threshold on the depth values [1]. The TOF camera used in this paper is a Kinect 360. Anyway, this is an inexpensive camera which is IR based. The TOF camera has resolution around 680*480. High resolution can help out the better result with this camera [2]. Kinect and TOF has a depth sensor for gesture recognition. Obviously arm movement and hand shape are using in gesture recognitions. Mostly, contour base on 2D feature shows a hand shape and simulating the hand shape with 2D is the most important and hard process [3] . Furthermore, the arm movement feature is affected by environmental changing, such as individual differences in body size, camera position and so on because the coordinate of centroids of hand region are used as arm movement feature [4]. In this work, we used depth, color data frames and user skeleton data to extract user hand blobs. The user’s body extracted using skeleton data adapted to segment depth data. Unnecessary parts are removed by thresholding technique. To extract the user's body, the color data frame is masked by depth data, then SLIC super-pixel segmentation algorithm is employed to cluster color data frame, after that hand blobs are detected based on skeleton data. The main contribution of this paper is developing hand detection methods based on depth and color fames and user skeleton data. It is perfect and robust in hand orientation, motion, position and postures. The proposed method operates accurately and efficiently in uncontrolled environments. 2. LITERATURE REVIEW There are new technologies such as Kinect and Zcam introduced in the market which has been used by many researchers for different application. Some of researchers worked on hand segmentation, hand counter, color distribution, etc. [5] proposed kd-tree structure for hand and head detection and tracking which is exploited to resolve ambiguities and overlaps. Park et al [2] proposed adaptive hand detection approach by using 3-dimentional information from Kinect and tracks the hand using GHT based method. [6] used RGB stream and depth data for hand detection in their research and contributed in 3D contour model for real time application. In addition, their result shows that proposed method is successful in handling real time interaction in desktop environment with clustering background method. [7] adapted region growing and bilateral filter for depth map enhancement and detection. The proposed method can significantly improve the quality of depth maps and enlarge Kinect’s applicant ion fields where high quality depth images are required. [8] used depth data for hand detection based on distances. Their method included background subtraction and shadow removal for removing redundant data. [9] also used depth and depth and color data for hand detection and sign language recognition. They implemented Finger-Earth Mover’s Distance (FEMD) as a new approach for sign language recognition. In addition, their methods have been implemented in two applications such as arithmetic computation and rock-paper-scissors game. [1] presented real time system for hand detection and gesture recognition on the base of ToF camera and the RGB. The proposed method not only improved detection rate, but also allows for the hand to overlap with the face, or with hands from other persons in the background. [10] worked on Sign Language Recognition with the help of 3D convolutional neural networks. 3D convolutional neural networks can extract spatio-temporal