Finger-Spelling Recognition System using Fuzzy Finger Shape and Hand Appearance Features Kittasil Silanon Department of Computer Engineering, Faculty of Engineering, Prince of Songkla University Hatyai, Songkhla, Thailand 90112 Kittasil.silanon@gmail.com Nikom Suvonvorn Department of Computer Engineering, Faculty of Engineering, Prince of Songkla University Hatyai, Songkhla, Thailand 90112 Nikom.SUVONVORN@gmail.com Abstract— In this paper, we introduce a method for finger- spelling recognition system. The objective is to help the deaf or non-vocal persons to improve their skills on the finger-spelling. Many researches in this field have proposed methods mostly based on hand posture estimation techniques. We propose an alternative flexible method based on fuzzy finger shape and hand appearance analysis. By using depth image, the hand is extracted and tracked using an active contour like method. Its features, such as, finger shape, and hand appearance, have been defined as chain code, which are input to the American finger-spelling recognition system by using a vote method. The performance of the system is tested in real-time environment, which results in around 70% recognition rate. Keywords—Finger-Spelling; Hand Posture Estimation; American Finger-Spelling I. INTRODUCTION In this paper, we present our research on sign language, especially related to the finger-spelling. The finger-spelling is a basic communication method for deaf and non-vocal persons, in which the hand posture as symbol will represents the alphabets of words of spoken language, such as, names, places, technical words and etc. However, most of these people, especially children, have problems with finger- spelling skills. Usually, the word-level vocabulary signs have been used for communicating with each other, and only 7% to 10% of the finger-spelling is used in the daily life. Evidently, the finger-spelling skills lag far behind the sign language skills. Our research goal to the field is to develop an automatic recognition system for the finger- spelling, in order to help these people to improve their skills. Actually, many systems specific to a language are proposed, for examples, finger-spelling of American (ASL) [11,13,15], British (BSL) [12], Australian (Auslan) [10], Chinese (CSL) [18], Japanese (JSL) [17] etc. Various researches have been proposed, but most of them cannot achieve the critical criteria, such as, accuracy, flexibility, and real time constraint. There are two principle approaches: glove-based and vision-based. The gloves-based methods [14,16] use electronic sensor devices for digitizing hand joint and finger motion, which give the precision of the hand posture that result in high recognition rate in real time, but these methods are very limited by the environment configuration. The vision-based approach consists of two groups of techniques. Firstly, the model-based method [1,5] uses a kinematic hand model to estimate the articulated hand (i.e., joint angle, finger position), leading to a full reconstruction of the articulated hand posture. Secondly, the appearance-based method [4,9,19] uses computer vision techniques to extract important features from images, such as, point, edge, contour or silhouette, for reconstructing the hand posture, and then, for recognizing the finger-spelling. In this paper, we proposed a vision-based method for hand posture estimation. The method combines both model and appearance-based method using finger shape and hand appearance features, to finally recognize the American finger-spelling. The system consists of four main parts: 1) hand segmentation, to segment the region of interest of the hand form image sequence, 2) key hand posture selection, to determine the key frame representing the hand posture of finger-spelling from image sequences, 3) hand feature definition, to define the finger shapes and hand appearance features as chain code sequence, 4) finger-spelling recognition, to recognize the finger-spelling from hand features by simply using a scored voting method. The paper details the four parts of our method, then the experimentation results and conclusion, respectively. II. HAND SEGMENTATION In this step, the method is focused on the segmentation of the region of interest of the hand from the rest of the image. In our experimentation, the hand is simply defined by the closest object to the camera. Since we used depth image, as shown in Fig.1.b, thus the segmentation of the hand from the complex background can be done by using predefined threshold to obtain the hand’s region in image, as shown in Fig.1.c. (a) (b) (c) Fig. 1. Depth information: (a) image (b) depth image (c) hand region. ISBN: 978-1-4799-3724-0/14/$31.00 ©2014 IEEE 419