Color-Based Recognition of Gesture-traced 2D Symbols Danilo Avola, Paolo Bottoni, Alexandru Dafinei, Anna Labella Department of Computer Science, Sapienza University of Rome Via Salaria 113, 00198 Rome, Italy {avola, bottoni, labella}@di.uniroma1.it, a dafinei@yahoo.it Abstract Gesture-based interfaces provide an intuitive and natural way to convey 2D graphical command symbols for interacting with applications and services. Users can prefer performing gestures “in the air” or through suitable tools, e.g. tablets, depending on the context of usage and the available tools. A collection of device-independent algorithms and tools for analysing gestures would facilitate such a plasticity in the inter- action process. This paper describes how a set of 2D graphical sym- bols, traced by moving a led pen in front of a single- camera, can be recognised in a gesture recognition sys- tem, using the same algorithms employed for recognis- ing these symbols when used on a tablet. 1 Introduction The main goal in the design of multimodal inter- faces is to increase the naturality of human-machine communication process, exploiting typical communi- cational behaviours of human beings. Gesture- and speech-based modalities are considered among the most promising ways for human-machine communica- tion, as they are most common in human-human com- munication. For the case of visual concepts (e.g. graph- ical symbols), these are better expressed through ges- ture modality, when it is possible to achieve an immedi- ate abstract visualization of temporal, geometrical and spatial characteristics. Users can perform gestures finalized to command a device using a specific tool (e.g. movement sensors, tablet PC) or simply by tracing signs with their hands captured by some optical device. Depending on sev- eral factors, e.g. application environments (e.g. kind of application/service, kind of device) or technical con- texts (e.g. indoor, outdoor, presence of light), different interaction supports may be preferred. The type of support, as well as the choice to focus on the hand movements, heavily influence the approaches used in the recognition process and the involved algorithms. A possible classification of gesture recognition al- gorithms is based on the way by which they extract movement information from low level data. In partic- ular, algorithms can be classified as tracker based or vision based [3]. To the first class algorithms belong which base the recognition process on haptic devices (e.g. data gloves, body suits) by which to capture the whole gesture informative content, while in the sec- ond class we find algorithms that obtain information by analyzing a sequential set of images representing a human-computer interaction scene. This paper describes a simple gesture recognition process in to the second class, where gesture move- ments, performed by the user through the management of a led pen (or some other simple gadget coloured with a uniform tonality), is detected by a color tracking algo- rithm that analyzes a sequential set of images captured by a single-camera. Once the spatial and temporal in- formation of the tracked color has been obtained, a sketch recognition engine is adopted to interpret the related graphical symbol. The basic idea of the system is that the user can define, through a sketch-based interface (i.e. based on the stroke identification), a set of recognizable ges- tures. In fact, the core system is based on a sketch recognition algorithm built to face the main aspects in this field multi-domain definition, real-time recognition and tracing style interpretation [1]. For this reason, our approach identifies the trajectory (i.e. spatial and temporal information) of the tracked color as a set of sequential points through which to define a stroke rep- resenting an abstraction of the user’s gesture. In the following, Section 2 presents the system ar- chitecture and shows some preliminary experimental results and Section 3 concludes the paper. 1