Personal and Ubiquitous Computing manuscript No. (will be inserted by the editor) A Database-Based Framework for Gesture Recognition Vassilis Athitsos · Haijing Wang · Alexandra Stefan Received: date / Accepted: date Abstract Gestures are an important modality for human-machine communication. Com- puter vision modules performing gesture recognition can be important components of in- telligent homes, assistive environments, and human-computer interfaces. A key problem in recognizing gestures is that the appearance of a gesture can vary widely depending on variables such as the person performing the gesture, or the position and orientation of the camera. This paper presents a database-based approach for addressing this problem. The large variability in appearance among different examples of the same gesture is addressed by creating large gesture databases, that store enough exemplars from each gesture to capture the variability within that gesture. This database-based approach is applied to two gesture recognition problems: handshape categorization and motion-based recognition of American Sign Language (ASL) signs. A key aspect of our approach is the use of database indexing methods, in order to address the challenge of searching large databases without violating the time constraints of an online interactive system, where system response times of over a few seconds are oftentimes considered unacceptable. Our experiments demonstrate the benefits of the proposed database-based framework, and the feasibility of integrating large gesture databases into online interacting systems. Keywords gesture recognition · hand pose estimation · embeddings · American Sign Language · indexing methods · image and video databases 1 Introduction Gestures are an important modality for human-machine communication, and robust gesture recognition can be an important component of intelligent homes, assistive environments, and human-computer interfaces in general. A key problem in recognizing gestures is that the appearance of a gesture can vary widely depending on variables such as the person Vassilis Athitsos, Haijing Wang, Alexandra Stefan Computer Science and Engineering Department University of Texas at Arlington Arlington, Texas, USA Tel: +1-817-272-0155 Fax: +1-817-272-3784