A Multimodal Communication with A Haptic Glove: On the Fusion of Speech and Deixis Over a Raised Line Drawing Francisco Oliveira * Virginia Tech 2202 Kraft Drive Blacksburg, Virginia, USA oliveira@cs.vt.edu Francis Quek † Virginia Tech 2202 Kraft - 1130 KWII Blacksburg, Virginia, USA quek@cs.v.edu ABSTRACT Mathematics instruction and discourse typically involve two modes of communication: speech and graphical presenta- tion. For the communication to remain situated, dynamic synchrony must be maintained between the speech and dy- namic focus in the graphics. In normals, vision is used for two purposes: access to graphical material and awareness of embodied behavior. This embodiment awareness keeps com- munication situated with visual material and speech. Our goal is to assist blind students in the access to such instruc- tion/communication. We employ the typical approach of sensory replacement for the missing visual sense. Haptic fingertip reading can replace visual material. For the em- bodied portion of the communication, we want to make the blind student aware of the deictic gestures performed by the teacher over the graphic in conjunction with speech. We propose the use of haptic gloves paired with computer vision based tracking to help blind students maintain reading focus on a raised line representation of a graphical presentation to which the instructor points while speaking. In this initial phase of our research, we conducted three experiments that show that: 1) The gloves convey sense of direction; 2) The gloves do not interfere in fingertip reading; 3) A person can navigate with the help of this system while listening to a story; 4) It is possible to fuse the information received from both modes. We discuss these findings in this paper. General Terms Multimodal, awareness, embodiment, gesture, sensory re- placement * Francisco Oliveira is a PhD candidate at Virginia Tech † Francis Quek is the Director of the Center for Human Com- puter Interaction at Virginia Tech Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. PETRA ’08 Athens, Greece Copyright 200X ACM X-XXXXX-XX-X/XX/XX ...$5.00. Categories and Subject Descriptors H.5 [Information Interfaces and Presentation]: Haptic I/O 1. INTRODUCTION According to the American Foundation for the Blind, 45% of individuals with severe visual impairment or blindness have a high school diploma [5]. Around 80% of sighted stu- dents receive the degree. The National Federation of the Blind (NFB) estimates that in the United States alone, more than 93,000 children on school age are blind [19]. Further- more, blind students are typically one to three years behind their seeing counterparts [27]. There is evidence that blind students are capable of spa- tial reasoning, and by extension, mathematical reasoning [15, 18, 8, 14]. Assuming that the environmental conditions which affect learning (e.g. family conditions) are equal, the reason for this deficit must lie either in the paucity of learn- ing material or in the communication of that material to students who are blind. A variety of devices to produce raised line drawings rapidly are available. While there is room for improvement in the development of better tactile display devices, this paper explores the understudied area of improving mathematics instruction by aiding the communi- cation between teacher and student. Human to human communication has to be ‘situated’. In brief, the discourse must maintain common ground between the communicants [3]. Since mathematical instruction typ- ically involves a spatial/graphical component, this common ground must be maintained in the communication with re- spect to both speech and the graphical material. In commu- nication between sighted people in the presence of an object of discussion, the fusion of gestural information (e.g. point- ing), speech (what is said), and the object referenced (the deictic field) are required to situate the discourse and facili- tate comprehension [17, 2, 9]. For example, in a typical lec- ture setting, seeing students can fuse the information about the presentation graphic and the speech of the teacher, us- ing the embodied behavior of the teacher to fuse the salient portion of the graphic with the co-temporal speech. For students who are blind, the graphical material may be replaced with raised line material, but access to the gestu- ral/deictic behavior of the teacher is an open problem. For example, a thorough analysis of the techniques and equip- ment available to enhance learning in a lecture setting was done by Supalo [24], a chemistry graduate student at Penn- sylvania State University, who is blind. The problem with Permission to make digital or hard copies of part or all of this work or personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. PETRA'08, July 15-19, 2008, Athens, Greece. Copyright 2008 ACM 978-1-60558-067-8... $5.00