Grasping gestures: Gesturing with physical artifacts ELISE VAN DEN HOVEN 1 AND ALI MAZALEK 2 1 User-Centered Engineering Group, Industrial Design Department, Eindhoven Universityof Technology, Eindhoven, The Netherlands 2 Synaesthetic Media Lab, Digital Media Program, Georgia Institute of Technology, Atlanta, Georgia, USA (RECEIVED July 19, 2010; ACCEPTED October 6, 2010) Abstract Gestures play an important role in communication. They support the listener, who is trying to understand the speaker. How- ever, they also support the speaker by facilitating the conceptualization and verbalization of messages and reducing cog- nitive load. Gestures thus play an important role in collaboration and also in problem-solving tasks. In human–computer interaction, gestures are also used to facilitate communication with digital applications, because their expressive nature can enable less constraining and more intuitive digital interactions than conventional user interfaces. Although gesture re- search in the social sciences typically considers empty-handed gestures, digital gesture interactions often make use of hand- held objects or touch surfaces to capture gesturesthat would be difficult to track in free space. In most cases, the physical objects used to make these gestures serve primarily as a means of sensing or input. In contrast, tangible interaction makes use of physical objects as embodiments of digital information. The physical objects in a tangible interface thus serve as representations as well as controls for the digital information they are associated with. Building on this concept, gesture interaction has the potential to make use of the physical properties of hand-held objects to enhance or change the function- ality of the gestures made. In this paper, we look at the design opportunities that arise at the intersection of gesture and tan- gible interaction. We believe that gesturing while holding physical artifacts opens up a new interaction design space for collaborative digital applications that is largely unexplored. We provide a survey of gesture interaction work as it relates to tangible and touch interaction. Based on this survey, we define the design space of tangible gesture interaction as the use of physical devices for facilitating, supporting, enhancing, or tracking gestures people make for digital interaction pur- poses, and outline the design opportunities in this space. Keywords: Collaboration; Gesture; Hand Movements; Tangible Interaction; Touch–User Interface 1. INTRODUCTION Often without being aware of it, people make gestures with their limbs, sometimes even with their heads or full body. They nod in the direction of the person they are talking about, they gesticulate when angry behind their steering wheels and they use arms and hands when they tell stories enthusiasti- cally. People even gesture when the intended audience cannot see their gestures, for example, when they are on the phone. It is interesting to note that these gestures are more useful than they seem at first instance. Gestures not only facilitate com- munication for both the speaker and listener, they also facil- itate remembering and can lighten cognitive loads. Beyond the use of these everyday gestures, we are also interested in how gesture can be used in interaction with digital systems, known as gestural interaction. In particular, we explore the potential of combining gestures with tangible interaction. Gestural interaction has focused on gesture as a means of communicating with a computer system. Gestures are expres- sive and can convey rich meaning. They can also provide a means of interaction that feels more natural, is easier to learn, and can be less constraining than conventional user interfa- ces. The rich spatial and temporal gesture patterns possible can allow designers to create a myriad of gesturing com- mands, while still leaving room for personalization. Yet cap- turing and analyzing unencumbered gestures in free space remains difficult, posing technical challenges that are still far from overcome. As a result, users often have to hold, wear, or touch artifacts, such as pens, three-dimensional (3-D) sen- sors, gloves, or tables to facilitate the detection of position, orientation, movement, and eventually the intended gesture. However, in most cases the surface touched or the artifact in hand is used as sensing input only and does not add to the meaning of the gesture, or facilitate the gesture in any way. In contrast, tangible interaction makes use of physical artifacts (tangibles) as embodiments of digital information. The physical form as well as the means of interaction is Reprint requests to: Elise van den Hoven, Industrial Design department, Eindhoven University of Technology, Den Dolech 2, HG 2.53, P.O. Box 513, 5600 MB Eindhoven, The Netherlands. E-mail: e.v.d.hoven@tue.nl Artificial Intelligence for Engineering Design, Analysis and Manufacturing (2011), 25, 255–271. # Cambridge University Press 2011 0890-0604/11 $25.00 doi:10.1017/S0890060411000072 255