Kontrol: Hand Gesture Recognition for Music and Dance Interaction Kameron Christopher Jingyin He Raakhi Sinha Kapur Ajay Kapur California Institute of the Arts 24700 McBean Parkway Valencia, CA 91355, USA ABSTRACT This paper describes Kontrol, a new hand interface that extends the intuitive control of electronic music to traditional instrumentalist and dancers. The goal of the authors has been to provide users with a device that is capable of detecting the highly intricate and expressive gestures of the master performer, in order for that information to be interpreted and used for control of electronic music. This paper discusses related devices, the architecture of Kontrol, its potential as a gesture recognition device, and several performance applications. Keywords Hand controller, computational ethnomusicology, dance interface, conducting interface, Wekinator, wearable sensors 1. INTRODUCTION In today’s digital age, the traditional performing artist (be it instrumentalist or dancer) is faced with an ever-growing interest in extending their media. There is a long history of using micro controllers and sensors to create interfaces that interact in ways that appear more embodied. The hands are an important form of non-verbal communication in which humans rely for interaction. They help relay emotions, emphasize concepts, and often help in explanation. The hands are the basis of sign language, which relies heavily on hand movements and gesture for everyday communication and interaction with the world. The hand movements of a skilled musician can provide very useful information on what the musician is playing, how they are playing it, and possibly what they intend to do next. Musicians develop complex muscle memories in order to execute various performative tasks with the greatest effect and efficiency. Many of these movements have become intuitive or have developed naturally and the musicians themselves would not be able to describe them. Building an interface that is capable of detecting and making use of this information in musical ways was the goal of the authors. Many in the field of music have sought the creation of devices for musical control that function in a way that is more expressive and musical in a traditional sense. Tod Machover was an early innovator in these musical technologies with his idea of hyper-instruments [1]. In 1984, Michael Waisvisz brought popularity to the idea of hand based musical controllers with his introduction of The Hands [2]. The hands were constructed of numerous sensors attached to the user’s arms, hands, and pants that allowed for the control of music. Since then, many gesture-based designs for musical control have been created [3]. Simultaneously, numerous hand-based devices were being developed for non-musical uses, such as communication, computing, and gaming [4]. Mattel’s Power Glove was the first attempt to create a commercially available glove input gaming device and was being developed at the same time as the DataGlove [5], [6]. Stanford produced the Thumbcode as a digital sign language device in 1998 [7]. The development of hand-based devices has continued to grow in these fields[4]. Additionally, methods for gesture recognition in musical application continue to develop [3]. Kontrol aims to account for the traditional manner in which artists perform their art, and not interfere with their ability to do so. Kontrol is a hand interface built to extract the intricate expressivity of performers of various performative backgrounds, including dancers and instrumentalist. The goal of Kontrol is to provide a tool for the intuitive interpretation of gestures that interfaces with the performer in conjunction with the performative task. The use of triple-axis accelerometers on each finger and the hand provide high-resolution detail about a performer’s actions and an ergonomically lightweight and non- intrusive design for the performer. Kontrol builds upon these findings as a tool for intuitive expression in digital music. The design and architecture of Kontrol is described in section 2. Section 3 discusses the prospects of using gesture recognition with Kontrol. Several projects in which Kontrol has been used are described in section 4. Section 5 provides a conclusion. 2. Design and Architecture Kontrol was designed to be lightweight and unobtrusive. Six triple-axis accelerometer are especially customized to sit on the left hand: 5 on the fingers and 1 on top of the hand (Figure 1). The sensors are carefully calibrated and designed to lay out ergonomically on the left-hand and are connect through a multiplexer to an Arduino Fio, as seen in Figure 2. The use of accelerometers allow for the recognition and parameterization of the subtlest of movements. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NIME’13, May 27-30, 2013, KAIST, Daejeon, Korea. Copyright remains with the author(s). 