Rhythm’n’Shoes: a wearable foot tapping interface with audio-tactile feedback Stefano Papetti Dept. of Computer Science, University of Verona Strada Le Grazie, 15 37134 Verona, Italy stefano.papetti@univr.it Marco Civolani Dept. of Mathematics and Computer Science, University of Udine Via delle Scienze, 206 33100 Udine, Italy marco.civolani@uniud.it Federico Fontana Dept. of Mathematics and Computer Science, University of Udine Via delle Scienze, 206 33100 Udine, Italy federico.fontana@uniud.it ABSTRACT A wearable shoe-based interface is presented, which enables users to play percussive virtual instruments by tapping their feet. The mobile controller consists of a pair of sandals equipped with four force sensors and four actuators afford- ing audio-tactile feedback. The sensors provide data via wireless transmission to a host computer, where they are processed and mapped to a physically-based sound synthesis engine. Since the system provides OSC and MIDI compati- bility, alternative electronic instruments can be used as well. The audio signals are then sent back wirelessly to the audio- tactile exciters embedded in the sandals’ sole, and option- ally to headphones and external loudspeakers. The round- trip wireless communication only introduces very small la- tency, thus guaranteeing the coherence and unity of the mul- timodal percept and allowing tight timing while playing. Keywords interface, footwear, wireless, wearable, mobile, foot tapping, sensors, multimodal feedback, audio, tactile, embodied 1. INTRODUCTION In many cultures, both music and dance performers make use of foot tapping, from folk fiddlers and street buskers to flamenco and tap dancers. For instance, a fiddler stomping on a pub’s wooden floor can cheer on the audience mean- while supporting his or her own playing by adding a simple percussion part; buskers often include foot drums in their setup to add even complex percussion parts to their guitar playing. Moreover, traditional musical genres exist (e.g., from Ireland, Canada and USA) where players make exten- sive use of foot percussions (podo-rhythm) as main accom- paniment. As for dance, foot tapping can have both an expressive and rhythmic function, to the extent that some dance genres are centered on both the musical and gestural performance produced by the dancer’s feet. On the other hand, in everyday life many musicians and music enthusiasts alike find themselves“tapping songs”with their fingers, hands and feet. In particular, the feet are usually available for tapping even while working at one’s desk, or reading a newspaper while traveling on a train (co- Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NIME’11, 30 May–1 June 2011, Oslo, Norway. Copyright remains with the author(s). workers and other passengers permitting). Such tapping may represent the song’s main melody, rhythm, or even accurately simulate its percussions part. The gesture of playing rhythms with the feet offers spon- taneity and expressivity, at the same time enabling an em- bodied experience. In parallel, as musical skills grow, foot tapping generally becomes a fundamental resource for mu- sicians to develop a good timing, especially in the case of drummers and percussionists. Taking inspiration from these observations, and starting from a prototype shoe-based interface we had previously realized for interactive walking purposes [19], we decided to implement a wearable controller for foot tapping. We have then developed a mobile shoe-based musical interface called “Rhythm’n’Shoes”, that allows users to control percussive sounds generated by potentially any electronic instruments: the system is natively interfaced with a physically-based sound synthesis engine, however it is suitable for use with any instrument compatible with MIDI or OSC protocols. The interface provides the user with audio-tactile feedback through exciters embedded in the sole. A similarly immediate approach to playing rhythms, that avoid the use of virtual drum interfaces but instead takes inspiration from the common experience of hitting the chest or thighs with the hands, is described in [3]. The interface consists of a pair of gloves embedding piezo microphones, that are used as sensing devices. Several recent studies take into account novel percussion instruments [2], interfaces for percussion tasks [11], and re- lated technological solutions. Some examples are: a physi- cal modeling drum synthesizer and interface [8]; a wireless drumstick interface [22]. As for commercial products, a renowned electronic per- cussion instrument is the Korg Wavedrum, the Roland V- Drum and Clavia ddrum are two successful virtual drum kits, while many are the examples of electronic drum ma- chines, samplers or MIDI controllers offering velocity sen- sitive trigger pads to be played with the fingers (e.g. the AKAI MPC series). Various researches exist which consider the use of haptic feedback in digital musical interfaces and instruments [7, 18, 17]. With regard to interfaces for percussion tasks, hap- tic feedback is exploited in [14] to foster rhythm skills while playing a virtual drum kit, while in [5] a physically intu- itive haptic drumstick is discussed, where the haptic dynam- ics are altered to assist performers in playing single-handed drum rolls. 2. INTERFACE DESIGN This section describes the design of the interface in a bottom- up fashion, from the hardware implementation to the soft-