Wiiolin: a virtual instrument using the Wii remote Jace Miller & Tracy Hammond Sketch Recognition Lab Department of Computer Science & Engineering Texas A&M University {jacemiller, hammond}@tamu.edu ABSTRACT The console gaming industry is experiencing a revolution in terms of user control, and a large part to Nintendo’s introduction of the Wii remote. The online open source development community has embraced the Wii remote, integrating the inexpensive technology into numerous applications. Some of the more interesting applications demonstrate how the remote hardware can be leveraged for nonstandard uses. In this paper we describe a new way of interacting with the Wii remote and sensor bar to produce music. The Wiiolin is a virtual instrument which can mimic a violin or cello. Sensor bar motion relative to the Wii remote and button presses are analyzed in real-time to generate notes. Our design is novel in that it involves the remote’s infrared camera and sensor bar as an integral part of music production, allowing users to change notes by simply altering the angle of their wrist, and henceforth, bow. The Wiiolin introduces a more realistic way of instrument interaction than other attempts that rely on button presses and accelerometer data alone. Keywords Wii remote, virtual instrument, violin, cello, motion recognition, human computer interaction, gesture recognition. 1. INTRODUCTION Nintendo's Wii has rejuvenated the gaming industry, due to the public’s response to the controller. The Wii remote, or Wiimote as some have informally termed it, comes equipped with accelerometer sensors and an infrared (IR) camera. Leveraging this affordable technology opens up the door for new applications. This work describes a new way of interacting with the Wii remote and sensor bar to simulate playing a virtual string instrument. This paper describes the new digital musical device, the Wiiolin. The Wiiolin is a digital stringed instrument based on the Wii technology, where the Wii remote functions as the neck, and the IR sensor bar functions as the bow. The Wiiolin automatically switches between a violin and a cello based on the orientation in which the player is holding the virtual instrument. (See Figure 1.) The angle of the player’s wrist and bow defines which string the player is playing, whereas the buttons on the Wii remote define which note on the string is being played. Figure 1: The orientation of the Wii remote determines whether the Wiiolin performs as a violin or cello. 2. RELATED WORK A new genre of video games based on music production through instrument simulation was created with the release of the first Guitar Hero game developed by Harmonix in 2005. Since then, there have been other successful ventures in the field of video game instrument simulation, such as Rock Band and Donkey Konga. While these titles have achieved a level of popularity, they rely on instrument simulation through additional peripheral devices. Requiring consumers to purchase extra controllers in order to experience a game limits the size of the user base. Additionally, they are quite simplistic, removing many of the natural affordances that exist in playing the real instrument. One solution to this problem is to use the standard game controller as the instrument. The most notable attempt at using the Wii remote in an instrument simulation game has been Wii Music [6]. Wii Music allows the tempo and pitch of a song to be altered using information gathered from button presses and the controller’s accelerometer. Johnny Lee's Head Tracking for Desktop Displays project uses the sensor bar in a nonstandard, "reverse" way – holding the Wiimote fixed and primarily moving the sensor bar [5]. The Wiiolin combines the ideas of instrument simulation and using the sensor bar in a nonstandard way to create a new quintephone – an electronic instrument that generates sound by optical or mechanical computing, specifically in case, using physical interaction to guide intelligent musical feedback. It provides a higher accuracy of simulation than existing systems, in that it allows users a greater amount of fluid expression and musical freedom than existing systems through the ability to change both notes and instruments through direct physical Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NIME2010, 15-18th June 2010, Sydney, Australia Copyright remains with the author(s). Proceedings of the 2010 Conference on New Interfaces for Musical Expression (NIME 2010), Sydney, Australia 497