A MULTI-TIMELINES SCHEDULER AND A REHEARSAL FUNCTION FOR IMPROVING USERS’ SENSATION OF ORCHESTRAL CONDUCTING WITH A CONDUCTING SYSTEM Takashi Baba Kwansei Gakuin University takashi-baba@kwansei.ac.jp Mitsuyo Hashida Soai University hashida@soai.ac.jp Haruhiro Katayose Kwansei Gakuin University katayose@kwansei.ac.jp ABSTRACT The VirtualPhilharmony (VP) system conveys the sensa- tion of conducting an orchestra to a user (a conductor). VPs performances are created through interaction between the conductor and orchestra, exactly like real performance. “Concertmaster function has been already implemented by incorporating the heuristics of conducting an orchestra. A precisely predictive scheduler and dynamical template have been designed based on analyses of actual recordings. We especially focused on two more problems to emulate a more real orchestral performance; one was that each note in the template was controlled by single-timeline scheduler; the other was that the interaction and communication between the conductor and the orchestra in repeated practices were not simulated. We implemented “Multi-timelines sched- uler and “Rehearsal function to resolve these problems. 1. INTRODUCTION Musical conducting, which sometimes involves directing an orchestra with as many as 100 members, has been a highly regarded profession since its professional status was established in the 19th century. A conductors manner of musical expression is quite different from that of an in- strumentalist or a vocalist. Inspired by this unique man- ner of creative expression, composers and engineers in the field of computer science started to develop conducting in- terfaces in the late 1980s. The existing conducting sys- tems have been designed as a means of musical expression through computer-generated music, but they had a prob- lem that is a shortage of viewpoints of the musicality of the performers. We have been developing a conducting system called “Vir- tualPhilharmony (VP)” that is intended to convey a realis- tic sensation of conducting an orchestra. Actual perfor- mances are created through interaction between the con- ductor (the user) and orchestra. Our intent is to simu- late this interaction by incorporating into VP the heuris- tics of conducting an orchestra. We have already imple- mented “Concertmaster function, in which a precisely pre- dictive scheduler and dynamical template have been de- Copyright: c 2012 Takashi Baba et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 Unported License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. signed based on analyses of actual recordings. There were two problems in Concertmaster function; one was that each note in the template was controlled by single-timeline sched- uler; the other was that the interaction and communication between the conductor and the orchestra in repeated prac- tices like rehearsals were not simulated. We implemented “Multi-timelines scheduler” and “Rehearsal function” to resolve these two problems. The aim of the former is to control expressiveness for each voice part or musical in- strument in real-time. The aim of the latter is to revise the template in accordance with the user’s musical intention or habit during the repeated rehearsals. 2. RELATED WORK The mechanism of a conducting system performs four ba- sic actions: 1) extract the beat points from the users arm and hand motions by using gesture sensors; 2) calculate the tempi from the beat points; 3) predict the tempo of the next beat by using the previous tempi; and 4) schedule the playing of notes on the basis of the predicted tempo. One of the earliest related systems is the “Radio Baton” [1] developed by Mathews in 1987. The user controls a MIDI 1 by holding two batons over a sensor plate. Al- though this system was not intended for simulated con- ducting, the idea of tempo control based on user-given beats is similar to conducting. Morita et al. simulated some directions to each musical instrument using user’s left arm in conducting [2]. The po- sition of the left arm was recognized by a camera, dynam- ics of the instruments pointed by the left arm were sepa- rately controlled. “iFP” [3] by Okudaira et al. is a performance interface for making keyboard strokes or hand-shaking gestures. It extracts beat information and feeds it into an expressive performance template, where the expression data extracted from virtuoso piano performances are described. The tem- plate file describes deviations regarding dynamics, tempo transitions, and delicate controls within a beat of each note. By borrowing such expressiveness from real performances, iFP enables a user to add delicate nuances within a beat. It also provides parameters to balance the intensions of the user and the template. VP is based on iFP. “Personal Orchestra (now “The Virtual Conductor”)” [4] by Borchers et al. is an audio-based conducting system. It uses a phase vocoder to stretch the audio signal. A big 1 Musical Instrument Digital Interface