MAKE YOUR OWN ACCOMPANIMENT: ADAPTING FULL-MIX RECORDINGS TO MATCH SOLO-ONLY USER RECORDINGS TJ Tsai 1 Steven K. Tjoa 2 Meinard M ¨ uller 3 1 Harvey Mudd College, Claremont, CA 2 Galvanize, Inc., San Francisco, CA 3 International Audio Laboratories Erlangen, Erlangen, Germany ttsai@hmc.edu, steve@stevetjoa.com, meinard.mueller@audiolabs-erlangen.de ABSTRACT We explore the task of generating an accompaniment track for a musician playing the solo part of a known piece. Un- like previous work in real-time accompaniment, we focus on generating the accompaniment track in an off-line fash- ion by adapting a full-mix recording (e.g. a professional CD recording or Youtube video) to match the user’s tempo preferences. The input to the system is a set of recorded passages of a solo part played by the user (e.g. solo part in a violin concerto). These recordings are contiguous seg- ments of music where the soloist part is active. Based on this input, the system identifies the corresponding passages within a full-mix recording of the same piece (i.e. contains both solo and accompaniment parts), and these passages are temporally warped to run synchronously to the solo- only recordings. The warped passages can serve as accom- paniment tracks for the user to play along with at a tempo that matches his or her ability or desired interpretation. As the main technical contribution, we introduce a segmen- tal dynamic time warping algorithm that simultaneously solves both the passage identification and alignment prob- lems. We demonstrate the effectiveness of the proposed system on a pilot data set for classical violin. 1. INTRODUCTION Ima Amateur loves her recording of Itzhak Perlman per- forming the Tchaikovsky violin concerto with the Lon- don Symphony Orchestra. She has been learning how to play the first movement herself, and she would love to play along with the recording. Unfortunately, there are parts of the recording that are simply too fast for her to play along with. She finds an app that can slow down the parts of the Perlman recording that are difficult. All she has to do is up- load several solo recordings of herself performing sections of the concerto, along with the original full-mix recording that she would like to play along with. The app analyzes c  TJ Tsai, Steven K. Tjoa, Meinard M¨ uller. Licensed un- der a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: TJ Tsai, Steven K. Tjoa, Meinard M¨ uller. “Make Your Own Accompaniment: Adapting full-mix recordings to match solo- only user recordings”, 18th International Society for Music Information Retrieval Conference, Suzhou, China, 2017. her playing and generates a modified version of the Perl- man recording that runs in sync with her solo recordings. This paper explores the technical feasibility of such an application. In technical terms, the problem is this: given a full-mix recording and an ordered set of solo-only record- ings that each contain a contiguous segment of music where the soloist is active, design a system that can time- scale modify the full-mix recording to run synchronously with the solo recordings. 1 There are three main technical challenges underlying this scenario. The first challenge is to identify the passages in the full-mix recording that correspond to the solo-only recordings. The second challenge is to temporally align the corresponding passages in the full-mix and solo record- ings. The third challenge is to time-scale modify the full- mix recording to follow the calculated alignment without changing the pitch of the original recording. This paper fo- cuses primarily on the first two challenges, and it assesses the technical feasibility of solving these problems on a pi- lot data set. The main technical contribution of this work is to propose a segmental dynamic time warping (DTW) algorithm that simultaneously solves the passage identifi- cation and temporal alignment problems. We will simply adopt an out-the-box approach to solve the third challenge. The idea of generating accompaniment for amateur mu- sicians has been explored in two different directions. On one end of the spectrum, companies have explored fixed accompaniment tracks. Some examples include the popu- lar Aebersold Play-A-Long recordings for jazz improvisa- tion and Music Minus One for classical music. The ben- efit of fixed accompaniment tracks is their simplicity – all you need is a device that can play audio. The drawback of fixed accompaniment tracks is their lack of adaptivity – they do not respond or adapt to the user’s playing in any way. On the other end of the spectrum, academics have explored real-time accompaniment (e.g. see work by Raphael [23] [24] and Cont [3]). These are complex sys- tems that can track a musician’s (or group’s) playing and generate accompaniment in real-time. The benefit of real- time accompaniment is the adaptivity of the system. The drawbacks of real-time accompaniment systems are that they are not easy to use for the general population (e.g. re- quire software packages on a laptop) and may not be very 1 Without changing the pitch, of course! 79