“Virtual Dubbing”: a new concept for the regeneration of voice and acoustic ambience in cinematographic sound tracks Diego L. Gonzalez, Davide Bonsi, Domenico Stanzial Musical and Architectural Acoustics Laboratory, I-30124, Isola di San Giorgio Maggiore, Venice, Italy, {diego.gonzalez, davide.bonsi, domenico.stanzial}@cini.ve.cnr.it Cinematographic archive material poses strong research and technological challenges for its preservation as a cultural heritage and also for its utilization in new productions characterized by high quality post- production environments. The UE research program IST RACINE-S is aimed to the development of innovative hardware and software for the high quality reconstruction of highly damaged or even missing pieces of archive cinematographic material. The Acoustics Laboratory of the FSSG-CNR at Venice contributes in particular to the development of innovative algorithms for the audio reconstruction of damaged films. In this framework, a method for dialogues and acoustic ambience reconstruction called Virtual Dubbing has been proposed. Virtual Dubbing was conceived for converting the voice of a professional dubbing speaker into a target missing voice. The information about the missing voice is taken from sections of the film previous or following the damaged part. State of the art LPC voice transformation algorithms have been developed at a research level and further implemented in a DSP hardware platform. Moreover, a non-linear perceptive model working in parallel with the LPC method has been developed. This non-linear model sorts high level perceptive parameters which can be used for the analysis of the voice signals and for high level control of the LPC synthesis step. On the other hand, acoustical ambience reconstruction has been based on a new intensimetric approach which allows a true physical characterization of the sound field. 1 Introduction The restoring of archive films for preservation and reutilization in new productions poses strong research and technological challenges related mainly to the high quality standards required by the cinematographic industry. In the extreme case (not very unusual) some film segments to be restored are highly damaged or even missing. So, techniques for the regeneration of that highly damaged parts need to be developed. The UE research program IST RACINE-S is aimed to the development of innovative hardware and software for this scope: high quality reconstruction of images and sound on highly damaged or missing pieces of archive films. The Acoustics Laboratory of the FSSG-CNR at Venice contributes in particular to the development of innovative algorithms for audio reconstruction regarding dialogues and acoustic ambience. The principal development in this field has been the development of a production/post-production environment called Virtual Dubbing. Virtual Dubbing can be added to a traditional post-production environment maintaining the skills, procedures and facilities and allowing for a standard treatment of some associated problems like labial synchronization, reconstruction from script, etc. With the aid of the tools of Virtual Dubbing will be possible to reconstruct a target missing voice by appropriate transformation of the voice of a professional dubbing speaker. Furthermore, a correct and natural acoustical ambience can be given to the reconstructed sound tracks by means of an innovative procedure based on the intensimetric characterization of the acoustic field which we have called Acoustic Quadraphony. Both approaches are rooted in state-of-the-art research in the fields of voice transformation, non-linear dynamics and acoustic intensimetry. Voice reconstruction is implemented through a core of LPC-GMM (Linear Prediction Coding and Gaussian Mixture Models) voice transformation algorithms and a set of innovative non-linear algorithms for robust voice analysis. Acoustic quadraphony, instead, takes profit of the new generation of hot-wire intensimetric probes which allow direct measurement of the 4 components of the acoustic field: the pressure and the 3 components of the air velocity. In this paper are presented general aspects regarding the Virtual Dubbing procedure together with the non- linear approach for robust voice characterization. The details about the LPC-GMM algorithms for voice transformation and the use of intensimetric techniques for acoustic ambience reconstruction are given in two companion papers [1,2]. 2 Virtual Dubbing process In Figure 1 a schematic description of the “Virtual Dubbing” process is presented. 305