Virtual Actors Living in a Real World Nadia Magnenat Thalmann MIRALab University of Geneva 24 rue du Général-Dufour CH 1211 Geneva 4, Switzerland email: thalmann@cui.unige.ch Daniel Thalmann Computer Graphics Lab Swiss Federal Institute of Technology CH 1015 Lausanne, Switzerland email: thalmann@di.epfl.ch Abstract This paper presents techniques and tools for creating and animating virtual actors in real scenes. Several problems are explained: real objects hidden by virtual actors and virtual actors hidden by real objects, collision detection between the virtual actor and the real environment, correspondence between the real and the virtual cameras, casting shadows of the virtual actors on the real world. Case studies are presented like the virtual actress Marilyn walking with real people on a real street or sitting down on a real chair. 1. Introduction Computer-generated images are more and more used for advertising, simulation and special effects. Virtual or synthetic actors have been also created in the last few years. More recently, image processing has become popular for digital warping [1] or morphing [2] and to encrust new real actors into existing films, like in the Forrest Gump movie. Another popular approach especially for advertising combines computer-generated images into real images. However, very few experiences have been made to animate virtual actors in the real world. This paper describes the techniques to create scenes involving virtual actors in a real environment. Several cases are presented and the most important problems are discussed. To combine computer-generated images into real ones, there are two ways: • taking into account the real world during the generation of the images by the computer • composing the images without taking into account the real world In the second case, the method only consists in composing the images. Two kinds of processes are possible: • the analog process: a chromatic threshold is determined on the first source to cut and the value of the second source is substituted. This is mainly the blue box approach: the foreground images have a saturated blue background; this blue color is detected and replaced by the background image. • the digital process: each pixel of the image is coded and can be tested and replaced. A numerical threshold, masks or transparent background may be used. The more powerful approach is of course an image synthesis system working on digital images. Such a system should provide facilities for editing, composing, animating, deforming and modifying sequences of digital images. It also requires bi-directional connections with broadcast video systems, high quality images, large storage and fast access to images and a high-level user interface. For creating scenes involving virtual actors in the real world, we should really take into account the real world during the generation of the images by the computer. For example, consider a virtual actor passing behind a real tree: for some images of the actor, part of the body should be hidden. For more realism, the shadow of the actor should be cast on the real floor. This means that the computer- generated images are dependent on the real world. One way of solving these problems is to create virtual objects similar to the real ones and a virtual camera corresponding to the real camera which is used to shoot the real scene. However, this correspondence is generally hard to establish.