The Haunted Book Camille Scherrer * ECAL University of Art and Design Lausanne Julien Pilet † CVLab ´ Ecole Polytechnique F´ ed´ erale de Lausanne Pascal Fua ‡ CVLab ´ Ecole Polytechnique F´ ed´ erale de Lausanne Vincent Lepetit § CVLab ´ Ecole Polytechnique F´ ed´ erale de Lausanne Figure 1: Two examples of animated illustrations from the Haunted Book, and the Haunted Book setup. A video demonstrating the system is available at http://chipchip.ch/interaction.html ABSTRACT This paper describes an artwork that relies on recent Computer Vi- sion and Augmented Reality techniques to animate the illustrations of a poetry book. Because we don’t need markers, we can achieve seamless integration of real and virtual elements to create the de- sired atmosphere. The visualization is done on a computer screen to avoid cumbersome Head-Mounted Displays. The camera is hid- den into a desk lamp for easing even more the spectator immersion. 1 I NTRODUCTION Magic books are not new in Augmented Reality. Since the pio- neering work of Billinghurst and Kato [2], they have been used to enhance reading experience [3], visualize products in selling cata- logues [2], tell stories [5], and teach. However, for robust registra- tion, current applications rely on visual markers, which makes the illusion less convincing. For the Haunted Book we describe in this paper, we used recent Computer Vision techniques [4] that provide the same robustness as ARToolKit while relying on natural features instead of visual markers. As a result, there is no need to engineer the book. It gives the spectator a more natural feeling, and let the artist give free rein to his imagination. More specifically, we add animated elements to the illustrations of a poetry book. These elements, such as the raven in Fig. 2, match the illustration style and increase their impact. In the following, we describe our work, give reactions of pro- fessional artists, and briefly describe the technical part behind the Haunted Book. 2 THE HAUNTED BOOK The artist designed and produced the Haunted Book, inspired by old poetry books. The integration of very recent technologies into this * e-mail: camille.scherrer@ecal.ch † e-mail: julien.pilet@epfl.ch ‡ e-mail: pascal.fua@epfl.ch § e-mail: vincent.lepetit@epfl.ch old and dusty universe makes it unusual and particularly interesting. The Haunted Book is based on a poem written by Thomas Hood, The Haunted House. As the teller in the poem walking through the haunted house, the reader ”walks” through the book and discovers hidden creatures. The artist’s interpretations of these hidden creatures appear as a skeleton’s arm grabbing out of a letterbox, flying fish jumping out of an old cupboard, or loads of ugly insects running down a sofa. The animated engravings create a subtle and adapted way to enhance the illustrations by staying in the universe of the poem. The Augmented Reality technology used for the project allows this subtlety by staying discreet. 3 AUGMENTED REALITY TECHNOLOGY The book registration is completely based on natural features, using the method described in [4] for which the code is available at [1]. The registration system relies on one single reference image for each page to augment, for which the artist designed animated layers using AfterEffects. At run-time, the system estimates the homogra- phy registering the captured frame and the correct reference image, and applies it to the animated layer for augmentation. Since only one page at a time can be visible, the software keeps looking for the same page as long as it was detected successfully on the last frame. Otherwise, it looks for the next page. This keeps the delay short: In the worst case, the system would try all ten pages before finding the correct one, which takes less than a second and remains interactive. Drawing the augmentation layer in 2–D, on top of a reference image, makes easy for the artist to create interaction between the real book and the virtual creature. For example, the candlestick printed on the book occludes the virtual bird’s tail of Fig. 2. This effect is simply achieved by adding a static transparent layer in Af- terEffects, making transparent the corresponding areas of the aug- mentation layer. 4 REACTIONS To evaluate our artwork, we asked four professional artists the fol- lowing questions: • What is the artistic value of our work ? • What brings the Augmented Reality part of the work ?