Interactive Museum Guide Herbert Bay, Beat Fasel and Luc Van Gool Computer Vision Laboratory (BIWI) ETH Zurich Sternwartstr. 7 8092 Zurich, Switzerland {bay, bfasel, vangool}@vision.ee.ethz.ch ABSTRACT In this paper, we describe the prototype of an interactive museum guide. It runs on a tablet PC that features a touchscreen, a webcam and a Bluetooth receiver. This guide recognises objects on display in museums based on images of the latter which are taken directly by the visitor. Furthermore, the computer can determine the visitor’s lo- cation by receiving signals emitted from Bluetooth senders in the museum, so called BTnodes. This information is used to reduce the search space for the extraction of rele- vant objects. Hence, the recognition accuracy is increased and the search time reduced. Moreover, this information can be used to indicate the user’s current location in the museum. The prototype has been demonstrated to visitors of the Swiss National Museum in Zurich. Categories and Subject Descriptors H.5.2 [Information Interfaces and Presentation]: User Interfaces - Input devices and strategies; I.2.10 [Artificial Intelligence]: Vision and Scene Understanding; I.4.8 [Image Processing and Computer Vision]: Scene Analysis - Object recognition Keywords Object recognition, Interactive museum guide, Bluetooth 1. INTRODUCTION Many museums present their exhibits in a rather passive and non-engaging way. The visitor has to scan a booklet in order to find some general information about the object. However, searching for information about object after ob- ject is quite tedious and the information found does not always cover the visitor’s specific interests. One possibil- ity of making exhibitions more attractive to the visitor is to improve their interaction with the guide. In this pa- per, we present an interactive museum guide which is able to automatically find and retrieve information about the objects of interest on a laptop-like device. Moreover, it provides further links and references allowing the visitor to browse comfortably on the Internet for an even broader description of the object. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Smart Environments and Their Applications to Cultural Heritage. UbiComp 2005, September 11-14, 2005, Tokyo, Japan. 1.1 Related Work Recently, several approaches and methods have been pro- posed that allow visitors to interact with an automatic guide in a museum. Kusunoki et al. [7] proposed a system for children that uses a sensing board which can rapidly recognise types and locations of multiple objects. It cre- ates an immersive environment by giving audio-visual feed- back to the kids. Other approaches are robots that guide users through museums [4, 10]. However, such robots are difficult to adapt to different environments, and they are not appropriate for individual use. An interesting ap- proach using hand-held devices, like mobile phones, was proposed by [5], but their recognition technique is limited to constant illumination. 1.2 Our Approach We present an interactive, image-based museum guide that is invariant to changes in lighting, viewpoint, scale (zoom) and rotation. Our method was implemented on a tablet PC using a conventional USB webcam for image acquisi- tion, see Figure 1. This hand-held device allows the visitor to simply take a picture of an object of interest from any position and is provided, almost immediately, with a de- tailed description of it. Figure 1: Tablet PC with the USB webcam fixed on the screen. The interface of the object recognition software is operated via a touchscreen. Furthermore, this device can be extended to display loca- tion dependent information on a map, such as the closest emergency exits, the toilets or the direction to the next