International Journal of Computer Vision 58(3), 237–247, 2004 c  2004 Kluwer Academic Publishers. Manufactured in The Netherlands. Constructing Virtual Cities by Using Panoramic Images KATSUSHI IKEUCHI, MASAO SAKAUCHI, HIROSHI KAWASAKI AND IMARI SATO Institute of Industrial Science, The University of Tokyo, 6-4-1 Komaba, Meguro-ku, Tokyo 153-8505, Japan Received January 7, 2002; Revised May 19, 2003; Accepted October 14, 2003 Abstract. Simultaneously acquired omni-directional images contain rays of 360 degree viewing directions. To take advantage of this unique characteristic, we have been developing several methods for constructing virtual cities. In this paper, we first describe a system to generate the appearance of a virtual city; the system, which is based on image-based rendering (IBR) techniques, utilizes the characteristics of omni-directional images to reduce the number of samplings required to construct such IBR images. We then describe a method to add geometric information to the IBR images; this method is based on the analysis of a sequence of omni-directional images. Then, we describe a method to seamlessly superimpose a new building model onto a previously created virtual city image; the method enables us to estimate illumination distributions by using an omni-directional camera. Finally, to demonstrate the methods’ effectiveness, we describe how we implemented and applied them to urban scenes. Keywords: omni directional camera, VR, spatio-temporal analysis, ITS 1. Introduction The virtual city, a virtual reality system to display an urban scene, is one of the most promising tools for car navigational aids, shopping guides, and city planning. The construction of a virtual city consists of modeling and rendering stages. In the modeling stage, geomet- ric and photometric properties of buildings in an urban area are acquired in digital form. Then, using the ac- quired photometric and geometric properties, the ren- dering process generates the urban area’s appearance from desired viewing directions under desired illumi- nation conditions. Currently, such a modeling process is done manually; as a result, the cost of the modeling process is prohibitive. In this paper, we describe how, in an effort to reduce that high cost, we developed auto- matic techniques for acquiring the properties of urban scenes. There have been many research projects for mod- eling large scale scenes such as those found in urban cities. These projects can be largely devided into two major techniques: the model-based technique and the image-based technique. So far, the model based method is mainly applied for city modeling purposes. The model-based method for virtual cities represents those cities with geomet- ric shape and photometric texture models. Usually, such geometric models of urban scenes are constructed manually. For planning purposes, representing non- existent urban scenes by the use of manual processes to construct 3D geometric models from drawings is unavoidable. For some applications, e.g., navigational purposes, the task is to convert existing urban scenes into geometric models; in such cases, it is desirable to automatically measure actual buildings in order to obtain geometric models for virtual cities. Zhao and Shibasaki (1998) and Fruh and Zakhor (2001) have proposed an efficient method for acquiring and con- structing textured 3D models by using a car-mounted range sensor. Teller et al. (2001) and Antone and Teller (2001) have proposed a unique technique to cap- ture and reconstruct urban areas by using an omni- directional camera. Some researchers believe that 3D geometric models constructed from aerial views can handle relatively simple shapes such as rectangles or cylinders.