IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 2, FEBRUARY 2005 241 Image Registration for Image-Based Rendering Angus M. K. Siu and Rynson W. H. Lau Abstract—Image-based rendering (IBR) has received much attention in recent years for its ability to synthesize photo-real- istic novel views. To support translational motion, existing IBR methods either require a large amount of reference images or assume that some geometric information is available. However, rendering with a large amount of images is very expensive in terms of image acquisition, data storage, and memory costs. As IBR accepts various kinds of geometric proxy, we may use image registration techniques, such as stereo matching and structure and motion recognition, to obtain geometric information to help reduce the number of images required. Unfortunately, existing image registration techniques only support a small search range and require closely sampled reference images. This results in a high spatial sampling rate, making IBR impractical for use in scalable walkthrough environments.Our primary objective of this project is to develop an image registration technique that would recover the geometric proxy for IBR while, at the same time, reducing the number of reference images required. In this paper, we analyze the roles and requirements of an image registration technique for reducing the spatial sampling rate. Based on these requirements, we present a novel image registration technique to automatically recover the geometric proxy from reference images. With the distinguishing feature of supporting a large search range, the new method can accurately identify correspondences even though the reference images may only be sparsely sampled. This can significantly reduce the acquisition effort, the model size, and the memory cost. Index Terms—Image-based rendering (IBR), image matching, image registration, object recognition. I. INTRODUCTION A S IMAGE-based rendering (IBR) can generate photo-re- alistic novel views from reference images, it has received much attention in recent years. It is generally regarded as a pow- erful alternative to traditional geometry-based rendering in com- puter graphics. The focus of IBR studies has been extended from the earlier restricted view synthesis, as in QuickTime VR [9], to the recent scalable walkthrough environments, as in Plenoptic Stitching [3]. In order to support translational movement, most existing IBR methods may use a large number of reference images to capture all potential views. This approach, however, does not Manuscript received March 31, 2003; revised March 3, 2004. This work was supported in part by a CERG grant from the Research Grants of Hong Kong (RGC Reference Number: CityU 1308/03E) and in part by a SRG grant from City University of Hong Kong (Project Number: 7001465). The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Reiner Eschbach. A. M. K. Siu is with the Department of Computer Science, City University of Hong Kong, Hong Kong, SAR (e-mail: angus@cs.cityu.edu.hk). R. W. H. Lau is with the Department of Computer Engineering and Infor- mation Technology, City University of Hong Kong, HongKong, SAR (e-mail: Rynson.Lau@cityu.edu.hk). Digital Object Identifier 10.1109/TIP.2004.840690 scale well because the number of images required, i.e., the spa- tial sampling rate, is a polynomial function to the size of the scene. For example, although feature tracking and optimization algorithms are used in [3], the sampling rate is as high as 1 600 images for a 49-m capturing path (i.e., one image for every 3 cm). Our objective of this project is to develop a method that can effectively reduce the spatial sampling rate to a state that it would be truly practical to model large scenes. In [8], the minimum sampling curve shows that the spatial sampling rate can be effectively reduced if geometric informa- tion is available. A method that is often used to automatically extract intrinsic geometric information from reference images is image registration. It is basically an image processing oper- ation to identify correspondences among a number of images. If the spatial sampling rate is be reduced, the image registration process will need to search a larger range for correspondences. This, however, introduces several challenges. First, the point matching error increases rapidly with the increase in the search range. It is necessary to control the error through some means. Second, when the reference images are taken further apart from each other, the change in image content due to perspective pro- jection becomes significant and can no longer be approximated by simple translational movement as in block-based video com- pression methods such as MPEG. More accurate methods are needed to estimate the transformation of image contents. Third, a large search range will lead to a dramatic increase in computa- tional cost. Depending on the matching algorithm, the matching time may be , where is the search range. In video com- pression, the search range is only around 16 pixels. However, in IBR, it can be larger than 100 pixels. This means that existing image registration techniques are not suitable for IBR applica- tions. In this paper, we present a novel image registration method with a large search range. The outline of this paper is as follows. In Section II, we review existing IBR methods and the image registration techniques used. In Section III, we discuss the tasks involved in the image registration process in IBR and how these tasks are related to the spatial sampling rate. In Section IV, we present our image registration method in detail. In Section V, we show and evaluate some experimental results of the proposed method. Finally, in Section VI, we briefly conclude the paper and discuss possible future work. II. LITERATURE REVIEW Many IBR methods have been proposed to construct different types of Plenoptic functions [1], with image registration tech- niques being used in various stages of the method, such as image compression, disparity map computation, and feature tracking. In this section, we review representative IBR methods with an emphasis on the roles of the image registration process. 1057-7149/$20.00 © 2005 IEEE