International Journal of Computer Vision 36(2), 101–130 (2000) c  2000 Kluwer Academic Publishers. Manufactured in The Netherlands. Systems and Experiment Paper: Construction of Panoramic Image Mosaics with Global and Local Alignment HEUNG-YEUNG SHUM AND RICHARD SZELISKI Microsoft Research, One Microsoft Way, Redmond, WA 98052, USA hshum@microsoft.com szeliski@microsoft.com Received October 1998; Revised March 1999; Accepted August 1999 Abstract. This paper presents a complete system for constructing panoramic image mosaics from sequences of images. Our mosaic representation associates a transformation matrix with each input image, rather than explicitly projecting all of the images onto a common surface (e.g., a cylinder). In particular, to construct a full view panorama, we introduce a rotational mosaic representation that associates a rotation matrix (and optionally a focal length) with each input image. A patch-based alignment algorithm is developed to quickly align two images given motion models. Techniques for estimating and refining camera focal lengths are also presented. In order to reduce accumulated registration errors, we apply global alignment (block adjustment) to the whole sequence of images, which results in an optimally registered image mosaic. To compensate for small amounts of motion parallax introduced by translations of the camera and other unmodeled distortions, we use a local alignment (deghosting) technique which warps each image based on the results of pairwise local image registrations. By combining both global and local alignment, we significantly improve the quality of our image mosaics, thereby enabling the creation of full view panoramic mosaics with hand-held cameras. We also present an inverse texture mapping algorithm for efficiently extracting environment maps from our panoramic image mosaics. By mapping the mosaic onto an arbitrary texture-mapped polyhedron surrounding the origin, we can explore the virtual environment using standard 3D graphics viewers and hardware without requiring special-purpose players. Keywords: image mosaics, virtual environment modeling, panoramas, parametric motion estimation, global alignment, local alignment 1. Introduction The automatic construction of large, high-resolution image mosaics is an active area of research in the fields of photogrammetry, computer vision, image process- ing, and computer graphics. Image mosaics can be used for many different applications (Kumar et al., 1995; Irani et al., 1995a). The most traditional ap- plication is the construction of large aerial and satel- lite photographs from collections of images (Moffitt and Mikhail, 1980). More recent applications include scene stabilization and change detection (Hansen et al., 1994), video compression (Irani et al., 1995a, 1995b; Lee et al., 1997) and video indexing (Sawhney and Ayer, 1996), increasing the field of view (Heckbert, 1989; Mann and Picard, 1994; Szeliski, 1994) and resolution (Irani and Peleg, 1991; Chiang and Boult, 1996) of a camera, and even simple photo editing (Burt and Adelson, 1983). A particularly popular application is the emulation of traditional film-based panoramic photography (Malde, 1983) with digital panoramic mosaics, for applications such as the construction of virtual environments (McMillan and Bishop, 1995; Szeliski, 1996) and virtual travel (Chen, 1995).