Fast construction of dynamic and multi-resolution 3608 panoramas from video sequences Zhigang Zhu a, * , Guangyou Xu b , Edward M. Riseman c , Allen R. Hanson c a Department of Computer Science, City College of New York, Convent Avenue and 138th Street, NY 10031, USA b Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China c Department of Computer Science, University of Massachusetts at Amherst, MA 01003, USA Received 11 February 2004; received in revised form 14 September 2005; accepted 15 September 2005 Abstract This paper presents a unified approach to automatically build dynamic and multi-resolution 3608 panoramic (DMP) representations from image sequences captured by hand-held cameras mainly undertaking rotation and zooming for natural scenes with moving targets. A simple (yet stable) rigid motion model and a closed-loop-based mosaicing algorithm are proposed to generate cylindrical mosaics automatically. Multi-resolution representations are built for interesting areas by separating zooming sub-sequences from a pan/zoom sequence. Moving objects are detected and separated from images based on motion information, and then more accurate contours are extracted using a modified active contour algorithm. The DMP construction method is fast, robust, and automatic, achieving five frames per second for image sequences with 384!288 color images on a Pentium III 800 MHz PC. The construction of the DMP representation can be used in virtual reality, video surveillance, and very low bit-rate video coding. q 2005 Elsevier B.V. All rights reserved. Keywords: Panoramic representation; Video mosaicing; Multi-resolution; Moving object extraction 1. Introduction Panoramic representations of visual scenes have a wide application scope, including virtual reality (VR), interactive 2D/3D video, tele-conferencing, content-based video com- pression and manipulation, and full-view video surveillance. A wide field of view (FOV) lens, e.g. a fish-eye [1] or panoramic lens [2–6], can be a solution for obtaining panoramic views. However, in addition to the high cost of these specially designed image sensors, images obtained by such sensors have substantial distortions, and mapping an entire scene into the limited sensor target of a standard video camera compromises image quality. Constructing a panoramic representation by mosaicing image sequences captured by ordinary cameras, on the other hand, meets the requirements of the aforementioned applications for high image resolution. However, very few of the current algorithms and systems are able to properly detect and represent foreground objects and to efficiently deal with large zoom effect. This motivates our work described in this paper. 1.1. Overview of our approach Our overall goal in image-based modeling is to create realistic 2D/3D panoramas from video sequences with the more general motion of a hand-held video camera [7–11]. The construction of layered panoramas and stereo mosaics of 3D scenes from translating cameras with constrained 6 DOF motion has been proposed in our previous work [8–11], where motion parallax is large and thus is used to recover the 3D structure of the static scenes. In this paper we deal with video sequences of scenes with moving objects, which are taken by hand-held cameras undergoing 3D rotation (mainly panning), zooming, and small translation. In this case, the motion parallax, if not zero, can be neglected due to the small translation. A new approach is proposed to automatically build a Dynamic and Multi-resolution 3608 Panorama (DMP) from such a video sequence. For applications of image-based modeling and rendering, we can control the camera’s motion so this requirement can be easily satisfied. Nevertheless, this is often the case for the operation of a video camera by a cameraman for a video program. Therefore, although Image and Vision Computing 24 (2006) 13–26 www.elsevier.com/locate/imavis 0262-8856/$ - see front matter q 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.imavis.2005.09.006 * Corresponding author. Tel.: C1 212 650 8799; fax: C1 212 650 6248. E-mail address: zhu@cs.ccny.cuny.edu (Z. Zhu). URL: http://www-cs.engr.ccny.cuny.edu/wzhu/.