Inertial Sensors Aided Image Alignment and Stitching for Panorama on Mobile Phones Qingxuan Yang Google Research Beijing, China qingxuan@google.com Chengu Wang IIIS, Tsinghua University Beijing, China wangchengu@gmail.com Yuan Gao Department of Computer Science and Technology Tsinghua University Beijing, China gaoy03@gmail.com Hang Qu Google Research Beijing, China quhang@google.com Edward Y. Chang Google Research Beijing, China edchang@google.com ABSTRACT In this paper, we propose using signals collected from in- ertial sensors on cameras to speed up image alignment for panorama construction. Inertial sensors including accelerom- eters and gyroscopes are first calibrated to improve sensing accuracy. These sensors are then used to estimate the posi- tion and orientation of each captured image frame. By know- ing the relative displacement of image frames, alignment can be performed with good accuracy and computational effi- ciency. Through examples we illustrate the effectiveness of inertial-sensor assisted panorama. ACM Classification Keywords I.3.3 Computer Graphics: Picture/Image Generation. General Terms Algorithms, Experimentation, Performance. Author Keywords image alignment, panorama, inertial navigation systems, im- age stitching INTRODUCTION While mobile phones are usually equipped with cameras, due to style and cost considerations, very few are equipped with an advanced lens. However, new generation smart phones such as the iPhone and Nexus come with high-capacity com- putational/graphical processing units and inertial navigation systems. In this work, we show that these CPUs/GPUs and inertial navigation systems (INS) can be utilized to improve the quality of images taken by inexpensive lenses of mobile phones. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MLBS’11, September 18, 2011, Beijing, China. Copyright 2011 ACM 978-1-4503-0928-8/11/09...$10.00. One useful imaging application on mobile phones is panorama, which aligns and stitches frames of limited field of view into ones with a larger field of view. With such capabil- ities, a mobile-phone camera could “scan” a document, a room, or an outdoor wide-angle scene for example. Tradi- tional panorama support requires a user to steadily hold a camera and then slowly pan the camera from one side to the other, sequentially snapping photos. Such requirements en- sure that each frame quality is of good enough quality and consistent enough with other frames such that frame align- ment can be performed effectively. However, such stringent usage requirements makes for poor user experience. Further- more, an undesirable camera movement, such as a camera tilt/rotation or uneven-speed panning, can degrade stitching quality. In this work, we show that using inertial sensors (accelerom- eters and gyroscopes) on mobile phones to register cam- era position, orientation, and moving trajectory can provide valuable information for the alignment, and subsequent stitch- ing of captured frames. In particular, this allows a user the ability to scan a scene with any camera movement trajec- tory. Each frame can then be processed and adjusted based on the camera’s orientation when the photos were taken. Next, alignment of adjacent frames can be performed using INS provided information. The inter-frame displacement in- formation provided by the sensors reduces the search space for finding matching inter-frame features. Since the search space is largely reduced and denser sampling can be con- ducted to ensure high-quality alignment and stitching, both stitching quality and speed can be improved. This paper makes two key contributions. First, we propose non-intrusive calibration methods to improve sensor accu- racy. Second, we show how collected signals from iner- tial sensors can be used to obtain speed enhancements in panorama construction. In the following sections we first describe detailed techniques in Section II, and then show the results in Section III. Section IV introduces future work. 21