Real-time Panoramic Mapping and Tracking on Mobile Phones Daniel Wagner, Alessandro Mulloni, Tobias Langlotz, Dieter Schmalstieg Graz University of Technology Figure 1. Outdoor and indoor panoramas created on the fly using real-time mapping and tracking. ABSTRACT We present a novel method for the real-time creation and tracking of panoramic maps on mobile phones. The maps generated with this technique are visually appealing, very accurate and allow drift-free rotation tracking. This method runs on mobile phones at 30Hz and has applications in the creation of panoramic images for offline browsing, for visual enhancements through environment mapping and for outdoor Augmented Reality on mobile phones. KEYWORDS: Panorama creation, Tracking, Mobile phone INDEX TERMS: H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems-Artificial, augmented, and virtual realities; I.4.1 [Image Processing and Computer Vision]: Scene Analysis-Tracking 1 INTRODUCTION Tracking for outdoor Augmented Reality (AR) applications has very demanding requirements: It must deliver an accurate registration with respect to a given coordinate system, be robust and run in real time. Despite recent improvements, outdoor tracking still remains a difficult problem. Recently, mobile phones have become increasingly attractive for AR. With the built-in camera as the primary sensor, phones facilitate intuitive point- and-shoot interaction with the environment. Most outdoor tracking systems rely on inertial sensors to improve robustness. Even though some modern smart phones integrate a linear accelerometer, it is of little help in typical AR scenarios since it only delivers translational motion. Instead, most successful approaches rely on gyroscope sensors that measure rotations, which are primary sources for tracking instabilities. However, no mobile phone today possesses such sensors. It is likely that mobile phones will soon be equipped with gyroscopic sensors too. Schall et. al have shown [11] that carefully integrating a panoramic tracker into a system with GPS, compass, linear accelerometer and gyro can further improve the system’s robustness. In this paper we describe a natural-feature mapping and tracking method that is efficient, robust and allows for 3-degree- of-freedom tracking in outdoor scenarios on mobile phones. Assuming pure rotational movements, the method creates a panoramic map from the live camera stream (see Figure 1). The conceptual approach is similar to simultaneous localization and mapping (SLAM) [4][6]: For each video frame, the camera is first registered based on features in the map; In a second step, the map is then extended with new features from viewing directions that have not been observed before. Yet, while traditional SLAM systems create a sparse map of the environment and refine features over multiple observations (typically using triangulation), our approach creates a dense map of features, which are mapped during their first observation and not refined again. The first camera image is completely projected onto the environment map. When possible, the orientation and position of the first frame in the map can be derived from the phone’s accelerometer and compass. For all successive frames, the camera pose is updated – based on the existing data in the map – and the map is extended by only projecting areas that have not yet been {wagner|mulloni|langlotz|schmalstieg} @ icg.tugraz.at Institute for Computer Graphics and Vision; Inffeldg. 16, 8010 Graz, Austria