Interactive Annotation on Mobile Phones for Real and Virtual Space Registration Hyejin Kim * GIST U-VR Lab Gerhard Reitmayr † Graz University of Technology Woontack Woo ‡ GIST U-VR Lab ABSTRACT Registration of real space and virtual information is a fundamental requirement for any augmented reality system. This paper presents an interactive method to quickly create a 3D room model and an- notate locations within the room to provide registration anchors for virtual information. The method operates on a mobile phone and uses a visual rotation tracker to obtain orientation tracking for in- situ applications. The simple interaction allows non-expert users to create models of their environment and thus contribute marked-up representations to an online AR platform. Keywords: Annotation, mobile AR, interactive modeling, visual rotation tracking Index Terms: H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems—Artificial, augmented, and vir- tual realities; H.5.2 [Information Interfaces and Presentation]: User Interfaces—Interaction styles 1 I NTRODUCTION Current work for creating annotations in mobile AR applications fo- cuses on outdoor environments based on the ubiquitous tracking so- lution which combines GPS and orientation sensors. AR Browsers such as Junaio 1 allow users to place 3D objects or information tags at GPS locations and move them on a virtual ground plane relative to the user’s current location. Langlotz et al. [3] demonstrate an- notation authoring using panoramic images of the user’s location, providing more accurate placement of the annotations. However, no system is currently designed for indoor environments. In this paper, we focus explicitly on annotations for indoor en- vironments and individual rooms. Our system allows to quickly capture the dimensions of a room, approximated as a box, and to annotate and mark-up locations and items in the room. It operates at interactive frame-rates on a mobile device and provides simple touch-interaction to specify room dimensions, location and extends of rectangular areas on the room’s surface. These areas serve as anchors for linking virtual information to the real space represented by the room. 2 I NTERACTION AND ANNOTATIONS The interactive annotation method comprises of four interaction modes, one for each of the tasks: room modeling, marking up lo- cations, annotating, and viewing in AR and VR. In this section, we assume that we have an estimate of the camera orientation R relative to an unknown reference rotation, but with the direction of gravity aligned to the -z axis. The details of this orientation tracking are described in section 3. * e-mail: hjinkim@gist.ac.kr † e-mail:reitmayr@icg.tugraz.at ‡ e-mail:wwoo@gist.ac.kr 1 http://www.junaio.com h1 h2 d1 d2 (= d1) α1 α2 Figure 1: Picking algorithm setup with person height. 2.1 Room modeling In room modeling mode, user measures the dimension of a room, approximated as a box. Standing in a fixed position, and looking at a corner or edge of a wall with the floor computes the distance of that point from the user to the intersection with the floor using the human height and the current camera rotation R. We assume that the user’s operating height h 1 from ground to mobile device is calibrated and known to the system (see Figure 1). Then the modeling system computes an angle α 1 from cos α 1 =  f w ·  g |  f w ||  g | . (1) Here,  f w and  g denote forward vector with respect to the reference coordinate system and the gravity vector, (0, 0, -1). The forward vector can be obtained from  f w = R -1 (0, 0, -1) T , where, R -1 is inverse of current camera rotation. Then, by using the height h 1 and the angle α 1 , a distance d 1 is computed as d 1 = h 1 tan α 1 . (2) Distances on the floor like d 1 can be used to define the width and depth of the box from the user’s position. However, to obtain a 3D model, we need the height as well. Thus, the system requires one more interaction to measure the height h 2 by pointing at the ceiling edge of the first wall. d 1 and d 2 are the same because ceiling and floor are parallel planes. Then, using equation (1) the angle α 2 = π - α 1 . Finally, h 2 is computed using d 2 and α 2 , h 2 = d 2 tan α 2 . (3) After picking a floor and ceiling line in the first wall, the user ro- tates and looks at each of other three walls clockwise while picking only one floor or ceiling line.