Adaptive User Interfaces in Augmented Reality Simon Julier Mark A. Livingston † J. Edward Swan II † Yohan Baillot Dennis Brown † Naval Research Laboratory Washington D.C. Abstract In this position paper, we consider the issue of develop- ing a flexible software architecture which will allow mul- tiple interaction techniques to operate simultaneously and in parallel with one another. We describe a set of inter- action techniques, identify their architectural requirements, and consider methods for prioritizing different techniques and arbitrating between them when they have conflicting objectives. The discussion is made within the context of a mobile AR system for urban situation awareness. 1 Introduction Augmented reality (AR) presents information in its con- text in the 3D environment. This capability raises a set of questions such as: what information needs to be shown, what is the most appropriate way to show it, and how do the display elements interact with one another and the real world? These questions become even more complex and interesting when we utilize the AR display metaphor of “X- ray vision” (a capability that can only be provided by AR), and when we visualize over a large region such as a city. The human mind possesses vast capabilities for process- ing and integrating information. One could argue, on this basis, that the AR system should simply present all avail- able information and let the user decide for themselves what is the necessary data to perform the task at hand. This ar- gument pre-supposes that the information is presented in a physically realistic (and photo-realistic) manner—i.e. that the AR system can present all the visual cues that we use in human vision. Most AR systems, however, are neither ca- pable nor likely to be capable of producing real-time photo- realistic visualizations of their information databases. In- deed, there is no constraint that such information even have ITT Advanced Engineering and Sciences / Naval Research Labora- tory. Corresponding email: julier@ait.nrl.navy.mil † Virtual Reality Laboratory, Naval Research Laboratory a physically realistic representation. This will become acute when we discuss the physically unrealistic representation of occluded surfaces in Section 2.2. Even if photorealistic visualizations could be produced in real time (with proper lighting and shadows and other difficult rendering tasks), it would require tremendous com- putational power. Much of this effort would be wasted; the information would be filtered out by the human visual sys- tem. While it is difficult to know a priori what portions of the image will be filtered out when the user examines the image for information, it is the domain of user-centered de- sign to determine what information is necessary, and exactly what set of cues conveys that information precisely. Recent research into AR systems have developed a num- ber of user interface techniques including information fil- tering [8], occlusion representation [11, 7], adaptation to registration error [12], adaptive label placement [3], and 3D multimodal interaction [9]. Despite the complementary nature of these techniques, no AR system has, to date, has attempted to combine all of these techniques into a single display. This position paper considers the problem of developing an architecture to allow the use of multiple techniques si- multaneously. Section 2 describes each of the techniques in more detail and classifies them in terms of their effects on the rendering pipeline and the architectural support compo- nents which are required. A proposed architecture is de- scribed in Section 3. Section 4 summarises the paper and suggests open topics for discussion. 2 Visualization and Interaction Techniques This section describes five user interface algorithms. Each algorithm is described in terms of the need for that algorithm, an implementation of it, and a characterization of the architectural requirements. After the individual algo- rithms are presented, we categorize them by their effects on the geometry in the rendering pipeline from the database to the screen.