A Mixed Reality 3D Conferencing Application Hirokazu Kato, Mark Billinghurst, Suzanne Weghorst, Tom Furness Human Interface Technology Laboratory University of Washington Box 352-142 Seattle, WA 98195, USA { kato, gro f, wegho rst, t fume s s } @hit 1.washin gton. edu ABSTRACT We describe a Mixed Reality conferencing application which uses the overlay of virtual images on the real world to support three dimensional remote computer supported collaborative work. Remote collaborators are represented on Virtual Monitors which can be freely positioned about a user in space. Users can collaboratively view and interact with virtual objects using a shared virtual whiteboard. This is possible through precise virtual image registration using fast and accurate computer vision techniques also described in the paper. Keywords Mixed Reality, Augmented Reality, CSCW INTRODUCTION Computers are increasingly used to enhance collaboration between people. As collaborative tools become more common the Human-Computer Interface is giving way to a Human-Human Interface mediated by computers. This emphasis adds new technical challenges to the design of Human Computer Interfaces. These challenges are compounded for attempts to support three-dimensional Computer Supported Collaborative Work (CSCW). Although the use of spatial cues and three-dimensional object manipulation are common in face-to-face collaboration, tools for three-dimensional CSCW are still rare. However new 3D interface metaphors such as virtual reality may overcome this limitation. Virtual Reality (VR) appears a natural medium for 3D CSCW; in this setting computers can provide the same type of collaborative information that people have in face-to- face interactions, such as communication by object manipulation, voice and gesture [12]. Work on the DIVE project [3], GreenSpace [6] and other fully immersive multi-participant virtual environments has shown that collaborative work is indeed intuitive in such surroundings. However most current multi-user VR systems are fully immersive, separating the user from the real world and their traditional tools. As Grudin [4] points out, CSCW tools are generally rejected when they force users to change the way they work. This is because of the introduction of seams or discontinuities between the way people usually work and the way they are forced to work because of the computer interface. Ishii describes in detail the advantages of seamless CSCW interfaces [5]. Obviously immersive VR interfaces introduce a huge discontinuity between the real and virtual worlds. An alternative approach is through Mixed Reality (MR), the overlaying of virtual objects onto the real world. In the past researchers have explored the use of MR approaches to support face-to-face collaboration. Projects such as Studierstube [10], Transvision [9], and AR2 Hockey [7] allow users can see each other as well as 3D virtual objects in the space between them. Users can interact with the real world at the same time as the virtual images, bringing the benefits of VR interfaces into the real world and facilitating very natural collaboration. In a previous paper we found that this meant that users collaborate better on a task in a face-to-face MR setting than for the same task in a fully immersive Virtual Environment [2]. In this paper we report on an application of MR techniques for supporting remote collaboration. We have developed a MR conferencing system that allows virtual images (Virtual Monitors) of remote collaborators to be overlaid on the users real environment. Our Mixed Reality conferencing system tries to overcomes some of the limitations of current desktop video conferencing, including the lack of spatial cues [11], the difficulty of interacting with shared 3D data, and the need to be physically present at a desktop machine to conference. While using this system, users can easily change the arrangement of Virtual Monitors, placing the virtual images of remote participants about them in the real world and they can collaboratively interact with 2D and 3D information using a Virtual Shared Whiteboard. The virtual images are shown in a lightweight head mounted display, so with a wearable computer our system could be made portable enabling collaboration anywhere in the workplace. In developing a multi-user mixed reality video conferencing system, precise registration of the virtual images with the real world is one of the greatest challenges. In our work we use computer vision techniques and have developed some optimized algorithms for fast, accurate real time registration. In this paper we first describe our conferencing SAM.700.003.0199