Abstract: In this paper we describe ongoing research that aims at the development of a ge- neric demonstration platform for virtual proto- type modeling by utilizing multimodal – speech and gesture – interactions in Virtual Reality. Particularly, we concentrate on two aspects. First, a knowledge-based approach for assembling CAD-based parts in VR is intro- duced. This includes a system to generate meta-information from geometric models as well as accompanying task-level algorithms for virtual assembly. Second, a framework for modeling multimodal interaction using gesture and speech is presented that facilitates its ge- neric adaptation to scene-graph-based applica- tions. The chosen decomposition of the re- quired core modules is exemplified by an ex- ample of a typical object rotation interaction. Keywords: Virtual Reality, Virtual Assembly, Multimodal Interaction, Interaction De com- position, Task-Level Interfaces 1 Introduction The project “Virtuelle Werkstatt” aims to combine and improve research results in multimodal human- computer interaction and immersive CAD systems to develop a generic demonstration platform for Vi rtual Reality (VR) based prototyping. Multimodal interac- tion is concerned with the processing of gesture and speech based user input to drive the modifications of a 3D-visualized scene. Immersive CAD is concerned with the design, exploration and evaluation of virtual prototypes. The intention of our project is to provide a facility for a natural, mutimodal communication in a virtual construction scenario, which is similar to the way two humans would interact with speech and gestures. To enable natural multimodal interaction in virtual environments, results from different research communities have to be considered: If speech recog- nition as an already active research field is put aside, multimodal interaction can be realized by combining VR with the advances of Artificial Intelligence (AI). The integration of multimodal interaction in existing state-of-the-art VR-principles should hide the spe- cific internal functional details and should provide a form of reusable interface components. The project is carried out in a newly built VR-system consisting of a 3-sided Cave, projected by 6 D-ILA projectors and using passive stereo via circular po- larization filters. The gestures of the user are tracked with a marker-based infrared camera system. For precise hand-posture tracking we use two wireless Data-Gloves. The sound set-up is an eight channel system with loudspeakers in each corner of the cave. The computer cluster for application and rendering is an ‘Artabel Fleye 160’ Linux cluster, including 5 server nodes (double Pentium III-Class PCs) and 8 graphic nodes (single Pentium IV-Class PCs) with NVIDIA GeForce 3 graphic boards. The nodes are connected via a 2GBit/s Myrinet network for distrib- uted OpenGL rendering. Figure 1: A “Citymobile” vehicle in a VR-demo setting The first part of this paper presents the notion of port-concepts and connection properties of assembly objects and how they can be extracted from purely geometrical descriptions. It also shows the represen- tation of these connection properties in a knowledge base and task-level algorithms for assembling and disassembling aggregates and adjusting existing connections. The second part deals with a framework for multimodal interaction that handles integration of detected gesture and speech via a temporal aug- mented transition network (tATN) integration scheme and the application adaptation in form of scene graph components. 2 Assembly Simulation for Interactive Virtual Prototype Modeling 2.1 A Knowledge-Based Approach for Repre- senting Connection Prope rties To interact with and assemble components in virtual reality, it is necessary to augment the plain CAD- models with information about their connection properties. A conceptual model of part matings has Virtuelle Werkstatt: A Platform for Multimodal Assembly in VR P.Biermann, B. Jung, M. Latoschik, I. Wachsmuth Laboratory for Artificial Intelligence and Vi rtual Reality University of Bielefeld http://www.techfak.uni-bielefeld.de/ags/wbski/