Immersive Multi-User 3D Video Communication I. Feldmann 1 , O. Schreer 1 , P. Kauff 1 , R. Schäfer 1 , Z. Fei 2 , H.J.W. Belt 2 , Ò. Divorra 3 1 Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institute, Germany 2 Philips Research, Netherlands 3 Telefonica Research, Spain ABSTRACT The interest in immersive video conference systems exists now for many years from both sides, the commercialization point of view as well as from a research perspective. Nevertheless, so far the user acceptance of such systems was still very limited. This situation changed recently. Technological advances in fields like display and camera technology as well as processing hardware lead the way to a new generation of immersive tele-conference systems. On one hand, large scale and high definition displays significantly enhance the feeling of virtual presence. Nowadays, commercial solutions benefit from these facts. Besides this, research in the area of multi-user 3D display technology shows promising results. On the other hand, new fast graphics board solutions allow a high algorithmic parallelization in a consumer PC environment. In this way, real time, high quality and high resolution implementations of more sophisticated 2D and 3D acquisition algorithms, such as volume based approaches, become more and more realistic. From this point of view this paper summarizes first results and experiences of the European FP7 research project 3DPresence which aims to build a three party and multi-user 3D tele-conferencing system. The goal of this paper is to discuss general issues and problems of future generation immersive mutli-user 3D video conference systems. Further on, it provides challenging first results and proposes solutions for critical questions. INTRODUCTION Research and development in the area of video communication from a local to one or multiple remote sides has a long tradition. Especially in the past few years the interest in generating a so called tele-presence increased rapidly. The work in this area includes topics like naturalness, feeling of physical presence, gesture awareness and eye contact etc. Recent high-end commercial solutions such as Cisco’s TelePresence (see Figure 1), Polycom’s TPX, and HP’s HALO partially remove some of the tele-presence shortcomings of traditional systems with immersive high-quality audio and high-definition life-size video. Still, these systems do not present the remote participants in life-sized 3D, limiting the naturalness and thereby the sense of tele-presence. In addition, a fundamental problem is that eye contact is unnatural and that directional gaze awareness is missing. An example based on a commercial system is given in Figure 2. On the left of Figure 2 the viewing direction of the remote participant towards the most right local participant appears correct. But as seen in Figure 2 right, from the position of the most right local participant, the eye contact and viewing direction is completely misleading, although the remote participant is looking directly to the local participant on its display.