INCREMENTAL 3D BODY RECONSTRUCTION FRAMEWORK FOR ROBOTIC TELEPRESENCE APPLICATIONS Luis Almeida Institute of Systems and Robotics University of Coimbra IPT, Tomar, Portugal email: laa@ipt.pt Paulo Menezes Institute of Systems and Robotics University of Coimbra Coimbra, Portugal email: paulo@isr.uc.pt Lakmal D. Seneviratne (*) Division of Engineering King’s College London, UK email: lakmal.seneviratne@kcl.ac.uk Jorge Dias (*) Institute of Systems and Robotics University of Coimbra Coimbra, Portugal email: jorge@deec.uc.pt ABSTRACT This research proposes an on-line incremental 3D recon- struction framework that can be used on telepresence robots or human robot interaction (HRI). The aim is a low cost solution that enables users to communicate and interact re- motely experiencing the benefits of a face-to-face meeting. By exploring computers graphics techniques and spatial au- dio we intend to induce sensations of being physical in the presence of other people. There is a wide variety of re- search opportunities including high performance imaging, multi-view video, virtual view synthesis, etc. One fun- damental challenge in geometry reconstruction from tradi- tional cameras array is the lack of accuracy in low-texture or repeated pattern region. Our approach explores virtual view synthesis through motion body estimation and hybrid sensors composed by video cameras and a depth camera based on structured-light or time-of-flight. We present a full 3D body reconstruction system that combines visual features and shape-based alignment. Modeling is based on meshes computed from dense depth maps in order lower the data to be processed and create a 3D mesh representa- tion that is independent of view-point. KEY WORDS 3D reconstruction, augmented reality, human robot interac- tion, tele-presence; virtual view synthesis. 1 Introduction This work presents an on-line incremental 3D reconstruc- tion framework that can be used on telepresence robots, hu- man robot interaction (HRI) or augmented reality (AR) ap- plications. The project, based on recent low cost depth sen- sors, intends to create a domestic easy to install 3D acqui- sition and display system that enables socialization, collab- oration and entertainment. Exploring computers graphics techniques, spatial audio and artificial vision enables us to 1 The authors gratefully acknowledge support from Institute of Systems and Robotics at University of Coimbra (ISR-UC). (*) on sabbatical leave at Khalifa University of Science, Technology and Research (KUSTAR), Abu Dhabi, UAE induce sensations of being physical in the presence of other people useful on several domains like elderly loneness min- imization problem[20], tele-rehabilitation[18][32], com- panion robot, education, socialization, 3DTV, entertain- ment, etc. Phones and internet chat/audio/video conferencing programs (ex: VOIP, NetMeeting, Skype) have been used for socialization, nevertheless they are not able to create the remote person presence feeling. Means of communications that enable eye contact, gestures reconnaissance, body lan- guage and facial expressions are required. Figure 1. The concept goal: An mobile telepresence robot (Hilario) carrying an auto stereoscopic 3D display, video cameras, depth sensor, microphones and speakers enables users to communicate and interact remotely experiencing the benefits of a face-to-face meeting in full size. The concept goal is depicted on figure 1. An mobile telepresence robot carrying an auto stereoscopic display, video cameras, depth sensor, microphones and speakers en- ables users to communicate and interact remotely experi- encing the benefits of a face-to-face meeting in full size. On recent years, there has been significant advances on 3D displays not requiring special glasses, mostly due the game console industry and 3DTV. Flat-panel auto stereoscopic Proceedings of the IASTED International Conference November 7 - 9, 2011 Pittsburgh, USA Robotics (Robo 2011) DOI: 10.2316/P.2011.752-068 286