EUROGRAPHICS 2017/ P. Benard and D. Sykora Poster Proxy Clouds for RGB-D Stream Processing: A Preview Adrien Kaiser 1,2 Jose Alonso Ybanez Zepeda 2 Tamy Boubekeur 1 1 : LTCI, Telecom ParisTech, Paris-Saclay University 2 : Ayotle SAS proxies 2.5D RGB-D stream 2.5D RGB-D stream 3D primitives 3D textured mesh filtering hole filling selection triangulation build & update resampling rendering simulation mapping navigation measurement tracking transmission input processing output applications raw RGB-D data proxy clouds resampling noise removal hole filling meshing Figure 1: (Left) Proxy Clouds Workflow. From a stream of RGB-D frames, proxies are built and updated through time (Sec 2.1). They are used as priors to process the frames (resampling, filtering or hole filling) for better tracking, mapping, automated navigation or measurement. A selection of proxies based on the current RGB-D frame can be used for lightening data transmission or as triangulation prior for fast depth data meshing, with application to rendering or simulation. (Right) Data Improvement. Raw RGB-D data (top) and Proxy Clouds-improved data after 100 frames (bottom) showing results of real time noise removal, hole filling, point cloud resampling and meshing. Blue surrounded areas highlight regions where improvement using Proxy Clouds is significant compared to the low quality input RGB-D frames. Abstract Modern consumer depth cameras are widely used for 3D capture in indoor environments, for applications such as modeling, robotics or gaming. Nevertheless, their use is limited by their low resolution, with frames often corrupted with noise, missing data and temporal inconsistencies. In order to cope with all these issues, we present Proxy Clouds, a multiplanar superstructure for real-time processing of RGB-D data. By generating a single set of planar proxies from raw RGB-D data and updating it through time, several processing primitives can be applied to improve the quality of the RGB-D stream or lighten further operations. We illustrate the use of Proxy Clouds on several applications, including noise and temporal flickering removal, hole filling, resampling, color processing and compression. We present experiments performed with our framework in indoor scenes of different natures captured with a consumer depth sensor. Categories and Subject Descriptors (according to ACM CCS): I.4.3 [Computing Methodologies / Image Processing and Computer Vision]: Enhancement—Geometric Correction 1. Introduction 1.1. Objectives Modern consumer depth cameras are attractive with an affordable price and many possible applications of their real time RGB-D stream output, ranging from human computer interaction to aug- mented reality, through geometry capture. Although such technolo- gies made great progress over the last decade, the limited quality of their RGB-D stream still limit their application spectrum. It mostly originates in the low resolution of the frames and the inherent noise, incompleteness and temporal inconsistency attached to single view capture. Proxy Clouds aim at analyzing and structuring such streams to im- prove them on the fly, under real time embedded constraints with limited memory. They take the form of a lightweight planar super- structure stable through time which gives priors to apply several processing primitives to the RGB-D frames (Sec. 2.2), to reinforce the data and simplify or lighten subsequent operations. Our system takes a raw RGB-D stream as input and outputs an enhanced RGB- D stream together with the optional set of proxies associated with the current RGB-D data (see the workflow in Fig. 1 left). 1.2. Previous Work Plane Detection in RGB-D Stream Methods that build high level models of captured 3D data are mostly based on RANSAC, the Hough transform or Region Growing algorithms. In our embed- ded, real time, memory-limited context, we take inspiration from the RANSAC-based method proposed by Schnabel et al. [SWK07] for its time and memory efficiency, by repeating plane detection through time to acquire a consistent model and cope with the stochastic nature of RANSAC. c  2017 The Author(s) Eurographics Proceedings c  2017 The Eurographics Association. DOI: 10.2312/egp.20171039