REAL-TIME DEPTH DIFFUSION FOR 3D SURFACE RECONSTRUCTION Karthik Mahesh Varadarajan, Markus Vincze Vision for Robotics, Automation and Control Institute TU Wien, Austria. {kv, mv}@acin.tuwien.ac.at ABSTRACT Range data obtained from conventional stereo-cameras employing dense stereo matching algorithms typically contain a high amount of noise, especially under poor illumination conditions. Furthermore, lack of reliable depth estimates in low-texture regions can result in poor 3D surface reconstruction. Anisotropic diffusion algorithms have been used recently in stereo matching, depth estimation and 3D surface reconstruction. However, these algorithms typically have long execution times, preventing real- time operation on resource constrained systems and robots. Moreover, most of these techniques suffer from excessive smoothing at depth discontinuities resulting in loss of structure, especially in areas where the 2D image does not provide structural cues to guide the depth diffusion. These algorithms are also unsuitable for diffusion of extremely sparse depth data such as in the case of homogenous surfaces. This paper addresses these issues by novel denoising and diffusion techniques. The results presented demonstrate the run-time efficiency and fidelity of reconstructed depth surfaces. Index Terms— Depth, dense stereo, anisotropic diffusion, real-time, multi-grid 1. INTRODUCTION Originating from the landmark paper by Perona and Malik [8], anisotropic diffusion algorithms have been used extensively in the past for noise removal and segmentation in electro-optic images. These algorithms have also been adapted to the context of stereo depth maps, following the paper by Scharstein and Szeliski [9]. Other algorithms that use diffusion in order to improve the quality of depth maps include Gaussian scale-space disparity estimation and anisotropic disparity field diffusion [5], Non-linear diffusion for depth enhancement [12, 6, 7], Layered scene representation [1], Betrami framework based non-isotropic regularization of the correspondence space in stereo vision [2], PDE based disparity enhancement [13]. Alternate depth enhancement schemes like median filtering and linear interpolation along epi-polar or scanlines [4] and segmentation stereo have been proposed, but suffer from deficiencies, such as assumption of co- planarity or rigid conformance to curvature metrics of points on depth surfaces. The bulk of these algorithms suffer from issues such as long execution times preventing real-time operation, loss of structure and depth edges and ineffective operation on extremely sparse range data. This paper seeks to address these concerns using a three-pronged approach. 2. REAL-TIME DEPTH DIFFUSION There are three major contributions of this paper. Firstly, this paper presents a novel image-agnostic statistical noise removal scheme for sparse 3D range or depth data that serves as an effective replacement for traditional image-agnostic heuristic depth filtering schemes such as median filtering. Secondly, this paper presents a novel adaptation of the Iterative Back Substitution (IBS) algorithm for piecewise isotropic and anisotropic estimation or diffusion of 3D depth data. The third main contribution of the paper stems from the design of the numerical optimization scheme for diffusion enabling rapid convergence in comparison with existing depth diffusion schemes. The execution time numbers achieved demonstrate suitability for real-time systems. The preservation of depth edges or discontinuities is also demonstrated in the estimation mode of operation. 2.1. Iterative Hysteresis Filtering and Morphological Reconstruction for De-noising Range Data In cases where the depth data is extremely sparse and noisy, such as in the case of low texture regions and poor illumination conditions, typical of indoor robotic applications, it is beneficial to use a depth pre-processing filter that eliminates large noisy pixels prior to diffusion. Furthermore, while the depth diffusion scheme presented in this paper inherently smoothes depth values, it can also be used in the estimation mode, wherein the scheme estimates range only at points at which the stereo algorithm fails to produce valid depth values, while keeping the values unchanged for known depth pixels. This mode is extremely useful for preserving depth discontinuities and depth edges, as well as when the sparsity of the data is high. These two modes are labeled filtering and estimation. The various steps in the proposed de-noising algorithm are: 1. The input depth map is divided into core-blocks and the standard deviation (σ c ) of each core-block is estimated, using values of known and valid depth pixels. Macro- blocks corresponding to each core-block are created, composing of a larger number of pixels and centered at the core-block and its standard deviation estimated as σ m. 2. For each core-block and macro-block, logical maps corresponding to all valid pixels, the values of which fall within a pre-determined threshold times σ c and σ m, - respectively, are estimated. The threshold for the macro- block is set higher than that for the core-block, thereby permitting greater deviation. 3. Valid pixels in the core-block that are flagged true in both the logical maps retain their original values in the filtered depth map. These pixels are well-behaved, in the 4149 978-1-4244-7993-1/10/$26.00 ©2010 IEEE ICIP 2010 Proceedings of 2010 IEEE 17th International Conference on Image Processing September 26-29, 2010, Hong Kong