A NOVEL INTERPOLATION SCHEME FOR RANGE DATA WITH SIDE INFORMATION Valeria Garro, Carlo dal Mutto, Pietro Zanuttigh, Guido M. Cortelazzo Department of Information Engineering, University of Padova, Italy Abstract Time-of-Flight matrix sensors currently available allow for the acquisition of range maps at video rate but usually have a limited resolution. At the same time high resolution color cameras are widely available. This makes highly desirable methods that are able to exploit the combined use of ToF sensors and color cameras to obtain high resolution range maps. This work presents a novel interpolation technique that exploits side information from a standard color camera to increase the resolution of range maps. A segmented version of the high resolution color image is used in order to identify the main objects in the scene, while a novel surface prediction scheme is used to interpolate the available depth samples. Critical issues like the joint calibration of the two devices and the unreliability of the acquired data have also been taken into account with ad- hoc solutions. The performance of the proposed scheme has been verified with both synthetic and real-world data and experimental results have shown how the proposed method allows to obtain a more accurate interpolation with sharper edges if compared with standard approaches. Keywords:Depth map, interpolation, super-resolution, calibration 1 Introduction In recent years a number of range measuring devices have been developed for 3D data acquisition, among them range cameras using matrix arrangements of Time of Flight (ToF) sensors are particularly interesting because they can obtain 3d measurement of dynamic scenes. These devices are based on different technologies, for example ToF cameras like Mesa Imaging’s SwissRanger TM [3] and Canesta [2] compute the points’ depth by emitting light signals and measuring the time delay the reflected light signals takes to reach the sensor again. Other range cameras like Zcam [1] implement a method based on optical shutter technology. Probably the main drawback of all these devices is their limited resolution (e.g. the SwissRanger TM works with a 176 × 144 pixel array). Super resolution of the range data returned by these devices is so an interesting research field. Previous works exploited additional information to improve depth map’s resolution combining ToF sensor with one or two high-resolution RGB cameras; the main idea is that depth discontinuities in range maps are often related to color and brightness changes in the corresponding color image. In [7] the enhancement is based on Markov Random Field (MRF) formulation of depth data solved via conjugate gradient; another approach [14] models a cost volume of depth probability and iteratively applies bilateral filtering. Another recent method [12] uses exclusively depth maps, without color image aid: many low-resolution depth maps of the same scene are aligned and merged together in order to obtain a single depth map with improved resolution. This method is restricted to static scenes’ acquisition. In this paper an alternate super resolution method for range maps is proposed. It works with a single additional registered RGB camera (an ancillary image represents information which can be collected very easily and at low cost). Our method relies on assumptions similar to those of [7] and [14]: depth map discontinuities and color or brightness changes in the related color image are correlated, especially in presence of edges. As previously said, our problem’s formulation differs from other methods because it does not use a probabilistic framework, instead it estimates surface’s patches with a novel interpolation strategy explicitly targeted to depth data derived from the depth estimation procedure developed for the compression algorithm presented in [15]. Before introducing the interpolation method the joint calibration of the two sensor is briefly described (an improved solution that makes use of a second color camera for higher accuracy is also presented). The proposed interpolation method encompasses three main steps. In the first step the color image is pre-processed with an efficient graph-based segmentation algorithm [8] in order to identify the main surfaces of the scene. Segmentation methods based on graph- cuts experimentally appear rather adequate for the considered tasks. However, in presence of highly textured surfaces, segmentation methods based on the combined use of depth and texture information [6, 11] can be adopted in order to avoid an oversegmentation effect. The connected components created in the output image, defined as regions with uniform pixel values, give an estimate of the surface regions that ideally belong to the same scene object. Each region can be approximated by a set of planar patches. In the second step the points of the low-resolution depth map are projected on the segmented color image. With this procedure a low-resolution depth grid is laid upon the color image. The elements of this grid are the depth values of the initial low-resolution depth