Quality improving techniques for free-viewpoint DIBR Luat Do a , Sveta Zinger a and Peter H.N. de With a,b a Eindhoven University of Technology, P.O. Box 513, 5600 MB Eindhoven, Netherlands b Cyclomedia Technology B.V., P.O. Box 68, 4180 BB Waardenburg, The Netherlands {Q.L.Do, S.Zinger, P.H.N.de.With}@tue.nl ABSTRACT Interactive free-viewpoint selection applied to a 3D multi-view signal is a possible attractive feature of the rapidly developing 3D TV media. This paper explores a new rendering algorithm that computes a free-viewpoint based on depth image warping between two reference views from existing cameras. We have developed three quality enhancing techniques that specifically aim at solving the major artifacts. First, resampling artifacts are filled in by a combination of median filtering and inverse warping. Second, contour artifacts are processed while omitting warping of edges at high discontinuities. Third, we employ a depth signal for more accurate disocclusion inpainting. We obtain an average PSNR gain of 3 dB and 4.5 dB for the ‘Breakdancers’ and ‘Ballet’ sequences, respectively, compared to recently published results. While experimenting with synthetic data, we observe that the rendering quality is highly dependent on the complexity of the scene. Moreover, experiments are performed using compressed video from surrounding cameras. The overall system quality is dominated by the rendering quality and not by coding. Keywords: 3D view interpolation, view synthesis, Depth Image Based Rendering (DIBR) 1. INTRODUCTION Three-dimensional (3D) video is nowadays broadly considered to succeed existing 2D HDTV technologies. The depth in a 3D scene can be created with e.g. stereo images or by explicitly sending a depth signal or map, as an addition to the texture image. An interesting feature in 3D imaging is to virtually move through the scene in order to create different viewpoints. This feature, called multi-view video has become a popular topic in coding and 3D research. Viewing a scene from different angles is an attractive feature for applications such as medical imaging, 1, 2 multimedia services 3 and 3D reconstruction. 4 Since the number of cameras is practically limited and consequently the number of viewing angles, research has been devoted to interpolate views between the cameras. The creation of such artificial views in 3D is called rendering. Pulli et al. 5 have shown that one can obtain an improved rendering quality by using the geometry of the scene. In our case, we employ a depth signal as a geometry description, besides the usual texture images. When employing depth signals, a well-known technique for rendering is called Depth Image Based Rendering (DIBR), which involves the projection of a viewpoint into another view. This projection is actually based on warping. 6 Our research reported here concerns multi-view rendering algorithms and aims at finding the best free-viewpoint rendering algorithm, when using multi-view texture and depth images to describe the scene. Previous research on warping while using DIBR from one reference image, reveals two inherent limitations, 7 which are viewpoint dependency of textures and disocclusions. To overcome these limitations, most recent methods 8–12 employ warping from two surrounding reference images to a virtual, or synthesized, viewpoint. Disocclusions from one reference camera are compensated by the other camera and for minimizing errors in viewpoint dependency, textures of the virtual image are blended from the surrounding images. Zitnick et al. 8 have already pointed out that to render a high quality image at an arbitrary viewpoint, one has to cope with three major challenges. First of all, empty pixels and holes due to sampling of the reference image have to be closed. Secondly, pixels at borders of high discontinuities cause contour artifacts. The third challenge involves inpainting disocclusions that remain after blending the projected images (these areas cannot be viewed from any of the surrounding cameras). In this paper, each principal challenge and its Copyright 2010 Society of Photo-Optical Instrumentation Engineers. One print or electronic copy may be made for personal use only. Systematic reproduction and distribution, duplication of any material in this paper for a fee or for commercial purposes, or modification of the content of the paper are prohibited.