Efficient quality enhancement of disparity maps based on alpha matting Nicole Brosch a , Matej Nezveda a , Margrit Gelautz a and Florian Seitner b a Institute of Software Technology and Interactive Systems, Vienna University of Technology, Favoritenstrasse 9-11/188-2, 1040 Vienna, Austria; b emotion3D GmbH, Gartengasse 21/3, 1050 Vienna, Austria ABSTRACT We propose an efficient disparity map enhancement method that improves the alignment of disparity edges and color edges even in the presence of mixed pixels and provides alpha values for pixels at disparity edges as a byproduct. In contrast to previous publications, the proposed method addresses mixed pixels at disparity edges and does not introduce mixed disparities that can lead to object deformations in synthesized views. The proposed algorithm computes transparencies by performing alpha matting per disparity-layer. These alpha values indicate the degree of affiliation to a disparity-layer and can hence be used as an indicator for a disparity reassignment that aligns disparity edges with color edges and accounts for mixed pixels. We demonstrate the capabilities of the proposed method on various images and corresponding disparity maps, including images that contain fuzzy object borders (e.g., fur). Furthermore, the proposed method is qualitatively and quantitatively evaluated using disparity ground truth and compared to previously published disparity post-processing methods. Keywords: Post-processing, stereo matching, disparity map, alpha matting, novel view 1. INTRODUCTION Disparity maps describe the relative depth of a scene in the form of horizontal displacements of pixel positions (i.e., disparities) between two images (i.e., a stereo pair) of the same scene that were taken from slightly shifted viewpoints. Together with one or both views, the disparity map can be used to synthesize a different viewpoint of the scene (i.e., a novel view). In novel view generation for stereoscopic display/3D-content the quality of an underlying disparity map contributes to the quality of a synthesized, novel view. Misalignment of disparity and color edges, inaccuracies due to mixed pixels and mismatches may lead to visual artifacts and object deformations in the novel views (e.g., Fig. 1, (a)-(d)). This paper proposes an (1) efficient disparity map post-processing method based on alpha matting that (2) improves the alignment of disparity and color edges even in the presence of mixed pixels (e.g., Fig. 1, (e)) and (3) provides alpha values at disparity edges as a byproduct. To this end, the proposed algorithm computes transparencies (alpha values) at disparity edges by performing per-view and cross-view alpha matting for each disparity-layer. Since these alpha values indicate the degree of affiliation to a disparity-layer, they can be used as an indicator for disparity reassignments that align disparity edges with color edges and account for mixed pixels. When using a fast approximate matting algorithm, 1 per-view alpha matting can be implemented very efficiently (i.e., 3 milliseconds per view with 384 × 288 pixels). Related works focus either on filter-based disparity map refinement as a post-processing step or integrate methods that handle mixed pixels in the respective stereo matching approaches. One possibility to enhance disparity maps is to apply general filtering techniques that locally (2D neighborhood) smooth disparities (e.g., 1–8 ) or reassign them according to local statistics (e.g., 9, 10 ). In the context of misalignment of disparity and color edges, especially edge-preserving filtering techniques that process the disparity map based on its corresponding color image are of great interest. For example, the joint bilateral filter 4 can be used to locally average disparities Further author information: (Send correspondence to N.B.) N.B.: E-mail: nicole.brosch@tuwien.ac.at M.N.: E-mail: matej.nezveda@tuwien.ac.at M.G.: E-mail: margrit.gelautz@tuwien.ac.at F.S.: E-mail: seitner@emotion3d.tv