Exploiting the Power of Stereo Confidences David Pfeiffer Daimler AG Sindelfingen, Germany david.pfeiffer@daimler.com Stefan Gehrig Daimler AG Sindelfingen, Germany stefan.gehrig@daimler.com Nicolai Schneider IT-Designers GmbH Esslingen, Germany stz.schneider@daimler.com Abstract Applications based on stereo vision are becoming in- creasingly common, ranging from gaming over robotics to driver assistance. While stereo algorithms have been inves- tigated heavily both on the pixel and the application level, far less attention has been dedicated to the use of stereo confidence cues. Mostly, a threshold is applied to the con- fidence values for further processing, which is essentially a sparsified disparity map. This is straightforward but it does not take full advantage of the available information. In this paper, we make full use of the stereo confidence cues by propagating all confidence values along with the measured disparities in a Bayesian manner. Before using this information, a mapping from confidence values to dis- parity outlier probability rate is performed based on gath- ered disparity statistics from labeled video data. We present an extension of the so called Stixel World, a generic 3D intermediate representation that can serve as input for many of the applications mentioned above. This scheme is modified to directly exploit stereo confidence cues in the underlying sensor model during a maximum a poste- riori estimation process. The effectiveness of this step is verified in an in-depth evaluation on a large real-world traffic data base of which parts are made publicly available. We show that using stereo confidence cues allows both reducing the number of false object detections by a factor of six while keeping the detection rate at a near constant level. 1. Introduction Stereo vision has been an actively researched area for decades. In recent years, stereo algorithms and applica- tions have matured significantly spawning products in fields ranging from industrial automation over gaming up to driver assistance systems. The underlying stereo algorithms and their properties are well understood, at least for the current real-time algorithms, typically approaches based on correla- tion [20] or semi-global matching (SGM) [10]. Benchmarks Figure 1: The Stixel World computed from stereo data. The scene is segmented into free space and vertical obstacles. The color (from red to green) represents the object distance. that compare stereo algorithms on a 100 % density level are available [19], also for the automotive domain [8]. The computation of stereo confidences has only recently been researched in more detail. Hu and Mordohai [12] per- formed an excellent review of known stereo confidence met- rics comparing them to ground truth scenes on a pixel level. In related work on confidence estimation for stereo or opti- cal flow computation, the so called sparsification plots are established as the main method to show the effectiveness of the considered confidence metric. This procedure gives a good impression with respect to how well the confidence helps reducing the average error of the disparity map when the least confident values are removed. However, no ex- plicit use of both the disparity map and the confidence map in further processing has been reported so far. Our work is centered around the driver assistance sce- nario. The main objective is to robustly extract free space and obstacle information from dense disparity maps and to represent the results in a compact and simple fashion. The Stixel World, firstly introduced by Badino et al.[2], is a very suitable representation for this task. Based on an occupancy map [1, 5], this scheme allows to extract the closest row of objects for each image column. In a general- ization of this work, we introduced the multi-layered Stixel World [17] that allows to detect all objects in a scene. A result of this scheme is shown in Figure 1. This paper extends our Bayesian approach [17] to use stereo confidence cues. The idea is that each disparity mea- 4321