A common framework for nonlinear diffusion, adaptive smoothing, bilateral filtering and mean shift Danny Barash a, * , Dorin Comaniciu b a Department of Chemistry and Courant Institute of Mathematical Sciences, New York University and Howard Hughes Medical Institute, 31 Washington Place, Main 1021, New York, NY 10003, USA b Real-Time Vision and Modeling Department, Siemens Corporate Research, 755 College Road East, Princeton, NJ 08540, USA Received in revised form 29 July 2003 Abstract In this paper, a common framework is outlined for nonlinear diffusion, adaptive smoothing, bilateral filtering and mean shift procedure. Previously, the relationship between bilateral filtering and the nonlinear diffusion equation was explored by using a consistent adaptive smoothing formulation. However, both nonlinear diffusion and adaptive smoothing were treated as local processes applying a 3 £ 3 window at each iteration. Here, these two approaches are extended to an arbitrary window, showing their equivalence and stressing the importance of using large windows for edge-preserving smoothing. Subsequently, it follows that bilateral filtering is a particular choice of weights in the extended diffusion process that is obtained from geometrical considerations. We then show that kernel density estimation applied in the joint spatial – range domain yields a powerful processing paradigm—the mean shift procedure, related to bilateral filtering but having additional flexibility. This establishes an attractive relationship between the theory of statistics and that of diffusion and energy minimization. We experimentally compare the discussed methods and give insights on their performance. q 2003 Elsevier B.V. All rights reserved. Keywords: Nonlinear diffusion; Adaptive smoothing; Bilateral filtering; Mean shift procedure 1. Introduction Nonlinear operations are becoming increasingly import- ant in visual processing applications. Since they are substantially more difficult to analyze, formulate and predict compared to linear operations, various innovative approaches have been proposed independently for low- level computer vision tasks. The integration of several approaches that rely on different mathematical tools (e.g. functional minimization, nonlinear PDEs, statistics and data analysis) is essential for obtaining high-quality results in real-life applications. This paper concentrates on edge-preserving smoo- thing. It extends previous work [1,2] on the relation- ship between nonlinear diffusion [19,20,26], adaptive smoothing [21], and bilateral filtering [24] to establish a connection to the mean shift procedure [8,9] in the joint spatial–range domain. Both nonlinear diffusion and adaptive smoothing are generalized to encompass large neighborhoods, while the bilateral filtering serves as a link between the extended nonlinear diffusion (i.e. nonlinear diffusion on extended neighborhoods) and mean shift filtering. The paper is divided as follows. Section 2 emphasizes the importance of extended neighborhoods in edge-preserving smoothing by analyzing smoothing on 1D 3-neighborhood, smoothing on 1D 5-neighborhood, and adaptive smoothing on 1D 5-neighborhood. This leads to the formulation in Section 3 of the extended nonlinear diffusion on 2D (2S þ 1 £ 2S þ 1)-neighborhood. In Section 4, it is shown that a specific choice of weights in the extended nonlinear diffusion, that is based on geometrical considerations, leads to bilateral filtering. By defining kernel density estimation in the spatial–range domain, we derive in Section 5 the mean shift procedure for filtering and show its extended flexibility over bilateral filtering. In Section 6 experiments and comparisons are presented, while in Section 7, conclusions 0262-8856/$ - see front matter q 2003 Elsevier B.V. All rights reserved. doi:10.1016/j.imavis.2003.08.005 Image and Vision Computing 22 (2004) 73–81 www.elsevier.com/locate/imavis * Corresponding author. Address: Genome Diversity Center, Institute of Evolution, Haifa University, Mount Carmel 31905, Isreal. E-mail addresses: dbarash@research.haifa.ac.il (D. Barash), comanici@scr.siemens.com (D. Comaniciu).