International Journal of Computer Vision manuscript No. (will be inserted by the editor) From local kernel to nonlocal multiple-model image denoising Vladimir Katkovnik, Alessandro Foi, Karen Egiazarian, Jaakko Astola Department of Signal Processing, Tampere University of Technology The date of receipt and acceptance will be inserted by the editor Abstract We review the evolution of the nonpara- metric regression modeling in imaging from the local Nadaraya-Watson kernel estimate to the nonlocal means and further to transform-domain ltering based on non- local block-matching. The considered methods are classi- ed mainly according to two main features: local/nonlocal and pointwise/multipoint. Here nonlocal is an alterna- tive to local, and multipoint is an alternative to point- wise. These alternatives, though obvious simplications, allow to impose a fruitful and transparent classication of the basic ideas in the advanced techniques. Within this framework, we introduce a novel single- and multiple- model transform domain nonlocal approach. The Block Matching and 3-D Filtering (BM3D) algorithm, which is currently one of the best performing denoising algo- rithms, is treated as a special case of the latter approach. 1 Introduction Suppose we have independent random observation pairs {z i ,x i } n i=1 given in the form z i = y i + ε i , (1) where y i = y(x i ) is a signal of interest, x i ∈ R d de- notes a vector of features or explanatory variables which determines the signal observation y i , and ε i = ε(x i ) is an additive noise, which we assume normally distributed with standard-deviation σ and mean zero. The problem is to reconstruct y(x) from {z i } n i=1 . In sta- tistics, the function y is treated as a regression of z on x, y(x)= E{z|x}. In this way, the reconstruction at hand is from the eld of the regression techniques. If a para- metric model cannot be proposed for y, then, strictly speaking, the problem is from a class of the nonpara- metric ones. Paradoxically, one of the most constructive ideas in nonparametric regression is a parametric local modeling. This localization is developed in a variety of modications and can be exploited for the argument fea- ture variables x, in the signal space y, or in the trans- form/spectrum domains. This parametric modeling in small  makes a big deal of difference versus the para- metric modeling in large . The idea of local smoothing and local approximation is so natural that it is not surprising it has appeared in many branches of science. Citing [72], we can mention early works in statistics using local polynomials by the Italian astronomer and meteorologist Schiaparelli (1866) and the Danish actuary Gram (1879) (famous for devel- oping the Gram-Schmidt procedure for orthogonaliza- tion of vectors). In the sixties-seventies of the twentieth century the idea became subject of an intensive theoreti- cal study and applications: in statistics due to Nadaraya (1964, [79]), Watson (1964, [112]), Cleveland and Devlin (1979, [17]) and in engineering due to Brown (1963, [9]), Savitzky and Golay (1964, [93]), Katkovnik (1976, [52], 1985, [53]). Being initially developed as local in x, the technique obtained recently a further signicant development with localization in the signal y domain as the nonlocal means algorithm due to Buades et al [10]. For imaging, the non- local modeling appeared to be extremely successful when exploited in transform domain. This is a promising direc- tion where the intensive current development is focused. The scope of this paper is twofold. First, we outline the evolution of the nonparametric regression modeling from the local Nadaraya-Watson estimates to nonlocal means and further to the nonlocal block-matching tech- niques. Second, we present a constructive contribution concerning a novel multiple-model modeling for the non- local block-matching techniques. A particular instance of this idea has been implemented in the block-matching 3-D (BM3D) image denoising algorithm (Dabov et al. [19]), which demonstrates a performance beyond the abil- ity of most modern alternative techniques (see, e.g., [66] or [108]). On one hand, the multiple-model interpreta- tion of the BM3D algorithm highlights a source of this outstanding performance; on the other hand, this very