IEEE SIGNAL PROCESSING LETTERS, VOL. 18, NO. 2, FEBRUARY 2011 103 Mean Square Error Estimation in Thresholding Soosan Beheshti, Senior Member, IEEE, Masoud Hashemi, Student Member, IEEE, Ervin Sejdic ´ , Member, IEEE, and Tom Chau, Senior Member, IEEE Abstract—We present a novel approach to estimating the mean square error (MSE) associated with any given threshold level in both hard and soft thresholding. The estimate is provided by using only the data that is being thresholded. This adaptive approach provides probabilistic confidence bounds on the MSE. The MSE bounds can be used to evaluate the denoising method. Our sim- ulation results confirm that not only does the method provide an accurate estimate of the MSE for any given thresohlding method, but the proposed method can also search and find an optimum threshold for any noisy data with regard to MSE. Index Terms—Confidence bounds, estimation evaluation, mean square error (MSE), thresholding. I. INTRODUCTION E STIMATING a set of unknown parameters corrupted by additive noise is one of the most important problems in science and engineering. A common measure for the quality of the estimators is the mean-square error (MSE). In many areas (e.g., estimation, denoising, modeling), the aim is to estimate and/or minimize a form of the MSE. When a suboptimal esti- mation algorithm in the sense of MSE is used, it is of our interest to compare the resultant MSE to the minimum (optimum) MSE (MMSE). However, computing the MMSE is usually cumber- some as well. To overcome this problem various application-de- pendent methods have been proposed to estimate bounds for MMSE. For example, in [1] a lower bound was provided for error estimation of diffusion filters which is based on covariance inequality and Cramer–Rao bounds [2]. Similar developments can be found in [3]–[6] which are geared towards estimating a desired unknown parameter from an observed pulse-frequency modulated signal. MSE estimators can also be found in appli- cations dealing with linear estimation and regression models (e.g., [7], [8]). Nevertheless, generalizations of these findings are often limited as they often deal with specific applications, classes of signals, and/or a range of signal to noise ratio. In par- ticular, these previous findings are generally not applicable in denoising applications, where estimation of MSE for various threshold values is desired. Manuscript received October 05, 2010; revised November 18, 2010; accepted November 21, 2010. Date of publication December 10, 2010; date of current version December 23, 2010. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Markku Renfors. S. Beheshti is with the Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON M5B 2K3 Canada (e-mail: soosan@ee.ry- erson.ca). M. Hashemi is with the Institute of Biomaterials and Biomedical Engineering (IBBME), University of Toronto, Toronto, ON M5S 1A1 Canada (e-mail: sayed- masoud.hashemiamroabadi@utoronto.ca). E. Sejdic ´ is with the Division of Gerontology, Beth Israel Deaconess Medical Center and Harvard Medical School, Harvard University, Boston, MA 02215 USA (e-mail: esejdic@bidmc.harvard.edu). T. Chau is with Bloorview Research Institute and the IBBME, University of Toronto, Toronto, ON M5S 1A1 Canada (e-mail: tom.chau@utoronto.ca). Digital Object Identifier 10.1109/LSP.2010.2097590 In this letter, we propose a method for the MSE estimation in denoising applications. Different denoising techniques exist for removing noise from the desired data, ranging from linear transformations such as Wiener filters [9] to modern approaches that are mostly concentrated on using wavelet coefficients and shrinkage for noise cancelation [10]–[14]. Our development is in line with these modern approaches. Specifically, our method is applicable for any thresholding technique and any type of signal structure. To achieve this goal, we first study the structure of the MSE in a thresholding scenario. Then, we provide prob- abilistic bounds on the MSE by using only the available noisy data. A similar method has been presented for MSE estimation for subspace selection and linear modeling (e.g., [15] and [16]). To develop an MSE estimator for the purpose of thresholding, the proposed method is based on the same fundamental princi- ples as this previous technique. Here we provide probabilistic confidence bounds on the MSE as a function of the available data. The letter is arranged as follows. Section II summarizes the background of thresholding and formulates the problem under consideration. In Section III, the structure of MSE in thresh- olding is provided. Section IV introduces the method for the MSE estimation by using only the available noisy data, while Section V covers the simulation results. Section VI draws the concluding remarks. II. PROBLEM STATEMENT AND MOTIVATION The unknown noiseless data is corrupted by an ad- ditive white Gaussian noise . The noisy data of length : (1) is available for . The additive noise is a sample of a zero mean random variable with variance . The data is projected onto another orthogonal basis that presents the noiseless data with possibly fewer nonzero coefficients than the data length. The most popular example of such a basis is the class of orthogonal wavelets. The associated coefficients of the data in the new bases are the result of the following inner product (2) where , are the complete orthonormal basis, and are the coefficients of noisy and noiseless data, and is the coefficient of the additive noise. Due to the orthonormality of the transformation, the transformed additive noise is zero mean with the same noise variance . Thresholding is classified into the two types of hard and soft. Assume that the value of threshold is , then we denote the hard thresholded coefficients with and the soft thresholded version with . We use the notation of where both hard and soft thresholding satisfy conditions. 1070-9908/$26.00 © 2010 IEEE