IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 17, NO. 6, JUNE 2008 897 Maximum-Entropy Expectation-Maximization Algorithm for Image Reconstruction and Sensor Field Estimation Hunsop Hong, Student Member, IEEE, and Dan Schonfeld, Senior Member, IEEE Abstract—In this paper, we propose a maximum-entropy expec- tation-maximization (MEEM) algorithm. We use the proposed al- gorithm for density estimation. The maximum-entropy constraint is imposed for smoothness of the estimated density function. The derivation of the MEEM algorithm requires determination of the covariance matrix in the framework of the maximum-entropy like- lihood function, which is difficult to solve analytically. We, there- fore, derive the MEEM algorithm by optimizing a lower-bound of the maximum-entropy likelihood function. We note that the clas- sical expectation-maximization (EM) algorithm has been employed previously for 2-D density estimation. We propose to extend the use of the classical EM algorithm for image recovery from randomly sampled data and sensor field estimation from randomly scattered sensor networks. We further propose to use our approach in den- sity estimation, image recovery and sensor field estimation. Com- puter simulation experiments are used to demonstrate the superior performance of the proposed MEEM algorithm in comparison to existing methods. Index Terms—Expectation-maximization (EM), Gaussian mix- ture model (GMM), image reconstrution, Kernel density estima- tion, maximum entropy, Parzen density, sensor field estimation. I. INTRODUCTION E STIMATING an unknown probability density function (pdf) given a finite set of observations is an important aspect of many image processing problems. The Parzen win- dows method [1] is one of the most popular methods which provides a nonparametric approximation of the pdf based on the underlying observations. It can be shown to converge to an arbitrary density function as the number of samples increases. The sample requirement, however, is extremely high and grows dramatically as the complexity of the underlying density function increases. Reducing the computational cost of the Parzen windows density estimation method is an active area of research. Girolami and He [2] present an excellent review of recent developments in the literature. There are three broad categories of methods adopted to reduce the computational cost of the Parzen windows density estimation for large sample sizes: a) approximate kernel decomposition method [3], b) data Manuscript received March 29, 2007; revised January 13, 2008. The associate editor coordinating the review of this manuscript and approving it for publica- tion was Dr. Gaurav Sharma. The authors are with the Multimedia Communications Laboratory, Depart- ment of Electrical and Computer Engineering, University of Illinois at Chicago, Chicago, IL 60607-7053 USA (e-mail: hhong6@uic.edu; dans@uic.edu). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TIP.2008.921996 reduction methods [4], and c) sparse functional approximation method. Sparse functional approximation methods like support vector machines (SVM) [5], obtain a sparse representation in approxi- mation coefficients and, therefore, reduce computational costs for performance on a test set. Excellent results are obtained using these methods. However, these methods scale as making them expensive computationally. The reduced set den- sity estimator (RSDE) developed by Girolami and He [2] pro- vides a superior sparse functional approximation method which is designed to minimize an integrated squared-error (ISE) cost function. The RSDE formulates a quadratic programming problem and solves it for a reduced set of nonzero coefficients to arrive at an estimate of the pdf. Despite the computational efficiency of the RDSE in density estimation, it can be shown that this method suffers from some important limitations [6]. In particular, not only does the linear term in the ISE measure result in a sparse representation, but its optimization leads to as- signing all the weights to zero with the exception of the sample point closest to the mode as observed in [2] and [6]. As a result, the ISE-based approach to density estimation degenerates to a trivial solution characterized by an impulse coefficient distribu- tion resulting in a single kernel density function as the number of data samples increases. However, the expectation-maximization algorithm (EM) [7] provides a very effective and popular alternative for estimating model parameters. It provides an iterative solution, which con- verges to a local maximum of the likelihood function. Although the solution to the EM algorithm provides the maximum like- lihood estimate of the kernel model for density function, the resulting estimate is not guaranteed to be smooth and may still preserve some of the sharpness of the ISE-based density estima- tion methods. A common method used in regularization theory to ensure smooth estimates is to impose the maximum entropy constraint. There have been some attempts to bind the entropy criterion with EM algorithm. Byrne [8] proposed an iterative image reconstruction algorithm based on cross-entropy mini- mization using the Kullback–Leibler (KL) divergence measure [9]. Benavent et al. [10] presented an entropy-based EM algo- rithm for the Gaussian mixture model in order to determine the optimal number of centers. However, despite the efforts to use maximum entropy to obtain smoother density estimates, thus far, there have been no successful attempts to expand the EM algorithm by incorporating a maximum-entropy penalty-based approach to estimating the optimal weight, mean and covariance matrix. In this paper, we introduce several novel methods for smooth kernel density estimation by relying on a maximum-entropy 1057-7149/$25.00 © 2008 IEEE