Face Super Resolution in Reduced Spaces by Using Shape and Texture Aydın Akyol Istanbul Technical University, Turkey akyolayd@itu.edu.tr Muhittin Gökmen Istanbul Technical University, Turkey gokmen@itu.edu.tr Abstract The problem of inferring a missing face image which is at much higher resolution from lower observations is called as Face Super Resolution or Hallucination prob- lem. Mostly the problem is approached in spatial domain by using the aligned textural information of the observa- tion. However the ignorance of the shape information limits the performance of these approaches. In Resolution Aware Fitting (RAF) algorithm it was successfully shown that superior results could be obtained by utilizing both shape and texture components together. Though the RAF algorithm provides more satisfactory results, warping and deformation operations on high resolution image during the optimization could undermine its effectiveness in real world applications. As a remedy in this work we propose a faster alternative by effectively transforming the problem into reduced dimensions and making image warping only at low resolution. Experimentally it was shown that better reconstructions could be obtained faster than the RAF algorithm. 1. Introduction High Resolution (HR) images are critical for image analysis and the posterior applications using this analysis. However it is known that the optics of an imaging system limits the amount of information that is received by the imager device and the imaging system yields blurred and under-sampled images. At that point Super Resolution (SR) techniques are used to overcome the limitations of imaging systems. A forward model is assumed to represent the image formation and the most common form used for this pur- pose is L H I HI n   (1) where I H denotes the HR image and I L is the deformed Low Resolution (LR) version of I H under the deformation operator H, which presumably consists of blurring B and decimation D operators; H = DB. Also n represents the observation gap in formation. In SR problem it is intended to approximate the inverse of this forward. Reaching to the exact backward model would not be possible due to the ill-posed nature of the deformation H. SR techniques approximate to the exact solution by regularizing the Least Squares (LS) solution, |I L -HI H | 2 . This generic ap- proach can be expressed in Bayesian formulation as the MAP estimation of I H   ˆ arg max ( | )( ) H H L H H I I pI I pI  . (2) The first term p(I L |I H ) refers to the LS solution (called also as Maximum Likelihood solution – ML solution) and p(I H ) defines the a priori information. Depending on the needs, this generic problem definition can be restricted by making assumptions on these components of the problem. In order to align with the right literature, it is important to state the exact problem setup under consideration. In this work we assume that the image domain is restricted to the face images and the deformation operator H is known. This problem setup is also known as Face Hallucination problem in literature. Hallucination is first declared by Baker & Kanade [6] to describe the problem of inferring a missing face image which is at much higher resolution from LR observations. Within the scope of this work Hallucination definition is restricted to the case where single observation exists. Image analysis in constraint domains, such as face, high frequency components (or called as facial details) are critically important. Minor errors on these details might be significant both for human and machine perception. It is expected that an effective face hallucination technique can bring enough high frequency content to maximize the identity of the subject under processing. Basing on the fact that the LS solution could mostly provide the low frequencies, the high frequency content could only be gained via regularization. In literature general tendency is to benefit from the textural priors in order to regularize the solution. Though there are plenty of regularizers proposed [9], here we are contended with mentioning only on a few good repre- sentatives which are not only successful but also close to our proposal. In [2] Gunturk et.al. define one of these successful regularizers. The subspace projection statistics of the texture data is used as the prior information. Though it is possible to use other projection techniques as in [5], due to its computational simplicity PCA is preferred. In [1] Liu et.al. state that subspace statistics would bring only the mid frequencies, and in order to add higher frequen- cies more customized constraints are required. In addition to the subspace projection statistics they use also a Mar- kov Random Field (MRF) in order to define a joint locality model. Though wealthier content can be obtained with this non-parametric step, the results suffer from unrealistic texture caused by global discontinuity. This experiment shows that even restricted image domains could have excessive variety which could not be represented by even complex locality models. As an alternative to texture models, a relatively new trend in literature is to utilize the shape information in MVA2011 IAPR Conference on Machine Vision Applications, June 13-15, 2011, Nara, JAPAN 14-6 438