IEEE SIGNAL PROCESSING LETTERS , VOL. X, NO. YY, MONTH ZZZZ 1 Superresolution using perceptually significant side information Vikas Ramachandra* and Truong Q. Nguyen Fellow, IEEE Abstract—We investigate the problem of super-resolution of images in the presence of side information. In some situations, when some information of the original image is available to the sender, it can be embedded into the low resolution images, either in the pixels themselves or in the headers. This information can be later used when required to reconstruct the superresolved image. For this, a novel multiresolution histogram matching based su- perresolution procedure is outlined. The proposed technique gives better results compared to contemporary resolution enhancement algorithms. I. I NTRODUCTION Superresolution (SR) is the problem of reconstructing a high resolution image from a single low resolution image or set of low resolution images, each of which contributes some unique information. SR is an ill-posed problem since there might be many high resolution images which give the same low resolution image set. We need to resort to regularization by imposing prior knowledge about the high resolution image, to restrict its solution space to a visually plausible set [1]. In ordinary SR at the receiver, one would impose a prior like the Markov random Field [20] or the constrained Total Variation norm model [13], [14] in conjunction with an estimation procedure to get an estimate of the high resolution image. These priors are inspired by general image statistics. However, if we had more specific ’side’ information about the actual image we are trying to reconstruct, our estimate would be much better. We believe that our method can perform better than con- ventional resolution enhancement techniques in the following two scenarios: • Superresolution of transmitted low resolution images over limited bandwidth. • Deblurring of print media (text and images on paper) captured by mobile phone cameras etc. In the above scenarios, the chosen side information can either be incorporated in the image headers or embedded into the low resolution image itself (before transmission or printing) using data embedding schemes such as Quantization Index Modulation [2]. In this paper, we look at what ’side’ information is perceptually well suited for SR. An algorithm that actually uses that information for high resolution image estimation is also explored. The authors are with the University of California, San Diego, 9500 Gilman Drive, San Diego CA 92093. Email: {vikas,nguyent}@ucsd.edu. Phone: 858 534 5669. * Corresponding author. A. Previous work and our contributions Previous related work which uses data embedding for im- age processing (non-security) applications includes improving coding efficiency by embedding color information in the image itself [4], using data embedding to enable good error conceal- ment for transmitted images [5], and for transmitted videos over lossy networks [6], and embedding information to help selectively filter transmitted compressed image regions [17]. However, none of the previous works use partial information embedding for superresolution and estimation of the original image like we do. Also, in papers like [12] and [16], one uses side information to enhance transmitted images, but these methods do not make use of the side information motivated by perceptually significant parameters like we propose. More recently, [3] embedded statistics of the original image into transmitted images, which could be used as an objective quality metric. Our work is inspired by recent advances in texture synthesis. It is discussed in [7], [10] and [11] that many textures can be reconstructed by matching the histograms of the filter responses of a set of well-selected bandpass filters. In [7], the authors proposed an iterative projection method with constraints imposed on the multiscale oriented pyramid coefficients, and were able to construct meaningful textures from random initial images. However, it was concluded that this method does not work for images. Our contributions in this paper are twofold. Firstly, we have modified the above mentioned texture synthesis method, turning it into an image deblurring scheme, making it suitable for superresolution. Our proposed modifications are explained in the sections which follow. Secondly, we have presented a framework for using side information for image resolution enhancement. Our technique is general in that it can be used to further augment the performance of any existing popular SR or image enhancement technique like [8] and [9] which make use of regularization of this ill posed inverse problem. Our method can be viewed as a method to enhance prior knowledge. We believe that this can also help do away with simplistic but unrealistic assumptions being made at present. The proposed algorithm follows these steps: • Side information embedding- At the image print output device or transmitter, embed a compact representation of the side information into the image(s) being stored/ sent, or, if possible, encode the side information in the header. • Extraction and decoding of the side information- At the receiver or image capture device, when required, decode the embedded/encoded information in the set of image(s)