On Sparse Representations of Color Images Xiaolin Wu and Guangtao Zhai Department of Electrical and Computer Engineering, McMaster University, Canada (e-mail: xwu@ece.mcmaster.ca) Abstract—We investigate an intrinsic and useful form of sparsity of color images that was largely overlooked in the literature of image/video processing. This sparsity of multispec- tral images is revealed and formulated by modeling the image formation process. The underlying new sparse representations of color images are general and can be exploited to improve the performance of existing image restoration algorithms, such as denoising, deblurring, and resolution upconversion. Key words: Sparse representations of images, image for- mation model, image restoration, inverse problem. I. I NTRODUCTION Most digital images and videos are multispectral, having at least three color bands (typically red, green and blue). In natural scenes, most light sources have a continuous spectrum, most objects have a uniform surface material of a certain reflectance and the surface curvature is quite small. As such, different spectral bands of the image signal have high correla- tions. This research is concerned with sparse representations of multispectral images. The subject is of significance and utility because a wide range of tasks in image processing and computer vision are performed with the assumption, either explicitly or implicitly, that the underlying image signal is sparse. Recent years have seen a great deal of renewed interests, enthusiasm and progress in sparsity-based image processing, particularly in image restoration. However, quite surprisingly, most published algorithms for image processing and analysis based themselves on the sparsity of luminance component of the image signal and overlooked the sparsities induced by spectral correlations. This leaves a slack in the performance of these algorithms. Mairal et. al extended the K-SVD algorithm [1] to color images in the searching of a dictionary based sparse representation of color images [2]. In this paper, to pick up the performance slack we investigate ways to for- mulate spectral correlations into inherent and computationally amenable sparse representations of multispectral images. Our investigation begins with an image formation model of digital color cameras. This image model and mild assumptions on illumination conditions and imaged objects reveal intrinsic sparsity properties of natural images. It turns out that these sparsities have simple, linear representations that are weighted sum of different spectral bands. This discovery allows the newly revealed sparsities of color images to be readily ex- ploited by an ℓ 1 minimization process, or by linear program- ming algorithmically. Upon the conclusion of our technical development, it will become self evident how the new results of this paper can be integrated into the general framework of image restoration and used as strong domain knowledge to improve the solution of the corresponding inverse problem. The remainder of the paper has the following flow of pre- sentation. The image formation model is reviewed in Section II, which leads to the sparse representation that is detailed in Section III. Typical applications of color image denoising and deconvolution are investigated in Section IV. And finally, Section V concludes the paper. II. I MAGE FORMATION MODEL A multispectral camera records the light reflections of a real world scene. We model the light reflection from the surface of an object by a spectral reflectance function. The surface is associated with a non-negative and bounded reflectance function f u,v (λ), where (u, v) denotes the point on the surface that is projected to pixel (u, v), and λ is the wavelength. When this surface point is illuminated by a light source with spectral distribution L(λ), the spectral distribution of the reflected light, as observed by the camera, is given by L(λ)f u,v (λ). A digital multispectral camera or scanner is equipped with sensors of K different types, K ≥ 3, each measuring a differ- ent spectral sub-band. In consumer electronics, for instance, three spectral subbands, red, green and blue, are commonly used. Type k sensor has its spectral response function γ k (λ), 1 ≤ k ≤ K. Therefore, the sample value in spectral subband k at pixel position (u, v) is given by x k (u, v)=  Λ γ k (λ)L(λ)f u,v (λ)dλ (1) where Λ is the spectral range of the camera. Fig. 1. Illustration of image formation.