IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 10, OCTOBER 2005 1647 Image Up-Sampling Using Total-Variation Regularization With a New Observation Model Hussein A. Aly, Member, IEEE, and Eric Dubois, Fellow, IEEE Abstract—This paper presents a new formulation of the regu- larized image up-sampling problem that incorporates models of the image acquisition and display processes. We give a new ana- lytic perspective that justifies the use of total-variation regulariza- tion from a signal processing perspective, based on an analysis that specifies the requirements of edge-directed filtering. This approach leads to a new data fidelity term that has been coupled with a total-variation regularizer to yield our objective function. This ob- jective function is minimized using a level-sets motion that is based on the level-set method, with two types of motion that interact si- multaneously. A new choice of these motions leads to a stable solu- tion scheme that has a unique minimum. One aspect of the human visual system, perceptual uniformity, is treated in accordance with the linear nature of the data fidelity term. The method was imple- mented and has been verified to provide improved results, yielding crisp edges without introducing ringing or other artifacts. Index Terms—Data fidelity, gamma correction, image up-sam- pling, interpolation, level-sets motion (LSM), observation model, regularization, total variation. I. INTRODUCTION D IGITAL-image magnification with higher perceived res- olution is of great interest for many applications, such as law enforcement and surveillance, standards conversions for broadcasting, printing, aerial- and satellite-image zooming, and texture mapping in computer graphics. In such applications, a continuous real-world scene is projected by an ideal (pin-hole) optical system onto an image plane and cropped to a rectangle . The resulting continuous image is acquired by a physical camera to produce a digital lower resolution (LR) image (i.e., lower than desired) defined on a lattice (following the notation of [1], [2]). This camera, including the actual optical compo- nent, is modeled as shown in Fig. 1 as a continuous-space filter followed by ideal sampling on . The problem dealt with in this paper is, given the still LR image , obtain the best per- ceived higher resolution (HR) image defined on a denser sam- pling lattice . Here, we hypothesize that an ideal HR image defined on a denser lattice can be obtained in principle di- rectly from by a virtual camera, which can similarly be mod- Manuscript received March 13, 2004; revised September 14, 2004. This work was supported in part by the Ministry of Defence, Egypt, and in part by the Nat- ural Sciences and Engineering Research Council of Canada. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Thierry Blu. H. A. Aly is with the Ministry of Defence, Cairo, Egypt (e-mail: haly@ieee.org). E. Dubois is with the School of Information Technology and Engineering, University of Ottawa, Ottawa, ON K1N 6N5 Canada (e-mail: edubois@uot- tawa.ca). Digital Object Identifier 10.1109/TIP.2005.851684 eled by filtering with a continuous-space filter followed by ideal sampling on . Our goal is then to obtain an estimate of denoted by with the highest perceptual quality. Many solution methods for the image magnification problem exist in the literature, with a broad quality range. We can categorize these solution methods into model-based and non-model-based ones. Non-model-based methods use linear or nonlinear (adaptive) interpolation. Linear interpolators range from straightforward pixel repeat [which is also called zero-order hold (ZOH)], bilinear, or bicubic [3] interpolation to embedding in spline kernel spaces [4]–[7]. Simple linear methods suffer from staircasing (blocking) of oblique edges, blurring of the object boundaries and texture, and ringing in smooth regions that are adjacent to edges. Splines produce better quality up-sampled images than those obtained by straightforward linear interpolators, but are known to produce oscillatory edges with significant ringing near them. Analysis of this effect based on image isophotes (iso-intensity contours) can be found in [8]. These drawbacks of the linear methods have led to research in adaptive methods, whose goal is to preserve the sharpness of strong edges in the up-sampled image . They adapt the interpolation method used according to the edges given in the LR image and, hence, are generally called edge-directed interpolation [9]–[12]. Another nonlinear approach exploits local correlation of the samples without explicitly extracting edges by defining a local metric that de- termines the local participation weight of each sample of in interpolating a sample of [13]–[17]. Adaptive methods can produce clearly visible edges as compared to those produced by the linear class, enhancing the overall perceived quality of the resulting images. However, this class has the drawbacks of relying on good edge estimation or local correlation and every implementation is sensitive to the orientations of the edges. De- spite the fact that the sharpness of the edges is being enhanced by adaptive methods, the crispness of long edges is not well handled and they are usually wavy, and blotching occurs on the boundaries of edges. Furthermore, there is no solid theoretical base that unifies the realization of the approaches of this class and every approach stands on its own. Model-based image up-sampling methods rely on modeling the imaging processes and using sophisticated regularization methods describing a priori constraints. According to the for- mulation of Fig. 1, it can be shown that can be related to by down-sampling, as shown in Fig. 2 [18], or, more generally, by arbitrary rate conversion in the case of . Without loss of generality, we only consider the case of ; the more general case can be handled by an up-sampling step to an inter- 1057-7149/$20.00 © 2005 IEEE