Covariant Derivatives and Vision Todor Georgiev Adobe Systems, 345 Park Ave, W10-124, San Jose, CA 95110, USA Abstract. We describe a new theoretical approach to Image Process- ing and Vision. Expressed in mathemetical terminology, in our formalism image space is a fibre bundle, and the image itself is the graph of a sec- tion on it. This mathematical model has advantages to the conventional view of the image as a function on the plane: Based on the new method we are able to do image processing of the image as viewed by the human visual system, which includes adaptation and perceptual correctness of the results. Our formalism is invariant to relighting and handles seam- lessly illumination change. It also explains simultaneous contrast visual illusions, which are intrinsically related to the new covariant approach. Examples include Poisson image editing, Inpainting, gradient domain HDR compression, and others. 1 Introduction It is a known fact that the human visual system does change the physical con- tents (the pixels) of the perceived image. We do not see luminance or color as they are, measured by pixel values. Higher pixel values do not always appear brighter, but perceived brightness depends on surrounding pixels. A popular example is the simultaneous contrast illusion [14, 15], where two identical gray patches appear different because of different surroundings. As a result of adap- tation, difference in lightness (perceived brightness) does not equal difference in pixel value. Some of those effects were already well understood in the general framework of Land’s Retinex theory [8]. Researchers like Horn [6], Koenderink [7], and others, have later contributed to the theory. Petitot [18] has proposed rigorous “neurogeometry” description of visual contours in images based on Rie- mannian connections. Following the above authors, we introduce the geometric idea of Image Space as fibred manifold and provide an understanding on how image processing in Image Space differs from image processing in the conventional approach, where images are simply functions on the plane. Compared to [18], we model lightness perception instead of contours, and we are using general linear connections that are not Riemannian. Viewing Image Space as Fibred Manifold allows us to do image processing on “the image as we see it”, and not on the physical image as function of x, y. Based on this construction, image processing is invariant with respect to certain specific changes in pixel values, for example due to change of lighting. A shadow on the image should not change the result of image processing operations, even A. Leonardis, H. Bischof, and A. Pinz (Eds.): ECCV 2006, Part IV, LNCS 3954, pp. 56–69, 2006. c  Springer-Verlag Berlin Heidelberg 2006