Int J Comput Vis (2008) 79: 13–30 DOI 10.1007/s11263-007-0087-3 Color Subspaces as Photometric Invariants Todd Zickler · Satya P. Mallick · David J. Kriegman · Peter N. Belhumeur Received: 9 August 2006 / Accepted: 10 September 2007 / Published online: 30 October 2007 © Springer Science+Business Media, LLC 2007 Abstract Complex reflectance phenomena such as specu- lar reflections confound many vision problems since they produce image ‘features’ that do not correspond directly to intrinsic surface properties such as shape and spectral re- flectance. A common approach to mitigate these effects is to explore functions of an image that are invariant to these photometric events. In this paper we describe a class of such invariants that result from exploiting color information in images of dichromatic surfaces. These invariants are derived from illuminant-dependent ‘subspaces’ of RGB color space, and they enable the application of Lambertian-based vi- sion techniques to a broad class of specular, non-Lambertian scenes. Using implementations of recent algorithms taken from the literature, we demonstrate the practical utility of these invariants for a wide variety of applications, including stereo, shape from shading, photometric stereo, material- based segmentation, and motion estimation. Keywords Photometric invariants · Shape invariants · Color spaces · Dichromatic reflection · Multispectral imaging · Surface reconstruction · Photometric stereo · Shape from shading · Stereo · Color-based segmentation · Color-based optical flow T. Zickler ( ) Harvard University, 33 Oxford St., Cambridge, MA 02138, USA e-mail: zickler@eecs.harvard.edu S.P. Mallick · D.J. Kriegman University of California, San Diego USA P.N. Belhumeur Columbia University, New York, NY 10027, USA 1 Introduction An image is the product of the shape, reflectance and illu- mination in a scene. For many visual tasks, we require only a subset of this information, and we wish to extract it in a manner that is insensitive to variations in the remaining ‘confounding’ scene properties. For 3D reconstruction, for example, we seek accurate estimates of shape, and we de- sign systems that are insensitive to variations in reflectance and illumination. One practical approach to these problems is to compute a function of the input images that is invariant to confound- ing scene properties but is discriminative with respect to de- sired scene information. A number of these invariants are described in the literature, with the simplest example be- ing a normalized-RGB image. For a Lambertian scene, the normalized RGB color vector at each pixel depends on the spectral reflectance of the corresponding surface patch but not its orientation with respect to a light source. It is a useful invariant for material-based segmentation. Like normalized-RGB, most existing invariants seek to isolate information about the material properties in a scene and are therefore designed to be invariant to local illumi- nation and viewing geometry. In contrast, this paper con- siders a class of invariants that deliberately preserve geom- etry information in a way that is invariant to specular re- flections. The proposed invariants provide direct access to surface shape information through diffuse shading effects, and since diffuse shading is often well approximated by the Lambertian model, they satisfy the ‘constant-brightness as- sumption’ underlying most approaches to stereo reconstruc- tion and structure-from-motion. In addition, these invariants provide access to surface normal information, which can be recovered using Lambertian-based photometric reconstruc- tion methods.