Good Continuation in Layers: Shading flows, color flows, surfaces and shadows Ohad Ben-Shahar Computer Science Ben Gurion University Israel ben-shahar@cs.bgu.ac.il Andreas Glaser Applied Mathematics Yale University New Haven, CT 06520-8285 andreas.glaser@yale.edu Steven W. Zucker Computer Science Yale University New Haven, CT 06520-8285 steven.zucker@yale.edu Abstract We extend the concept of good continuation in a uni- form fashion from boundaries to shading, hue, and tex- ture. Each has the property that local measurements yield an orientation, which we explicitly establish for hue using geometric harmonic techniques. Good continuation arises in a geometric sense, because these orientations all vary smoothly in an appropriate sense. Thus they correspond to flows. Taken together they define a layered set of flows, in the sense the “horizontal” computations within each flow provide global consistency while “vertical” computations across flows enable the identification of shading and shad- owing and different types of edges. Evidence is reviewed that primate visual systems enjoy such an organization. 1 “...space and color are not distinct elements but, rather, are interdependent aspects of a unitary pro- cess of perceptual organization.” Kanizsa [17] 1. Introduction Image segmentation is normally taken to be that pro- cess of partitioning the image into a complete cover of non- overlapping regions, with the boundaries of these regions related to the (projected) boundaries of objects in the world. One source of complexity in this process is shadowing, by which image intensities vary both as a function of surface orientation (e.g., shading) and as a function of light sources (e.g., cast shadows). Land’s retinex theory [19] suggested one way to manage this complexity, by ascribing abrupt im- age changes to material (or reflectance) discontinuities and smooth gradient changes to lighting. This developed into the intrinsic image concept [30], which emphasized that surface properties, geometry, and lighting all map into the 1 Acknowledgements: Research supported by AFOSR, DARPA, ONR and the Toman and Frankel Funds from Ben-Gurion University. image, and suggested representing them separately as im- ages. Undoing this map clearly involves an inverse prob- lem, which requires a model of some sort. One possibility is to try to learn the context of every possible measurement, a type of pseudoinverse [28]. Here we extend the notion of context in a different way, by considering natural images such as those in Fig. 1. Notice how space,reflectance, and lighting conspire together. We seek to find a representation rich enough to support unwinding this. The first requirement for such a representation is that it be rich enough to capture the above phenomena. But un- like special purpose algorithms applicable in one situation (e.g., [16, 13]), our second requirement is that it be gen- eral purpose. That is, the information that it makes explicit must support computations for unraveling many such phe- nomena. We do not yet have a formal solution to this problem that we can prove is complete. Instead, and consistent with the goals of this Workshop, we develop an argument based on a neurobiological analogy, several steps of which have been formalized and are complete. The demonstrations in the final section of this paper involve phenomena beyond the current capability of any single existing algorithm, and provide counterexamples to many. Constructively, however, we submit that any final solution will have an intermedi- ate representation at least as rich as the one we describe. Thus we see the contribution of this Workshop submission as consisting of (i) an enlargement of the framework for per- ceptual organization informed by (ii) the rich foundation for perceptual organization in primate visual systems. The core of our argument is that good continuation ap- plies to several key domains: boundaries, intensity (shad- ing); hue; texture; saturation, and so on, all of which enjoy a certain differential geometric structure. It is this struc- ture that relates to the Gestalt notion of good continuation. Computationally we propose a layered representation— similar in spirit to intrinsic images [30]—but different in 0-7695-2646-2/06 $20.00 (c) 2006 IEEE