Enforcing Integrability for Surface Reconstruction Algorithms Using Belief Propagation in Graphical Models Nemanja Petrovic Ira Cohen Brendan J. Frey Ralf Koetter Thomas S. Huang Beckman Institute University of Illinois Urbana, IL 61801 Abstract Accurate calculation of the three dimensional shape of an object is one of the classic research areas of computer vi- sion. Many of the existing methods are based on surface normal estimation, and subsequent integration of surface gradients. In general, these methods do not produce valid surface due to violation of surface integrability. We intro- duce a new method for shape reconstruction by integration of valid surface gradient maps. The essence of the new ap- proach is in the strict enforcement of the surface integra- bility via belief propagation across graphical model. The graphical model is selected in such a way to extract in- formation from underlying, possibly noisy, surface gradient estimators, utilize the surface integrability constraint, and produce the maximum a-posteriori estimate of a valid sur- face. We demonstrate the algorithm for two classic shape reconstruction techniques; shape-from-shading and photo- metric stereo. On a set of real and synthetic examples the new approach is shown to be fast and accurate, in the sense that shape can be rendered even in the presence of high lev- els of noise and sharp occlusion boundaries. 1 Introduction Recovering 3D shape of objects, classified as shape-from- X techniques, is a classic and fundamental computer vision research area. Shape-from-X refers to the recovery of shape from stereo, motion, texture, shading, etc. In this paper we focus our attention on the shape recovery from images of a static object made under different lighting conditions, also known as photometric stereo (PMS) and shape-from- shading (SFS). Shape-from-shading was first studied by Horn in the early 70’s and since then there has been a substantial literature dealing with the problem. The principal idea in SFS and PMS algorithms is to invert the mapping from surface gra- dients to image intensity. The mapping is referred to as the reflectance map. The reflectance map combines informa- tion about the light source, surface material and viewing geometry to form the generally non-linear relationship be- tween the image intensity and surface gradients. Because the mapping is not invertible locally using a single mea- surement of image intensity, SFS algorithms solve for the surface gradients using several different methods. A class of algorithms use additional constraints on the surface and minimize an energy function [6, 9]. Other algorithms use propagation of shape information from reference points to iteratively solve for the surface gradients, local surface as- sumptions or linear approximations to the reflectance map. A good reference comparing and describing different meth- ods can be found in [19]. Photometric stereo is a shape-from-shading algorithm us- ing several images to invert the reflectance map. It has been first introduced by Woodham in 1980 [17], and has been an ongoing research problem in the computer vision com- munity. The basic algorithm for PMS estimates the surface gradients locally for each pixel without using global con- straints. Among the more advanced methods are those that use local confidence measures to account for surface inter- reflections and shadowing [18, 10]. Not all SFS methods are based on surface normal recon- struction followed by gradient integration. There are meth- ods that directly estimate the absolute height [4]. For these methods there is no need to estimate the normal, and no need to enforce integrability. However, normal based meth- ods are still being used, and we think that problem of accu- rate reconstruction still remained unsolved at large. In this paper we describe the use of belief propagation in factor graphs to enforce integrability on valid surfaces. Belief propagation in factor graphs has been shown to be successful in dealing with complicated problems involving functions of many variables that could be factored into local functions. The message passing in factor graphs, referred to as the sum-product algorithm, is an efficient algorithm for computing marginals or posterior of variables in the graph. Various algorithms such as the forward-backward algorithm, the Viterbi algorithm, Kalman filtering, and Pearl’s belief propagation in Bayesian networks can be viewed as instances of the sum-product algorithm [12]. In cases of loopy factor