International Journal of Computer Vision 39(3), 195–228, 2000 c  2000 Kluwer Academic Publishers. Manufactured in The Netherlands. Optimal Structure from Motion: Local Ambiguities and Global Estimates * ALESSANDRO CHIUSO Electronic Signals and Systems Research Laboratory, Department of Electrical Engineering, Washington University, One Brookings Dr. 1127, St.Louis, MO 63130, USA; Universit` a di Padova, Dipartamento di Elettronica ed Informatica, Via Gradenigo 6/a, 35100 Padova, Italy chiuso@dei.unipd.it ROGER BROCKETT Division of Applied Sciences, Harvard University, 29 Oxford St., Cambridge, MA 02139, USA brockett@hrl.harvard.edu STEFANO SOATTO Electronic Signals and Systems Research Laboratory, Department of Electrical Engineering, Washington University, One Brookings Dr. 1127, St.Louis, MO 63130, USA; Department of Computer Science, Henry Samueli School of Engineering, University of California, Los Angeles, CA 90095-1596, USA soatto@ucla.edu Received August 6, 1999; Revised February 25, 2000; Accepted May 22, 2000 Abstract. “Structure From Motion” (SFM) refers to the problem of estimating spatial properties of a three- dimensional scene from the motion of its projection onto a two-dimensional surface, such as the retina. We present an analysis of SFM which results in algorithms that are provably convergent and provably optimal with respect to a chosen norm. In particular, we cast SFM as the minimization of a high-dimensional quadratic cost function, and show how it is possible to reduce it to the minimization of a two-dimensional function whose stationary points are in one-to-one correspondence with those of the original cost function. As a consequence, we can plot the reduced cost function and characterize the configurations of structure and motion that result in local minima. As an example, we discuss two local minima that are associated with well-known visual illusions. Knowledge of the topology of the residual in the presence of such local minima allows us to formulate minimization algorithms that, in addition to provably converge to stationary points of the original cost function, can switch between different local extrema in order to converge to the global minimum, under suitable conditions. We also offer an experimental study of the distribution of the estimation error in the presence of noise in the measurements, and characterize the sensitivity of the algorithm using the structure of Fisher’s Information matrix. Keywords: structure from motion, alternating minimization, least-squares, sphere, optical flow, bilinear optimization Revised version registered as Technical Report ESSRL 99-002. Preliminary version (October 1997) appears in the Proceedings of the IEEE CVPR’98. 1. Introduction The problem of “Structure From Motion” (SFM) deals with extracting three-dimensional information about the environment from the motion of its projection onto