Interactive Optimization of 3D Shape and 2D Correspondence Using Multiple Geometric Constraints via POCS Zhaohui Sun, A. Murat Tekalp, Nassir Navab, and Visvanathan Ramesh AbstractÐThe traditional approach of handling motion tracking and structure from motion SFM) independently in successive steps exhibits inherent limitations in terms of achievable precision and incorporation of prior geometric constraints about the scene. This paper proposes a projections onto convex sets POCS) framework for iterative refinement of the measurement matrix in the well-known factorization method to incorporate multiple geometric constraints about the scene, thereby improving the accuracy of both 2D feature point tracking and 3D structure estimates. Regularities in the scene, such as points on line and plane and parallel lines and planes, that can be interactively identified and marked at each POCS iteration, enforce rank and parallelism constraints on appropriately defined local measurement matrices, one for each constraint. The POCS framework allows for the integration of the information in each of these local measurement matrices into a single measurement matrix that is ªclosestº to the initial observed measurement matrix in Frobenius norm, which is then factored in the usual manner. Experimental results demonstrate that the proposed interactive POCS framework consistently improves both 2D correspondences and 3D shape/ motion estimates and similar results can not be achieved by enforcing these constraints as either post or preprocessing. Index TermsÐGeometric constrained shape recovery, structure from motion, interactive optimization, the factorization approach, projections onto convex sets POCS). æ 1INTRODUCTION RECOVERY of 3D scene structure and motion/camera parameters from a sequence of images under appropriate assumptions and constraints has long been studied in computer vision. However, it still remains as a challenging problem, limited mainly by the accuracy of 2D correspondence estimation and the validity of the underlying modeling assumptions. Among the most popular schemes that have been proposed so far are the factorization method, sequential or recursive methods, andbundleadjustmentmethod.Thefactorizationmethod[1]seeks the optimal 3D shape and motion parameters under orthographic projection, by singular value decomposition of a given measure- ment matrix subject to rank 3 constraint. It was later extended to more general camera models, such as weak perspective [2], para- perspective [3], affine [4], and projective factorization [5]. It was also extended to include uncertainty handling [6], [7], [8]. Sequential or recursive methods [9], [10], [11] compute and filter 3D shape and motion estimates by Kalman filtering such that the reprojections have the minimum distance to the 2D tracking) observations. Bundle adjustment method [12], [13], [14] usually involves a nonlinear optimization formulation that aims to minimizethereprojectionerrorintheimageplane.Acomprehen- sive review of other methods for SFM can be found in [15]. In all abovemethods,2Dcorrespondenceestimationtracking)andSFM havebeendoneindependentlyinsuccessivesteps,andnoexplicit geometric constraints are enforced, other than an implicit rank 3 constraint in the case of the factorization method. Usually,theaccuracyof3Dshapeaswellas2Dcorrespondence estimates can be significantly improved by enforcing known geometric constraints, such as collinearity, coplanarity, and parallelism, on the scene structure. The Facade system [16] employs prior 3D models of building primitives along with measuredcorrespondencesforaccuratereconstructionandrender- ing of architectures. In [17], geometric constraints such as parallelism and orthogonality are used to recover architectural models from the perspective information using a limited number of still images with proper calibration. Both [16] and [17] are not formulatedinaSFMframework.Itisdesirabletoincorporatesuch prior information in the SFM algorithm rather than utilize it as postprocessing or preprocessing stages. The bundle adjustment method has been extended to include multiple constraints [18], [14]; however, nonlinear optimization methods may not always converge to the global optimum. Inthispaper,weproposeaprojectionsontoconvexsetsPOCS) frameworkasopposedtoanonlinearoptimizationframework)to iteratively compute a measurement matrix that is ªclosestº to the observedunconstrainedmeasurementmatrixbutsatisfyingmulti- ple geometric constraints. Geometric constraints are interactively specified at each POCS iteration. The basic premise of our approach is that geometric constraints that are interactively marked by the user are more reliable than 2D feature correspon- dences that are estimated by automatic motion tracking. Hence, if there is an inconsistency between the 2D correspondences in the noisy measurement matrix) and the user specified constraints, the proposed procedure modifies the noisy measurement matrix to makeitconsistentwiththegeometricconstraints.Intheproposed framework, a local measurement matrix is defined for each geometric constraint whose elements are manipulated to fit the constraint. The POCS framework allows for the integration of the information in each of these local measurement matrices into a single measurement matrix that is ºclosestº to the initial observed measurementmatrixinFrobeniusnorm,whichisthenfactoredin the usual manner. The proposed method is most suitable for off- lineprocessing,suchasinenvironmentmodelingfrommonocular video because of the interactivity required). In the following, a mathematicalstatementoftheproblemisgiveninSection2,where geometric constraints such as points on a line and plane, and on parallellinesandplanesareformulatedasclosed,convexsets.The POCSmethodforiterativeoptimizationofthe3Destimatessubject to multiple geometric constraints is described in Section 3. We demonstrate how the geometric relationship can be improved by interactive enforcement of the local constraints in Section 4 and conclude the paper in Section 5. 2PROBLEM FORMULATION We formulate the problem of interactive 3D shape recovery optimization via multiple geometric constraints in the framework of POCS [19], [20], [21], where M geometric constraints are modeled as closed, convex sets C i , i  1; ... ;M, in the Hilbert space of measurement matrices ~ W. In the POCS framework, the initial observed) measurement matrix which may not satisfy all constraints)isiterativelyprojectedontotheconvexsets,yieldinga final measurement matrix ªclosestº to the initial matrix in Frobenius norm, but satisfying all constraints when the intersec- tionofallconstraintsetsisnonempty.Inthefollowing,wedefine 562 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 4, APRIL 2002 . Z. Sun is with Imaging Science Technology, Kodak Research Laboratories, Building 65, 1700 Dewey Ave., Rochester, NY 14650-1816. E-mail: sun@image.kodak.com. . A.M. Tekalp is with the Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY 14627-0126. E-mail: tekalp@ece.rochester.edu. . N. Navab and V. Ramesh are with the Imaging and Visualization Department, Siemens Corporate Research, Princeton, NJ 08540. E-mail: {navab, rameshv}@scr.siemens.com. Manuscript received 28 June 2000; revised 28 Mar. 2001; accepted 31 July 2001. Recommended for acceptance by A. Shashua. For information on obtaining reprints of this article, please send e-mail to: tpami@computer.org, and reference IEEECS Log Number 112351. 0162-8828/02/$17.00 ß 2002 IEEE