A General Method for Errors-in-Variables Problems in Computer Vision Bogdan Matei and Peter Meer Electrical and Computer Engineering Department Rutgers University, Piscataway, NJ, 08854-8058, USA matei, meer@caip.rutgers.edu Abstract The Errors-in-Variables (EIV) model from statistics is often employed in computer vision though only rarely under this name. In an EIV model all the measurements are cor- rupted by noise while the a priori information is captured with a nonlinear constraint among the true (unknown) val- ues of these measurements. To estimate the model pa- rameters and the uncorrupted data, the constraint can be linearized, i.e., embedded in a higher dimensional space. We show that linearization introduces data-dependent (het- eroscedastic) noise and propose an iterative procedure, the heteroscedastic EIV (HEIV) estimator to obtain consistent estimates in the most general, multivariate case. Analyti- cal expressions for the covariances of the parameter esti- mates and corrected data points, a generic method for the enforcement of ancillary constraints arising from the un- derlying geometry are also given. The HEIV estimator min- imizes the first order approximation of the geometric dis- tances between the measurements and the true data points, and thus can be a substitute for the widely used Levenberg- Marquardt based direct solution of the original, nonlinear problem. The HEIV estimator has however the advantage of a weaker dependence on the initial solution and a faster convergence. In comparison to Kanatani’s renormaliza- tion paradigm (an earlier solution of the same problem) the HEIV estimator has more solid theoretical foundations which translate into better numerical behavior. We show that the HEIV estimator can provide an accurate solution to most 3D vision estimation tasks, and illustrate its per- formance through two case studies: calibration and the es- timation of the fundamental matrix. 1 Introduction In 3-D computer vision problems the geometry of the scene imposes various constraints on the available mea- surements: in uncalibrated stereo the matched points lie on corresponding epipolar lines, the motion field is assumed to satisfy the image brightness constancy, the 3-D points are associated with their image in the focal plane by a projec- tion matrix, etc. These constraints relate the true values of the measurements , to a parameter by a nonlinear -valued function (1) The support of the NSF grants IRI 99-87695 and IIS 98-72995 is gratefully acknowledged. In practice, instead of the true values, only the error cor- rupted measurements are available (2) where stands for general and independent prob- ability density with mean and covariance . Throughout the paper the subscript ‘o’ is used to distinguish between the true values and the noisy measurements. The expres- sion (1) with noise affecting the variables (2) is called an errors-in-variables model (EIV)[1]. When (1) is linear in it can be written as (3) The th element of , , is called a carrier or basis function. The process of decoupling the non-linear con- straint (1) into the carriers depending on the variables and the parameter is called linearization. Through lin- earization (1), a manifold in is mapped into a linear manifold in . Linearization is used often in vision tasks since it leads to an eigenvalue problem which can be eas- ily solved. However, the solution is only rarely satisfactory and has to be refined by a subsequent nonlinear procedure [6]. Traditionally, the linearized estimate is found as the smallest eigenvector of the scatter matrix defined with the noisy variables The solution, is called in statistics the Total (Orthog- onal) Least Squares (TLS) estimate. The TLS estimate is consistent (i.e. asymptotically convergences toward the true ) only when the noise affecting the carriers is inde- pendent and identical distributed (i.i.d.) [16, pp.33–43] (4) The unsatisfactory performance of is well known. For example, the eight-point algorithm yields useless esti- mates even in the presence of small noise. Normalization of the image points, however, significantly improves the accu- racy of the TLS estimate by reducing the condition number of [5]. We will show in the sequel that this normalization is only a partial remedy.