2996 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 59, NO. 11, NOVEMBER 2014 LMI-Based Estimation of Scene Points in Vision Systems with Generalized Cameras Graziano Chesi Abstract—This paper considers the problem of estimating the position of a scene point from its image projections onto gener- alized cameras, i.e., cameras that can be modeled by a spherical projection followed by a perspective one. The sought estimate is defined through a geometric criterion, specifically the minimiza- tion of the angles between the projections on the sphere of the available image points and the corresponding projections of the es- timate. A solution based on convex optimization with linear matrix inequalities (LMIs) is proposed for addressing this problem, which provides a candidate of the sought estimate. Moreover, a condition is provided for establishing exactness of the found candidate, i.e., establishing whether the found candidate is a minimizer of the considered geometric criterion. Index Terms—Generalized camera, linear matrix inequality (LMI), scene estimation, vision system. I. I NTRODUCTION It is well-known that estimating the position of a scene point from its image projections onto two or more cameras is a fundamental problem in vision systems, also known as multiple-view triangulation, see, e.g., [1]. Indeed, this problem has numerous applications, in particular an important one is in vision-based control of robots, see, e.g., [2], [13]. Due to image noise and calibration errors, only an estimate of the sought scene point can be obtained, whose accuracy depends on the criterion chosen to match the available image points with the image projections of the candidate estimate. For perspective cameras, numerous contributions can be found, which typically consider a geometric criterion for defining the estimate of the sought scene point. A commonly adopted geometric criterion is the minimization of the reprojection error in the L 2 norm, for which several solutions have been proposed. In [3], the authors show how the exact solution with two views can be obtained by computing the roots of a one-variable polynomial of degree six. For the case of three views, the exact solution is obtained in [4] by solving a system of polynomial equations through methods from computational commutative algebra. For these cases and others with more views, [5] derives a solution based on convex optimization exploiting the fundamental matrix. This paper considers multiple-view triangulation in a vision system with generalized cameras, i.e., cameras that can be modeled by a spher- ical projection followed by a perspective one. This class of cameras include the classic perspective cameras as well as non-perspective ones such as fisheye cameras. The sought estimate is defined through a geometric criterion, specifically the minimization of the angles Manuscript received September 23, 2012; revised June 26, 2013; accepted November 7, 2013. Date of publication August 28, 2014; date of current version October 21, 2014. This work was supported in part by the Research Grants Council of Hong Kong under Grant HKU711213E. Recommended by Associate Editor D. Regruto. The author is with the Department of Electrical and Electronic Engi- neering, The University of Hong Kong, Pokfulam, Hong Kong (e-mail: chesi@eee.hku.hk). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TAC.2014.2351657 between the projections on the sphere of the available image points and the corresponding projections of the estimate. A solution based on convex optimization with linear matrix inequalities (LMIs) is proposed for addressing this problem, which provides a candidate of the sought estimate. Moreover, a condition is provided for establishing exactness of the found candidate, i.e., establishing whether the found candidate is a minimizer of the considered geometric criterion. A conference version of this paper (without the exactness condition) appeared in [6]. II. PRELIMINARIES Notation: R: real numbers set; P n : set of vectors in R n with last entry equal to 1; x ∈ R 3 : scene point; O i ∈ R 3×3 : rotation matrix of the ith generalized camera; c i ∈ P 3 : center of the spherical projec- tion; ξ i ∈ R: distance between c i and the center of the perspective projection given by c i − ξ i O i e 3 ; K i ∈ R 3×3 : intrinsic parameters matrix; a i ∈ R 3 : spherical projection of x; b i ∈ P 3 : image projection of x in normalized coordinates; p i ∈ P 3 : image projection of x in pixel coordinates; ‖v‖∈ R: Euclidean norm of v; v T : transpose of v; e j ∈ R 3 : j th column of the 3 × 3 identity matrix; SOS: sum of squares of polynomials; subject to (s.t.). We consider n generalized cameras observing a scene point x. The ith generalized camera consists of a spherical projection followed by a perspective one. Such a camera can be modeled using the so called unified model, see, e.g., [7]. In this model, the spherical projection of x is a i = σ i (x) (1) where σ i (x)= O T i (x − c i ) ‖O T i (x − c i )‖ . (2) The image projection of x in normalized coordinates is b i = τ i (x) (3) where τ i (x)= 1 e T 3 a i + ξ i ‖a i ‖  e T 1 a i e T 2 a i e T 3 a i + ξ i ‖a i ‖  . (4) The image projection of x in pixel coordinates is p i = K i b i . (5) The dependence of p i on x is denoted as p i = φ i (x). (6) Let ˆ p i and ˆ φ i be the available estimates of p i and φ i (the latter defined by the available estimates ˆ K i , ˆ ξ i , ˆ O i , ˆ c i of K i ,ξ i , O i , c i ). The multiple-view triangulation problem for generalized cameras is given {(ˆ p i , φ i ) ,i =1,...,n} , estimate x. (7) 0018-9286 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.