International Journal of Computer Vision 45(2), 129–156, 2001 c  2001 Kluwer Academic Publishers. Manufactured in The Netherlands. Three-Dimensional Reconstruction of Points and Lines with Unknown Correspondence across Images * Y.-Q. CHENG AND X.G. WANG Robotics Institute, NSH, Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, PA 15213, USA R.T. COLLINS, E.M. RISEMAN AND A.R. HANSON Department of Computer Science, University of Massachusetts, Amherst, MA 01003, USA riseman@cs.umass.edu Received July 7, 2000; Accepted July 6, 2001 Abstract. Three-dimensional reconstruction from a set of images is an important and difficult problem in computer vision. In this paper, we address the problem of determining image feature correspondences while simultaneously reconstructing the corresponding 3D features, given the camera poses of disparate monocular views. First, two new affinity measures are presented that capture the degree to which candidate features from different images consistently represent the projection of the same 3D point or 3D line. An affinity measure for point features in two different views is defined with respect to their distance from a hypothetical projected 3D pseudo-intersection point. Similarly, an affinity measure for 2D image line segments across three views is defined with respect to a 3D pseudo-intersection line. These affinity measures provide a foundation for determining unknown correspondences using weighted bipartite graphs representing candidate point and line matches across different images. As a result of this graph representation, a standard graph-theoretic algorithm can provide an optimal, simultaneous matching and triangulation of points across two views, and lines across three views. Experimental results on synthetic and real data demonstrate the effectiveness of the approach. Keywords: feature correspondence matching, point/line affinity measure, weighted bipartite graph matching, maximum network flow 1. Introduction Three-dimensional model acquisition remains a very active research area in computer vision. One of the key questions is how to reconstruct accurate 3D mod- els from a set of calibrated 2D images via multi- image triangulation. The basic principles involved in 3D model acquisition are feature correspondence de- termination and triangulation, with the two commonly ∗ This work was funded by the RADIUS project under DARPA/Army TEC contract number DACA76-92-C-0041, the DARPA/TACOM project under contract number DAAE07-91-C-R035, and the Na- tional Science Foundation (NSF) under grant number CDA8922572. used types of image features being points and lines. Usually, 2D features are extracted first, such as cor- ners, curvature points, and lines from each image. Then, the correspondence of these features is established be- tween any pair of images, usually referred to as “the correspondence problem.” Finally, the 3D structure is triangulated from these 2D correspondences. Many reconstruction papers assume the correspon- dence problem has been solved (Aggarwal et al., 1981; Bedekar and Haralick, 1996; Gruen and Baltsavias, 1998; Ito and Aloimonos, 1988; Lee et al., 1986; Lee and Joshi, 1993; Lessard et al., 1989). Unfortunately, in many applications, this information is not avail- able and mechanisms to achieve correspondence are