Introduction A model-based tracking algorithm, such as Worrall [1], Gennery [2], Koller [3], Harris [4], Schneiderman [5] and Lowel [6], consists of the following repeated stages in one frame of the captured image sequence: (1) matching between a known projected model and the image feature; (2) measurement of the difference between the model fea- ture and the corresponding image feature; (3) pose estimate; and (4) updated model projection onto the image plane. The first two stages are dependent on images, while the last two are not. Some mathematical methods are used in the third stage, such as least squares method, Newton’s method, statistic analysis and Kalman filtering. In stage four, computer graphics under perspective projection is applied. A complete tracking cycle starts from stage one with projected model from the last cycle. First, some low level image processings, mainly the edge detection [7], are used to detect image features which corre- spond to specific model edges. The image features to be detected are usually contrast edges. Since the edge detection requires high computing band width, Lowe [6,8] and Gennery [2] used additional hardware dealing with edge detection. Stephens [9] employed a transputer to perform some separate work in the whole tracking algorlthm. When the image edges are detected whether globally or locally, the best matched one for a specific model edge is considered as the matched fea- ture. Deriche [10] gave a good representation for image line, and the Mahalanobis distance is used for matching in a neigh- borhood of a projected model edge. Lowe [6,8] selected the 1077-2014/99/030167 + 21 $30.00/0 © 1999 Academic Press Real-Time 3D Motion Tracking with Known Geometric Models n this paper a new model-based tracking algorithm is proposed for real-time performance. The matching process includes two aspects of: (1) feature extraction using local minimum energy and I(2) global matching of known 3D models against the projected features. The algorithm is robust to change in lighting and background. The small motion hypothesis was used for fitting of feature energy which is defined as the negative absolute value of the edge strength. An autoregressive AR(1) model is employed for detecting incorrect matches in terms of the feature energy. We have found a new invari- ance-based method to eliminate false matches caused by strong shadow or occlusion. The invariance is the ratio of trigonometric functions of the angles formed by a polygon. In order to calculate the vertices of the object surface in an image, regression technique in terms of matched features is efficient in our approach. A linear least squares method and the orthonormal rotation matrix are used for motion esti- mation and pose update of the six degrees of freedom. Also, an Extended Kalman Filter is introduced to guarantee a smooth motion estimation and prediction. © 1999 Academic Press Zheng Li and Han Wang* School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798 Email: hw@ntu.edu.sg Real-Time Imaging 5, 167–187 (1999) Article No. rtim.1997.0111, available online at http://www.idealibrary.com on * Corresponding author.