Fast optical flow using 3D shortest path techniques Changming Sun * CSIRO Mathematical and Information Sciences, Locked Bag 17, North Ryde, NSW 1670, Australia Abstract Optical flow or image motion estimation is important in the area of computer vision. This paper presents a fast and reliable optical flow algorithm which produces a dense optical flow map by using fast cross correlation and 3D shortest path techniques. Fast correlation is achieved by using the box-filtering technique which is invariant to the size of the correlation window. The motion for each scanline or each column of the input image is obtained from the correlation coefficient volume by finding the best 3D path using dynamic programming techniques rather than simply choosing the position that gives the maximum cross correlation coefficient. Sub-pixel accuracy is achieved by fitting the local correlation coefficients to a quadratic surface. Typical running time for a 256 £ 256 image is in the order of a few seconds on a 85 MHz computer. A variety of synthetic and real images have been tested, and good results have been obtained. q 2002 Elsevier Science B.V. All rights reserved. Keywords: Motion estimation; Optical flow; Image motion; Dynamic programming; 3D shortest path; Sub-pixel accuracy; Fast cross correlation; Similarity measure 1. Introduction Optical flow or image motion is the displacement of each image pixel in an image sequence. Image motion estimation is a fundamental issue in low-level vision and is used in many applications such as robot navigation, object tracking, image coding, and structure reconstruction. There are several types of methods for estimating image motion or optical flow [1]. These methods can be divided into correlation-based [2–5], energy-based [6], phase-based [7], and gradient-based [8–11] methods. Anandan [2] described a hierarchical framework for the determination of dense motion fields from a pair of images. It is based on a Laplacian pyramid and uses a coarse-to-fine matching strategy. Que ´not presented an algorithm for the computation of optical flow using orthogonal dynamic programming [12]. The principle is to minimise a sum of square of differences (SSD) between a pair of images. The dynamic programming is performed alternatively on horizontal and vertical image stripes while reducing the stripe spacing and width. Liu et al. presented a survey of different approaches toward the goal of higher performance and presented experimental studies on accuracy versus efficiency trade-offs [13]. Camus [14] presented algorithms that perform correlation search over time instead of over space to achieve linear performance. His method produces quantized motion estimates. Barron et al. investigated the accuracy, reliability and density of velocity measurements of a number of regularly cited optical flow techniques [1]. It is our intention in this paper to address some of the efficient and reliable implementation aspects of image motion estimation algorithms by using fast correlation and dynamic programming techniques. The method described in this paper is correlation-based. The novel aspects of our method are: (1) development of fast algorithms for the calculation of similarity or dissimilarity measures for motion estimation purposes; (2) the use of dynamic programming techniques to find a shortest path in the 3D correlation coefficient volume for each of the scanlines or column of the input image. This means the motion vectors are obtained by optimal matching for the entire scanline rather than searching for the maximum correlation coeffi- cient for each point independently. Sub-pixel accuracy motion estimates are obtained by using a simple formula for local extreme calculation. The rest of the paper is organised as follows. Section 2 reviews the box-filtering techniques and derives our fast correlation method for motion estimation. The detailed optical flow estimation method is described in Section 3. Section 4 shows the experimental results obtained using our fast image motion estimation method and several other regular cited methods applied to a variety of images. 0262-8856/02/$ - see front matter q 2002 Elsevier Science B.V. All rights reserved. PII: S0262-8856(02)00112-9 Image and Vision Computing 20 (2002) 981–991 www.elsevier.com/locate/imavis * Tel.: þ61-2-9325-3207; fax: þ 61-2-9325-3200. E-mail address: changming.sun@csiro.au (C. Sun).