Optical Flow Estimation using Laplacian Mesh Energy Wenbin Li Darren Cosker Matthew Brown Rui Tang MTRC, Department of Computer Science University of Bath, BA2 7AY, UK {w.li,d.p.cosker,m.brown,r.tang}@bath.ac.uk Abstract In this paper we present a novel non-rigid optical flow algorithm for dense image correspondence and non-rigid registration. The algorithm uses a unique Laplacian Mesh Energy term to encourage local smoothness whilst simul- taneously preserving non-rigid deformation. Laplacian de- formation approaches have become popular in graphics re- search as they enable mesh deformations to preserve local surface shape. In this work we propose a novel Laplacian Mesh Energy formula to ensure such sensible local defor- mations between image pairs. We express this wholly with- in the optical flow optimization, and show its application in a novel coarse-to-fine pyramidal approach. Our algorith- m achieves the state-of-the-art performance in all trials on the Garg et al. dataset, and top tier performance on the Middlebury evaluation. 1. Introduction Optical flow estimation is an important area of computer vision research. Current algorithms can broadly be clas- sified into two categories – variational methods and dis- crete optimization methods. The former is a continuous approach [5, 6, 18] to estimate optical flow based on modi- fications of Horn and Schunck’s framework proposed in [9]. Such approaches can provide high subpixel accuracy but may be limited by minimization of the non-convex energy function. The latter [4, 14] is based on combinatorial opti- mization algorithms such as min-cut and max-flow, which can recover non-convex energy functions and multiple local minima but may suffer from discretization artifacts, e.g. the optical flow field boundary is aligned with the coordinate axes. One desirable property of optical flow techniques is to preserve local image detail and also handle non-rigid image deformations. Under such deformations, the preservation of local detail is particularly important. Garg et al. [7] impose this by maintaining correlations between 2D trajectories of different points on a non-rigid surface using a variational framework. Pizarro et al. [12] propose a feature matching approach based on local surface smoothness, and also show particular application to non-rigidly deforming objects. In computer graphics research, a common requirement is that surface meshes are globally editable, but capable of maintaining local details under mesh deformations. In or- der to provide a flexible representation to allow computation and preservation of such details, Laplacian mesh structures have previously been described [13, 11]. Such schemes im- pose constraints in differential Laplacian coordinates calcu- lated upon groups of triangles associated with each vertex. Meshes have previously been used in optical flow estima- tion [8]. However, this is to reduce processing complexity as opposed to specifically imposing smoothness. In this paper we present an variational optical flow mod- el which introduces a novel discrete energy based on Lapla- cian Mesh Deformation. Such deformation approaches are widely applied in graphics research, particularly for pre- serving local details [13, 11]. In our work we propose that the same concept, i.e. that of an underlying mesh which pe- nalizes local movements and preserves smooth global ones, can be of great use for optical flow and tracking. Constraints on the local deformations expressed in Laplacian coordi- nates encourage local regularity of the mesh whilst allow- ing global non-rigidity. Our algorithm applies a mesh to an image with a resolution up to one vertex per pixel. The Laplacian Mesh Energy is described as an additional term for the energy function, and can be applied in a straight- forward manner using our proposed minimization strategy. In addition, a novel coarse-to-fine approach is described for overcoming the loss of small optical flow details during its propagation between adjacent pyramid levels. We evaluate our approach on the widely recognized Mid- dlebury dataset [2] as well as the publicly available non- rigid data set proposed by Garg et al. [7]. Our approach provides excellent performance ranked in the top tier of the Middlebury evaluation 1 , and either outperforms or shows comparable accuracy against the leading publicly available non-rigid approaches when evaluated on the non-rigid data set of Garg et al. 1 http://vision.middlebury.edu/flow/eval/results/results-e1.php 2013 IEEE Conference on Computer Vision and Pattern Recognition 1063-6919/13 $26.00 © 2013 IEEE DOI 10.1109/CVPR.2013.315 2433 2013 IEEE Conference on Computer Vision and Pattern Recognition 1063-6919/13 $26.00 © 2013 IEEE DOI 10.1109/CVPR.2013.315 2433 2013 IEEE Conference on Computer Vision and Pattern Recognition 1063-6919/13 $26.00 © 2013 IEEE DOI 10.1109/CVPR.2013.315 2435