Bilateral Filtering-Based Optical Flow Estimation with Occlusion Detection Jiangjian Xiao, Hui Cheng, Harpreet Sawhney, Cen Rao, and Michael Isnardi Sarnoff Corporation {jxiao, hcheng, hsawhney, crao, misnardi}@sarnoff.com Abstract. Using the variational approaches to estimate optical flow between two frames, the flow discontinuities between different motion fields are usually not distinguished even when an anisotropic diffusion operator is applied. In this pa- per, we propose a multi-cue driven adaptive bilateral filter to regularize the flow computation, which is able to achieve the smoothly varied optical flow field with highly desirable motion discontinuities. First, we separate the traditional one-step variational updating model into a two-step filtering-based updating model. Then, employing our occlusion detector, we reformulate the energy functional of op- tical flow estimation by explicitly introducing an occlusion term to balance the energy loss due to the occlusion or mismatches. Furthermore, based on the two- step updating framework, a novel multi-cue driven bilateral filter is proposed to substitute the original anisotropic diffusion process, and it is able to adaptively control the diffusion process according to the occlusion detection, image inten- sity dissimilarity, and motion dissimilarity. After applying our approach on vari- ous video sources (movie and TV) in the presence of occlusion, motion blurring, non-rigid deformation, and weak textureness, we generate a spatial-coherent flow field between each pair of input frames and detect more accurate flow disconti- nuities along the motion boundaries. 1 Introduction Optical flow estimation has been investigated by computer vision researchers for a long time [10, 12, 19, 3, 4, 11, 1, 6]. Given two input images, how to compute accurate optical flow is still challenging problem in computer vision especially when the images have severe occlusion and non-rigid motion. The basic idea of optical flow computation is maintaining the brightness constancy assumption, which relates the image gradient, ▽I , to the components u and v of the local optical flow. Since this is an ill-posed problem, some additional constraints are required to regularize the motion field during the flow estimation. From the well-known aperture phenomenon, a larger region of integration is more preferable to produce stable motion estimation but it may be more likely contain multiple motions in this region and cannot handle non-rigid deformation very well [4]. Therefore, the fundamental problem of optical flow estimation is still how to design an effective anisotropic smoothness regularizer, such that it not only maintains variable spatial coherence inside each piecewise-smooth region but also keeps accurate flow discontinuities at the motion boundaries. A. Leonardis, H. Bischof, and A. Pinz (Eds.): ECCV 2006, Part I, LNCS 3951, pp. 211–224, 2006. c  Springer-Verlag Berlin Heidelberg 2006