Motion and structure from multiple cues; image motion, shading flow, and stereo disparity q Ali Khamene a, * and Shahriar Negahdaripour b a Imaging and Visualization Department, Siemens Corporate Research, 755 College Road East, Princeton, NJ 08540, USA b Electrical and Computer Engineering Department, University of Miami, Coral Gables, FL 33124, USA Received 2 January 2002; accepted 25 February 2003 Abstract We investigate the recovery of 3-D motion and structure from the stereo images of a sta- tionary environment. A Kalman filter-based framework is proposed for the reconstruction of 3-D structure from multiple visual cues, through the integration of image motion and stereo disparity with the shading flow that is induced by the rotational motion of the source. A com- mon scenario involves the coupled motion of artificial source(s) and stereo cameras that are installed on mobile submersible vehicles. Utilizing shading flow with the image motion leads to devising a more robust 3-D motion estimation algorithm, in addition to the important role in depth recovery/refinement by constraining the local surface gradients. Collectively, use of multiple cues enhances robustness with respect to perturbation in any of the cues. Results of experiments with synthetic and real imagery are presented to evaluate the performance of the proposed algorithm, including the construction of a composite 3-D depth map of an underwater scene from a sequence of stereo pairs. Ó 2003 Elsevier Science (USA). All rights reserved. Keywords: Motion estimation; Stereo; 3-D reconstruction; Image mosaicking Computer Vision and Image Understanding 90 (2003) 99–127 www.elsevier.com/locate/cviu q Research done at the UMÕs Underwater Vision and Imaging Laboratory with support from the NSF under Grant BES-9711528. * Corresponding author. E-mail addresses: ali.khamene@scr.siemens.com (A. Khamene), shahriar@miami.edu (S. Negahdar- ipour). 1077-3142/03/$ - see front matter Ó 2003 Elsevier Science (USA). All rights reserved. doi:10.1016/S1077-3142(03)00028-6