Y.-S. Ho (Ed.): PSIVT 2011, Part II, LNCS 7088, pp. 335–346, 2011. © Springer-Verlag Berlin Heidelberg 2011 Recovering Depth Map from Video with Moving Objects Hsiao-Wei Chen and Shang-Hong Lai Computer Science, National Tsing Hua University, No. 101, Section 2, Kuang-Fu Road, Hsinchu, Taiwan 30013, R.O.C. wei@hotmail.com, lai@cs.nthu.edu.tw Abstract. In this paper, we propose a novel approach to reconstructing depth map from a video sequence, which not only considers geometry coherence but also temporal coherence. Most of the previous methods of reconstructing depth map from video are based on the assumption of rigid motion, thus they cannot provide satisfactory depth estimation for regions with moving objects. In this work, we develop a depth estimation algorithm that detects regions of moving objects and recover the depth map in a Markov Random Field framework. We first apply SIFT matching across frames in the video sequence and compute the camera parameters for all frames and the 3D positions of the SIFT feature points via structure from motion. Then, the 3D depths at these SIFT points are propagated to the whole image based on image over-segmentation to construct an initial depth map. Then the depth values for the segments with large re- projection errors are refined by minimizing the corresponding re-projection errors. In addition, we detect the area of moving objects from the remaining pixels with large re-projection errors. In the final step, we optimize the depth map estimation in a Markov random filed framework. Some experimental results are shown to demonstrate improved depth estimation results of the proposed algorithm. Keywords: Depth map recovery, structure from motion, Markov random field. 1 Introduction In recent years, 3D imaging industry emerges rapidly. More and more movies take advantage of advanced 3D technology to reconstruct 3D motion or 3D scene for movie production or post-processing, which would produce amazing visual effect as if it were really happening. With the rapid development of the 3D movies, the 3D television also follows the trend. In each mall, we can see 3D TV exhibition in the spotlight. The digital camera also starts to include some 3D imaging function that supports 3D stereo images, 3D panorama shooting mode, and so on. In addition to these products, 3D digital photo frame and 3D digital printing are also the new related 3D products, and there will be more 3D products in the future. For those 3D display monitors, it needs the image files that contain 3D information. If we can convert the captured 2D images or videos into the 3D format, we can view the images and videos with 3D display. Thus, the 2D-to-3D media conversion is very important for the development of the 3D imaging industry.