Omnidirectional Depth Recovery based on a Novel Stereo Sensor * Chuanjiang Luo 1, 2 , Lei He 1, 2 , Liancheng Su 1, 2 , Feng Zhu 1 , Yingming Hao 1 , Zelin Shi 1 1 Shenyang Institute of Automation, Chinese Academy of Science, Shenyang, China 2 Graduate School of Chinese Academy of Science, Beijing, China cjluo@ustc.edu * This work is supported by National Science Foundation of P.R. China under granted number 60575024. Abstract Omnidirectional depth map generation is very important for mobile robot navigation and action planning. In this paper, we present design of a novel stereo sensor and an algorithm to recovery dense 3D depth map for a mobile robot. The vision system is composed of a perspective camera and two hyperbolic mirrors. Once the system has been calibrated and two image points respectively projected by upper and nether mirrors are matched, the 3D coordinate of the space point can be acquired by means of triangulation. Our method can be divided into two steps. An initial depth map can be calculated using efficient dynamic programming technique. We adopt graph cut algorithm in the second step. With a relatively good initial map, the process of graph cut converges very fast. We also show the necessary modification to handle panoramic images, including deformed matching template, adaptable template scale. Experiment shows that this proposed vision system is feasible as a practical stereo sensor for accurate 3D map generation. 1. Introduction A catadioptric vision system using diverse mirrors has been a popular means to get panoramic images [1], which contains a full horizontal field of view (FOV). This wide view is ideal for three-dimensional vision tasks such as motion estimation, localization, obstacle detection and mobile robots navigation. Omni- directional stereo is a suitable sensing method for such tasks because it can acquire images and ranges of surrounding areas simultaneously. For omnidiretional stereo vision, an obvious method is to use two (or more) cameras instead of each conventional camera [2]-[5]. Such two-camera (or more-camera) stereo systems are relatively costly and complicated compared to single camera stereo systems. Omnidirectional stereo based on a double-lobed mirror and a single camera was developed in [6]-[9]. A double lobed mirror is a coaxial mirror pair, where the centers of both mirrors are collinear with the camera axis, and the mirrors have a profile radially symmetric around this axis. This arrangement has the advantage to produce two panoramic views of the scene in a single image. But the disadvantage of this method is the relatively small baseline it provides. Since the two mirrors are so close together, the effective baseline for stereo calculation is quite small. We have developed a novel omnidirectional stereo vision optical device (OSVOD) based on a common perspective camera coupled with two hyperbolic mirrors, which are separately fixed inside a glass cylinder. As the separation between the two mirrors provides much enlarged baseline, in our system, the baseline length is about 200mm, the precision has improved correspondingly (Fig 1). The coaxial configuration of the camera and the two hyperbolic mirrors makes the epipolar line radially collinear, which makes the system free of the search process for complex epipolar curve in stereo matching (Fig 3). The OSVOD is mounted on top of mobile robot (Fig 2) looking downwards, at a height of approximately 0.75 meters above the ground plane. Albeit the calculative precision of triangulation was improved theoretically due to the wider baseline, more complex and difficult stereo matching problem should be brought on because of the wider disparity space that exists and more serious image distortion. A major aim of this paper is to propose an integrated framework, which mainly focuses on stereo matching to enhance the performance for depth map regeneration. Since our primary goal is to propose a precise and suitable algorithm for stereo matching to satisfy the reliability requirement via an omnidirectional stereo vision