ADVANCED SITUATIONAL AWARENESS AND OBSTACLE DETECTION USING A MONOCULAR CAMERA Abhijit Bhoite, Nikhil Beke, Sashank Nanduri, Tim Duffy, Myra Torres. Impact Technologies, LLC, 300 Canal View Boulevard, Rochester, NY 14623, United States ABSTRACT This paper presents a modular approach for a high resolution monocular camera based system to detect, track, and display potential obstacles and navigational threats to soldiers and operators for manned and unmanned ground vehicles. This approach enhances situational awareness by integrating obstacle detection and motion tracking algorithms with virtual pan-zoom-tilt (VPZT) techniques, enabling soldiers to interactively view an arbitrary region of interest (ROI) at the highest captured resolution. Depth determination from a single imager is challenging and an approach for depth information, along with size and motion information is developed to assign a threat level to each obstacle. Index Terms - Obstacle detection, Motion tracking, Depth perception, VPZT. 1. INTRODUCTION Passive sensors, such as high-resolution monocular cameras, have become a key technology that soldiers rely on in the field due to their simplicity and relatively low cost and weight [1]. However, the application of these high-resolution imaging technologies upon military ground vehicles is hindered by the limited resolution of their display panels and the bandwidth of their content delivery systems. To overcome this problem, the high resolution video is down sampled to match the resolution of the display screen. Thus, the soldiers are unable to view a local region of interest (ROI) at the “highest captured resolution” [2]. One solution to this problem is a pan-zoom-tilt (PZT) camera system, which allows for camera movement and zoom. For field applications, this solution is unsuitable as it introduces unnecessary complexities and numerous points of failure. A fixed camera solution solves the same problem with the use of virtual pan–zoom-tilt algorithms (VPZT). VPZT algorithms allow soldiers to pan, tilt, and zoom through a high-resolution video feed in real-time without mechanical components, enabling them to view any arbitrary ROI at the highest captured resolution. In some situations, soldiers do not have any direct visual access to the environment in which the vehicle is being operated. Under such circumstances of total reliance on indirect sensory input, early detection of obstacles and distant navigational threats are of critical importance. Many algorithms have been proposed in the past to detect obstacles on a moving platform [1], [3], [4]. However, there no known technology that can integrate VPZT algorithms with obstacle detection and threat assessment to increase the situational awareness of the soldier [2]. The paper presents a modular system approach to improve situational awareness in the field. As seen in Figure 1, the process starts with image capture using a camera system that can resolve obstacles at a distance of 200-500m. Environmental effects present many constraints to object classification. Fog and rain reduce the visibility and range of the surroundings due to absorption and scattering of natural or artificial illumination by fog particles. The captured image is thus degraded, losing its contrast and color fidelity. An image quality enhancement approach comprising of histogram equalization and contrast enhancement addresses and overcomes the image degradation due to environmental factors. The post-processed image is then evaluated for road and obstacle detection in real time. Once obstacles are identified, the region encompassing the obstacles is input to the motion tracking algorithms. The captured image sequence is used to distinguish stationary and moving obstacles. The relative distance of the obstacle from the vehicle is determined using a depth perception algorithm discussed in section 2.4. All stationary and moving obstacles are continuously analyzed for their threat level with respect to navigation of the vehicle depending on their size, speed, direction of motion, and distance from the vehicle. The key challenge is the ability to detect depth using monocular video. Most conventional approaches for depth perception use stereo vision or range based sensors. This approach for depth perception relies on optical geometry of the camera and motion detection to present relative distance. 2. TECHNICAL APPROACH 2.1 Obstacle detection Obstacles in the context of navigation are defined as stationary or moving objects that may impede the motion of the vehicle. All pixels in the image are classified as belonging to either an obstacle or background. Any pixel that visually differs from the background is considered an obstacle. An edge based segmentation technique for obstacle detection emphasizes enclosed boundaries corresponding to the obstacle while neglecting spurious edges due to shadows and occlusions. Figure 2 details the edge based segmentation 978-1-4244-9300-5/10/$26.00 ©2010 IEEE 30