Intelligent Vehicle Localization Using GPS, Compass, and Machine Vision Somphop Limsoonthrakul * , Matthew N. Dailey * , and Manukid Parnichkun † * Computer Science and Information Management † Mechatronics Asian Institute of Technology Klong Luang, Pathumthani, Thailand {somphop, mdailey, manukid}@ait.ac.th Abstract— Intelligent vehicles require accurate localization relative to a map to ensure safe travel. GPS sensors are among the most useful sensors for outdoor localization, but they still suffer from noise due to weather conditions, tree cover, and surrounding buildings or other structures. In this paper, to improve localization accuracy when GPS fails, we propose a sequential state estimation method that fuses data from a GPS device, an electronic compass, a video camera, and wheel encoders using a particle filter. We process images from the camera using a color histogram-based method to identify the road and non-road regions in the field of view in front of the vehicle. In two experiments, in simulation and on a real vehicle, we demonstrate that, compared to a standard extended Kalman filter not using image data, our method significantly improves lateral localization error during periods of GPS inaccuracy. I. INTRODUCTION Among the challenges involved in building a safe intel- ligent vehicle, localization is among the most important, because without precise knowledge of the vehicle’s location with respect to its surroundings, autonomy is impossible. Although GPS devices are extremely useful for localization, they are not sufficient by themselves, because satellite signal quality varies with weather and proximity to trees and buildings. The problem is especially acute in urban areas. Under these circumstances, accurate and robust localization relies critically on additional sensors or filtering techniques. There is a great deal of previous work using Kalman filters to improve GPS-based vehicle localization. Cooper et al. [1] propose an extended Kalman filter (EKF) model for vehicle navigation that incorporates a GPS device and an inertial navigation system (INS). Sadiadek et al. [2] improve the EKF for GPS/INS localization using fuzzy logic to adapt prediction and sensor noise strength. Thrapp et al. [3] and Bonnifait et al. [4] demonstrate EKFs that fuse GPS and odometry data, and Panzieri et al. [5] use an EKF to fuse GPS, INS, odometry, and laser scanner data. Machine vision techniques are also proving useful; Georgiev [6] presents a method using camera pose estimation to improve localization in urban environments when GPS performance becomes low. The method fuses GPS, odometry, and compass data using an EKF, but when the EKF’s uncertainty grows too large, monocular vision is used instead of the GPS signal. Agrawal and Konolige [7] present a localization method using stereo vision and GPS. In their work, visual odometry is fused with GPS measurements using an EKF. Although the EKF is efficient, linearizing the motion and sensor models can introduce inaccuracy, and its assump- tion of a Gaussian posterior distribution over vehicle poses means it can fail when the true distribution is non-Gaussian, especially when it is multi-modal [8], [9]. To solve this problem, Dellaert et al. introduce a localization method for indoor mobile robots using particle filter called Monte Carlo localization (MCL) [10] and apply the technique to the task of vision-based localization [11]. This work demonstrates the robustness of particle filters for localization with ambiguous sensor information. In our work, we complement a GPS device, compass, and wheel encoders with machine vision to address GPS inaccu- racy, and we use a particle filter to address linearization error and the limitations of the Gaussian posterior assumption. Our machine vision technique extracts road regions from the field of view in front of the vehicle. By comparing the observed road region with that expected based on a candidate vehicle position and a predefined map, we can compute the likelihood of the observation given the candidate vehicle position and, to the extent that the map and road region classification are accurate, thereby improve vehicle localization precision. II. ROAD REGION CLASSIFICATION We use a forward-pointing camera and road region clas- sification to improve localization accuracy. As shown in the flowchart in Fig. 1, we perform Gaussian smoothing to reduce image noise then classify each pixel in the image as road or non-road using a H-S color histogram. We then transform the classification results from the image plane to the (robot-relative) ground plane using a pre-calculated planar homography. The resulting robot-relative road region measurement vector can be used for vehicle localization. A. Hue-Saturation Histogram We use a 2D histogram to represent the distribution of road pixels’ color. Histograms are attractive because they are simple to calculate and easy to use. We use the hue and saturation components in the HSV color model [12] to determine whether each pixel is likely to be on the road or not because, unlike the RGB color space, HSV