VEHICLE STATE ESTIMATION USING VISION AND INERTIAL MEASUREMENTS Vishisht Gupta and Sean Brennan Department of Mechanical and Nuclear Engineering, Pennsylvania State University, University Park, PA Abstract: A novel method for estimating vehicle roll, pitch and yaw using machine vision and inertial sensors is presented that is based on matching images captured from an on-vehicle camera to a rendered representation of the surrounding terrain obtained from an on-board map database. United States Geographical Survey Digital Elevation Maps (DEMs) were used to create a 3D topology map of the geography surrounding the vehicle, and it is assumed in this work that large segments of the surrounding terrain are visible, particularly the horizon lines. The horizon lines seen in the captured video from the vehicle are compared to the horizon lines obtained from a rendered geography, allowing absolute comparisons between rendered and actual scene in roll, pitch and yaw. A kinematic Kalman filter modeling an inertial navigation system then uses the scene matching to generate filtered estimates of orientation. Experiments using an instrumented vehicle operating at the test track of the Pennsylvania Transportation Institute were performed to check the validity of the method, and the results reveal a very close match between the vision-based estimates of orientation versus those from a high-quality GPS/INS system. Keywords: Terrain Aided Localization, Inertial Navigation, Kalman Filter. 1. INTRODUCTION The high sampling rates needed for vehicle chassis stability control, autonomous navigation and fault detection are primarily realized with the use of inertial sensors such as yaw rate gyros and lateral accelerometers. Because these sensors rely on inte- gration to obtain position and orientation, biases can be introduced in these state estimates. These biases can grow unless external measurements of position and orientation, or assumptions about vehicle behavior, are applied. Common methods to correct for the drift of in- ertial sensors include fusing this data with GPS measurements (Bevly [2004] and Ryu and Gerdes [2002]), and/or wheel speed sensor data (Dis- sanayake and Whyte [2001] and Chung and Boren- stein [2001]. However, a novel corrector method is suggested by early missile and submarine guid- ance systems (Golden [1980],Denism and Roberts [1989],Hostetler and Andreas [1983]) where exter- nal terrain features obtained by radar and sonar are compared to expectations or virtual observa- tions from databases stored a priori. This tech- nique is called terrain aided localization (Madha- van [2004]) and is the subject of this work. Among sensors needed to observe terrain features for terrain aided localization of a vehicle, vi- sion systems are a particularly compelling choice. First, visibility of key features within the sur- rounding scene can be assumed as a precondition for driving. Vision sensors readily identify fea- tures that impose stark geometric constraints on a world model, for example horizons, road edges and