1 VISION-BASED LATERAL CONTROL OF VEHICLES J. Koˇ seck´ a, R. Blasi, C. J. Taylor and J. Malik Department of Electrical Engineering and Computer Sciences University of California at Berkeley Berkeley, CA 947, USA. email: janak,blasirs,camillo,malik@cs.berkeley.edu Keywords: lateral control, computer vision, autonomous systems ABSTRACT We describe the problem of automated steering us- ing computer vision, focusing the analysis and de- sign on appropriate lateral controllers. We inves- tigate various static feedback strategies where the measurements obtained from vision, namely offset from the centerline at some lookahead distance and the angle between the road tangent and the orienta- tion of the vehicle at some lookahead distance, are directly used for control. Within this setting we ex- plore the role of lookahead, its relation to the vision processing delay, the longitudinal velocity and road geometry. Results from ongoing experiments with our autonomous vehicle system are presented along with simulation results. INTRODUCTION This paper addresses the problem of designing con- trol systems for steering a motor vehicle along a highway using the output from a video camera mounted inside the vehicle. Several aspects of this problem have been examined extensively in the past, both in the psychophysics literature [7] as well as in control theoretic studies. In the kinematic setting there have been several attempts to formulate the vision-based steering task in the image plane [11], [3]. A stability analysis was provided for an om- nidirectional mobile base trying to align itself with a straight road [3] or nonholonomic mobile base following an arbitrary ground analytic curve [8]. The controllers designed based on kinematic models were either tested in simulation or in experiments at speeds below 20 m/s. However at higher speeds dy- namic effects are quite pertinent and the need for a dynamic model becomes apparent. The control problem in a dynamic setting, using measurements ahead of the vehicle, has been ex- plored by [9] who proposed a constant control law proportional to the offset from the centerline at a look-ahead distance. Their analysis showed that closed loop stability for this controller can always be obtained by increasing the look-ahead distance to an appropriate value. Dickmanns, et al [2] developed a Kalman-filter based observer which estimated the state of the vehicle with respect to the road along with the road geometry and used the estimate for full state feedback using a pole-placement method. Further studies typically use a small and fixed look- ahead distance and the control objective is formu- lated either at the look-ahead distance [5] or at the center of gravity of the vehicle [10]. An analysis of the tradeoffs between the performance requirements and robustness of the system can be found in [5]. This paper will discuss the problem of automated steering using computer vision, focusing on the analysis of the problem and controller design choice. We propose a static feedback strategy where the measurements obtained from vision, namely offset from the centerline and angle between the road tan- gent and the orientation of the vehicle at some look- ahead distance, are directly used for control. Within this setting we explore the role of lookahead, its rela- tion to the vision processing delay, longitudinal ve- locity and road geometry. MODELING The dynamics of the vehicle can be described by a detailed 6-DOF nonlinear model [10]. Since it is possible to decouple the longitudinal and lateral dy- namics, a linearized model of the lateral vehicle dy- namics is used for controller design. The linearized model of the vehicle retains only lateral and yaw dy- namics, assumes small steering angles and a linear tire model, and is parameterized by the current lon-