Looking-in and Looking-out Vision for Urban Intelligent Assistance: Estimation of Driver Attentive State and Dynamic Surround for Safe Merging and Braking Ashish Tawari 1 , Member, Sayanan Sivaraman 2 , Member, Mohan Manubhai Trivedi 1 , Fellow, IEEE, Trevor Shannon 2 and Mario Tippelhofer 2 Abstract— This paper details the research, development, and demonstrations of real-world systems intended to assist the driver in urban environments, as part of the Urban Intelligent Assist (UIA) research initiative. A 3-year collaboration between Audi AG, Volkswagen Group of America Electronics Research Laboratory, and UC San Diego, the driver assistance portion of the UIA project focuses on two main use cases of vital importance in urban driving. The first, Driver Attention Guard, applies novel computer vision and machine learning research for accurately tracking the driver’s head position and rotation using an array of cameras. The system then infers the driver’s focus of attention, alerting the driver and engaging safety systems in case of extended driver inattention. The second application, Merge and Lane Change Assist, applies a novel probabilistic compact representation of the on-road environ- ment, fusing data from a variety of sensor modalities. The system then computes safe and low-cost merge and lane-change maneuver recommendations. It communicates desired speeds to the driver via Head-up Display, when the driver touches the blinker, indicating his desired lane. The fully-implemented systems, complete with HMI, were demonstrated to the public and press in San Francisco in January of 2014. I. INTRODUCTION In 2012, there were 5.6 million police-reported motor ve- hicle crashes in the United States with over 33, 000 fatalities, a 3.3-percent increase from the previous year [1]. Fatalities in urban crashes alone increased by 4.9-percentage. The urban driving environment presents an array of challenges to the driver. Navigation and path planning are made more difficult due not only to the “urban canyon,” but also local traffic states, construction, other factors - like events as well as other road users such as pedestrians, cyclists, and other vehicles. The Urban Intelligent Assist project was formulated to address these challenges, to help drivers during in an in- creasingly urbanized world. A 3-year research collaboration between Audi AG, VW Electronics Research Laboratory, and 3 leading California universities, the goals of the project include enhancement of the drivers convenience, comfort, and safety in the urban driving environment. Various univer- sities have worked on applications including routing, parking prediction, and driver assistance tasks. In this work, we detail 1 Ashish Tawari Mohan M. Trivedi with the Laboratory of Intelligent and Safe Automobiles (http://cvrr.ucsd.edu/lisa/), University of California, San Diego, La Jolla, CA, USA {atawari, mtrivedi}@ucsd.edu 2 Sayanan Sivaraman, Trevor Shannon and Mario Tippelhofer with Volkswagen Group of America Electronics Research Laboratory in Bel- mont, CA, USA ssivaram@ucsd.edu {trevor.shannon, Mario.Tippelhofer}@vw.com two driver assistance applications, which observe the driver, and the on-road environment. Driving tasks involve decision making in three differ- ent time-scales: strategic, tactical and/or critical. These timescales have been proposed by prior researchers [2], [3], [4] in driver modeling. The strategically-planned maneuvers are associated with long-term time scale, minutes or hours of prior planning, and are often motivated by destination goal and sometimes comfort as well such as route planning, parking choice etc. Tactical, or short-term, timescales are on the order of seconds, and encompass many successive critical operations. Tactically-planned maneuvers are usually motivated by an uncomfortable situation, or occasionally by a recently modi- fied destination goal of the driver, for example, lane changes, turns, stops, upcoming exit etc. Finally, the critical or oper- ational timescale, on the order of hundreds of milliseconds, is the shortest possible timescale for human interaction and are a generally a result of a driver’s desire to remain safe in following the rules of the road (posted speed limit, road curves, etc.), while carefully operating the vehicle within its limits. Each of these time-scales presents its own research challenges. The tactical and operational time scale time- scales are of particular interest to us, as in former, the driver still has time to react to the feedback from an assistance system and in later, the system can intervene appropriately to provide autonomous assistance. In this paper, we report recently accomplished project and its findings including system design, results and demonstra- tions. In particular, we present two components of the sys- tem, called as, 1) Drive Attention Guard and 2) Merge/Lane Change Recommendation system. Attention Guard provides early detection of driver distraction by continuously monitor- ing driver and surround traffic situation. It further provides proactive countermeasures that maintain the ego-vehicle a safe following distance from lead vehicle and stay within marked lane lines. Merge/Lane Change Recommendation system monitors full surround of the ego-vehicle using range sensors to not only detect that a car is in the blind spot, but also if a vehicle approaching or being approached is entering the space in the adjacent lane. The system continuously adjusts predictions about the surround vehicles’ trajectory in order to assist driver during stress inducing situations like lane change and merging to a highway. 2014 IEEE Intelligent Vehicles Symposium (IV) June 8-11, 2014. Dearborn, Michigan, USA 978-1-4799-3637-3/14/$31.00 ©2014 IEEE 115