Driver Alert State and Fatigue Detection by Salient Points Analysis Javier Jim´ enez-Pinto, Miguel Torres-Torriti Dept. of Electrical Engineering Pontifi cia Universidad Catolica de Chile Santiago, Chile jejimenp@puc.cl, mtorrest@ing.puc.cl Abstract—Assessing a driver’s state of awarness and fatigue is especially important to reduce the number of traffic accidents often involving bus and truck drivers, who must work during several hours under monotonous road conditions. Two main challenges arise in resolving the state of alert: first, the system must be capable of detecting the driver’s face location; secondly, the driver’s facial cues, such as blinking, yawning, and eyebrow rising must be recognized. Our approach combines the well- known Viola-Jones face detector with motion analysis of Shi- Tomasi salient features within the face to determine the driver’s state of alert. The location of the eyes and blinking are cues whose detection is also important. To this end, the proposed method takes advantage of the high reflectivity of the retina to near infrared illumination employing a camera with an 850 nm wavelength filter. Motion analysis of the salient points, in particular cluster mass centers and spatial distribution, has proved successful in determining the driver’s state of alert. Index Terms—alert state assessment, fatigue detection, drowsi- ness detection, driver assistance, IR eye tracking, yawning detec- tion, eyebrow rising detection. I. I NTRODUCTION Road accidents take a heavy fi nancial and social toll on national economies. The economic cost of traffi c incidents is estimated to be 1% of gross national product in low-income countries, 1.5% in middle-income countries and 2% in high- income countries, totaling a global cost of US$518 billion per year [15]. Without appropriate actions to improve education, law enforcement, infrastructure and technology, a global in- crease of 67% is expected by year 2020. Although global statistics about accidents attributed to fatigue and drowsiness are not available because in many countries such details are not reported or classifi ed, the number of incidents in high- income countries is not negligible. For example, the NHTSA reported as much as 56.000 accidents back in 1996 [1], which increased to 1.35 million in 2002 [19]. The latter is about 0.7% of the reported accidents. These fi gures are even larger if other accidents related to the driver s state-of-alert, such as distracted driving (3.5%) and cell-phone use accidents (0.1%), are included. Some other alarming accident statistics due to fatigue, stress or distraction can be found in [7], [16]. In this context, developing systems to monitor a driver s state of awareness is fundamental. Several studies exist about physiological cues that can be used to assess a driver s awareness. Some techniques can be invasive, but fortunately, there are many behavioral changes that provide visual cues, namely, eye-blinking frequency and closure percentage over some window of time (PERCLOS), yawn frequency, head movement, eye-gaze, among other facial expressions. Hence, a variety of systems based on computer vision techniques have been proposed. A summary is presented in Table I, in which the approaches have been grouped ac- cording to the technique employed to extract the area of the head. A large number of them, e.g. [3], [10], [17], [18], [22], [24], employ color-based segmentation techniques, while a sig- nifi cant number of other approaches relies on the Viola-Jones detector, e.g. [7], [9], [13], [21], [27], [28], as shown in Table I. The comparison of the approaches is not easy because results are reported in different non-standard ways. Moreover, some approaches only track the eyes, while other focus on particular facial cues, such as yawning [6], [18]. However, it is possible to say that approaches based on color analysis are limited by illumination conditions and often cannot be applied at night. This has motivated some researchers to use near infrared (IR) cameras, exploiting the retinas high refl ectivity to 850 nm wavelength illumination [8], [14]. Some approaches employ neural-networks to extract the head and main features [4], [23], while other rely on a variety of template matching schemes [2], [6], [5], [11], [26]. This paper presents an approach to determine if the driver is becoming drowsy or inattentive by analyzing eye blinking, eyebrow rising and yawning cues. The novelty of the approach is in that in addition to the Viola-Jones face detector, it employs the Lucas-Kanade algorithm to track Shi-Tomasi salient points around the mouth and eyes. Such salient points do not only provide information useful to track the motion of the head, but also to determine if the driver is yawning or rising eyebrows. The group of salient points also provides a reference relative to which the pupil can be located when the camera does not perceive the glare of the retina under IR illumination. The approach is described in greater detail in the next section. The results presented in section III show that the approach yields high detection rates with a low level of false alarms and good tracking rates. The main conclusions are presented in section IV. II. PROPOSED APPROACH Defi ne an image frame at sampling instant k as I k : p → I k (p), p ∈ N 2 , i.e. the assignment of intensity values I (p) to Proceedings of the 2009 IEEE International Conference on Systems, Man, and Cybernetics San Antonio, TX, USA - October 2009 978-1-4244-2794-9/09/$25.00 ©2009 IEEE 455