AGGREGATED MAPPING OF DRIVER ATTENTION FROM MATCHED OPTICAL FLOW Roland Perko, Michael Schwarz, Lucas Paletta JOANNEUM RESEARCH Forschungsgesellschaft mbH DIGITAL – Institute for Information and Communication Technologies, Steyrergasse 17, Graz, Austria ABSTRACT Eye tracking research about driver distraction, applied to real world driving tasks, has so far demanded a massive amount of manual intervention, for the annotation of hundreds of hours of head camera videos. We present a novel methodology that enables the automated integration of arbitrary gaze localizations onto a visual object and its local surrounding in order to draw heat maps directly onto the environment. Gaze locations are tracked in video frames of the eye tracking glasses’ head camera, within the regions about the driver’s environment, using optical flow methodology. The high robustness and accuracy of the optical flow based tracking - measured with a residual mean error of ≈0.3 pixels on sequences, captured and verified in 576 individual trials - enables a fully automated estimation of the driver’s attention processes, for example in the context of roadside objects. We present results from a typical driver distraction study and visualize the performance of fully aggregated human attention behavior. Index Terms— Driver attention analysis, optical flow, tracking, geometric transformation, attention mapping. 1. INTRODUCTION Driver distraction has for decades been a central focus of eye tracking research and applications [1]. Driver distraction is one form of driver inattention and is claimed to be a contributing factor in over half of inattention crashes [2,3]. Eye tracking studies on driver attention have mainly been focused on studies in artificial environments, such as, in driver simulators [4-7]. Analyzing the focus of attention in real world driving conditions from eye tracking data usually involves massive human resources for the manual annotation of tens or hundreds of hours of head camera videos, in particular if the experiments involve a substantial number of drivers and trials [4,5,6]. We present a novel approach that enables the automated aggregation of arbitrary gaze localizations from multiple drivers towards a reference road infrastructure and its environment. Gaze allocations in the driver’s environment are tracked with optical flow based computer vision methodology in the head camera video sequence and finally projected onto a selected key video frame. Gaze distributions of different drivers’ videos are aggregated by matching the respective key video frames. This technology enables for the first time, up to our knowledge, to estimate the driver’s distraction patterns from drivers’ eye tracking videos, with respect to the environment, in an automated manner. We applied the approach in a driver distraction study that would usually involve massive manual annotation including unpredictable error margins from human interaction. We demonstrate the successful approach with results from the fully automated aggregation of point-of- regards (PORs, [8]), computation of dwell time and looking behavior on the target infrastructure. Figure 1 depicts the sensor setup used in the driver study: Eye Tracking Glasses (ETG) capture the eye movement behavior in a natural way, other sensors can be used for further data analysis that is not in the scope of this work. 2. RELATED WORK Eye tracking studies on driver attention have mainly been focused on studies in artificial environments, such as, in driver simulators [7]. In [9] authors studied the impact of increased cognitive load while driving to drivers’ visual searching behavior. Authors of [10] examined drivers’ eye glance behavior away from the road scene ahead during car following. In [11] driver distraction was investigated in the context of the effects of video and static advertising on human eye movements. The presented work is highly related to the one of [12]. They presented a complete system that reads speed signs in real-time, compares the driver’s gaze, and provides immediate feedback if it appears the sign has been missed by the driver. Figure 1: Study on driver distraction using eye tracking glasses.