3D Model-Based Gaze Estimation in Natural Reading: a Systematic Error Correction Procedure based on Annotated Texts Andrea Mazzei * Shahram Eivazi † Youri Marko ‡ Frederic Kaplan § Pierre Dillenbourg ¶ School of Computer and Communication Sciences, ´ Ecole Polytechnique F´ ed´ erale de Lausanne School of Computing, University of Eastern Finland Abstract Studying natural reading and its underlying attention processes re- quires devices that are able to provide precise measurements of gaze without rendering the reading activity unnatural. In this paper we propose an eye tracking system that can be used to conduct analyses of reading behavior in low constrained experimental settings. The system is designed for dual-camera-based head-mounted eye track- ers and allows free head movements and note taking. The system is composed of three different modules. First, a 3D model-based gaze estimation method computes the reader’s gaze trajectory. Second, a document image retrieval algorithm is used to recognize document pages and extract annotations. Third, a systematic error correc- tion procedure is used to post-calibrate the system parameters and compensate for spatial drifts. The validation results show that the proposed method is capable of extracting reliable gaze data when reading in low constrained experimental conditions. CR Categories: H.5.2 [Information Interfaces and Presentation]: User Interfaces—Input devices and strategies; Keywords: 3D Gaze Estimation, Systematic Error Correction 1 Introduction and Motivation Eye movements in reading have been intensely studied on visual display units (VDU), using monitor-integrated eye tracking tech- nology, typically accompanied with chin- and head-rest. Most of these investigations were characterized by experimental controls and constraints, necessary to accurately measure the task perfor- mance. Designing low-constrained experimental setups to study eye movements in natural reading on printed instructional material still remains a challenge. Reliable gaze data can be acquired by lim- iting participants movements or the printed material manipulability. However, depending on the experimental focus, these impositions may negatively affect the concentration of the subjects. In this pa- per, we describe an eye tracking method to study eye movements in natural reading in low constrained experimental conditions. The system: 1) is capable of accurately measuring the readers’ gaze points, 2) does not require commercial hardware setups for acquir- ing gaze data, 3) does not require experimental settings that may interfere with reading activity and comprehension. Participants are allowed to read on paper, to freely move and to take notes while * andrea.mazzei@epfl.ch † shahram.eivazi@uef.fi ‡ youri.marko@me.com § frederic.kaplan@epfl.ch ¶ pierre.dillenbourg@epfl.ch reading. Under low-constrained experimental conditions, the gaze data extraction suffers from systematic errors and experimental in- convenients. We here propose a method to address this problem. 2 Related Works and Our Contribution The majority of the eye tracking studies of reading have been char- acterized by rigid and rigorous experimental constraints that, on one side, limit the participants’ movements and range of actions, and, on the other, improve the accuracy of data collection. In eye tracking studies based on display-units participants were usually encouraged to remain still at a fixed distance from the monitor while performing the task [Schotter et al. 2012; Rayner et al. 2010; Pieters and Wedel 2004]. In such experimental setting chin- and headrests were usu- ally introduced in order to minimize head movements. A similar setup was used to study spatial attention shifts on print advertise- ments [Pieters and Wedel 2004]. Eye movements in reading were also studied through a comparative analysis between visual display units and traditional paper [Velde and Grunau 2003]. In the paper reading condition, the authors at- tached the printed instructional material on a vertical monitor. All subjects were invited to use a head holder during the experiment. A similar experimental control was introduced by [Siegenthaler et al. 2011] in an analysis focusing on reading patterns and visual fatigue between e-reader displays and printed paper. Researchers also have used head mounted eye tracking setups for reading research. This type of setup allows free movements during the experimental task. However, in many studies, experimenters still felt appropriate to introduce a chin-rest and other forms of con- trol in the experimental setting [Rayner et al. 2008; Holsanova et al. 2009]. A head-mounted eye tracker coupled with a computer mon- itor was used by [Holsanova et al. 2009] to investigate how people look at the textual and pictorial contents of advertisements. The distance between the participant’s head and the monitor was con- trolled. Newspapers reading was studied using a head-mounted eye tracker [Holsanova et al. 2009]. In that study a newspaper was attached to the table in order to maintain the same position recorded dur- ing the calibration procedure. This setting gave some freedom of movements to the user but limited the manipulation of the text. When studying natural reading and its underlying cognitive pro- cesses from an eye tracking perspective, most of the experimen- tal constraints should be relaxed in order to avoid any interference with note taking practices and reader’s concentration. However, conducting an experiment under these conditions may introduce systematic errors into the data collection process. In addition, the experimental procedure should be calibration free and resistant to unexpected violations of experimental rules, which may require a recalibration of the eye tracking system. These experimental con- ditions may increase the risk of exposure to data inaccuracies that may change during the experimental time, and that can be quanti- fied as drift of gaze-point locations [Weigle and Banks 2008]. There are various techniques to post correct gaze data. For exam-