Long Reading Regressions are Accompanied by a P600-like Brain Potential Evidence from Simultaneous Recording of Eye Movements and ERPs Introduction References Department of Psychology, Potsdam University, Germany Department of Psychology, Humboldt University Berlin, Germany : ♦ Subjects, Stimuli & Task Eye Tracking & EEG Recording Methods Summary & Conclusions Results: Regression-locked potentials Regressive Olaf Dimigen, Werner Sommer, & Reinhold Kliegl : ♦ ♦ : EXPERIMENT 1 Potsdam Sentence Corpus I (N=30) EXPERIMENT 2 Potsdam Sentence Corpus III (N=24) Forward Regressive Forward Above. ERPimages. Every horizontal line depicts one fixation- locked EEG segment. Amplitude is color-coded, red indicates positive voltages. Epochs were sorted by the amplitude of the outgoing saccade. The figure shows that long regressive saccades, in particular, but not forward saccades, are followed by a sustained positivity at centroparietal electrodes. Grand average fixation-locked ERP at electrode Pz. The ERP is time-locked to fixation onset on a word N when followed either by a forward saccade (to next word) or a short (one word), medium (one word) or long regression. Topographical maps show the voltage difference between long regressions and forward saccades in the expected P600 time window. (two words or more) saccade amplitude Mean fixation durations (FD) for fixations followed by backwards or forwards saccades of different amplitude (Fixation N). FDs for neighboring fixations are also given. FDs that differ significantly from the mean FD (horizontal line) are highlighted (*). ! Regression rates and FDs were higher in Exp. 1 which involved syntactically more complex sentences. ! Longer FDs were observed two fixations prior to long regressions (orange arrows). Please note the dedicated symposium on EM-EEG co-registration on thursday Contact: Olaf Dimigen, Department of Psychology, University of Potsdam, PO Box 60 15 53, 14415 Potsdam, Phone: +49-331-977-2127, dimigen@uni-potsdam.de The Co-Registration Technique EXPERIMENT 1 30 subjects (20 female, 22.6 yrs) Read Potsdam Sentence Corpus I: ! 144 normal sentences ! 5 - 11 words ! Wide range of grammatical structures including, for example, non-canonical (object first) sentences 0.25° per character ! IView-X HiSpeed 240 Hz Tracker ! EEG from 33 electrodes (SR 250 Hz, bandpass 0.02-30 Hz, average reference) EXPERIMENT 2 24 subjects (18 female, 27.0 yrs) Read Potsdam Sentence Corpus III: ! 144 easily comprehensible sentence pairs ! 9 - 12 words ! Frequent grammatical structures ! Originally designed to study word recognition nd (cloze probability manipulation in 2 sentence) 0.45° per character ! read single, unrelated German sentences for understanding ! for natural reading flow, sentence presentation was triggered by subject’s eye movements ! comprehension questions after 25% (33%) of sentences ! IView-X HiSpeed 1250 Hz Tracker ! EEG from 64 electrodes (SR 500 Hz, bandpass DC-100 Hz, average reference) ! Tracking from right eye, binocular viewing ! Tracker calibration (13 point) controlled on every trial (0.5° max. allowed error) ! Systems synchronized via TTL pulses Three (non-exclusive) accounts have been proposed to explain reading regressions [1]: Visuomotor accounts assume that regressions mainly compensate for prior saccadic overshoot. Word identification accounts hold that regressions are due to incomplete identification of a previously fixated word. Comprehension accounts attribute regressions to difficulties in the ongoing sentence comprehension, in particular syntactic parsing problems. The three functional types of regressions should roughly correspond to short, medium, and long backwards saccades, respectively. Because long regressions are likely to reflect a reader's problem with parsing a particular sentence [5], we hypothesized that they should be associated with a P600 potential. The P600, or Syntactic Positive Shift, is a long-lasting centroparietal positivity that begins 200-400 ms and peaks 500-900 ms after presentation of a target word. It is typically regarded as an index of syntactic reanalysis and repair because it has been observed for syntactic violations, garden-path sentences, and other syntactically complex sentences. We expected no P600 for short (intraword) regressions as these should be primarily determined by low-level visuomotor factors. Hypotheses We reanalyzed data from two co-registration experiments. Gaze and EEG were recorded while subjects read from left to right, moving their eyes freely. Results: Fixation durations Correlation between selected EEG and horizontal eye track before/after correction. Non-significant correlations (grey) indicate the absence of residual artifact [1] Vitu, F. (2005). Visual extraction processes and regressive saccades in reading. In G. Underwood (Ed.), Cognitive processes in eye guidance. Oxford, NY: Oxford University Press. pp. 1-32. [2] Dimigen, O., Hohlfeld, A., Sommer, W.; Jacobs, A., Engbert, R. & Kliegl, W. (2005). Measuring ERPs during left-to-right reading, Poster presented at the IX International Conference on Cognitive Neuroscience, Havana, Cuba [3] Dimigen, O., Sommer, W., Hohlfeld, A., Jacobs, Engbert, R., Kliegl, R. (2006). Concurrent recording of EEG and gaze position: Measuring effects of word predictability during left-to-right reading of normal sentences. Journal of Cognitive Neuroscience, Supplement 224 [4] Dimigen, O., Schild, U., Hohlfeld, A., Berg, P. & Sommer, W. Auditory Language Comprehension During Saccadic Eye Movements: An Investigation With Event-Related Brain Potentials [5] Frazier & Rayner (1982). Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14, pp. 178-210 [6] Engbert & Kliegl (2003). Microsaccades uncover the orientation of visual attention. Vision Research, 43, pp. 1035-1045 (manuscript under review). In two experiments the EEG was recorded during left-to-right reading. We assumed that long - but not short - regressions would index a reader’s problem with ongoing sentence comprehension. We therefore hypothesized that long regressions should be accompanied by a P600 potential, an ERP correlate of syntactic reanalysis. ! In both experiments, long regressions were accompanied by a long-lasting centroparietal positivity that resembled the P600 in its scalp topography. ! ! We can rule out that the effects were merely due to residual ocular artifacts. We can largely exclude significant modulations through baseline effects and differential overlap. ! A smaller and delayed P600-like effect was also found for short regressions. This could mean that some proportion of short regressions is due to higher-level comprehension problems. Alternatively, it could indicate that late word identification problems. In summary, the present analyses established a direct relationship between oculomotor and neurophysiological of comprehension difficulty. Future application of the co-registration technique - with dedicated sentence materials - may help to clarify the cognitive processes that underlie regressive saccades as well as the P600. ! The effect was found despite the absence of syntactic violations and structural ambiguities in the sentences. However, simultaneous eye tracking presumably allowed us to identify individual cases of comprehension breakdown. The effect’s latency - beginning ~100 ms after saccade onset - is compatible with traditional P600 findings if one assumes that comprehension problems typically arose on the fixation preceding the regression or - as FDs suggest - one more fixation before that. positivities also appear for correlates EXPERIMENT 1 EXPERIMENT 2 19.1% regressions 13.7% regressions EEG recordings during normal vision are highly susceptible to signal distortion [2,3]. ´ Ocular EEG artifacts were corrected with Surrogate Multiple Source Correction (MSEC, Berg & Scherg, 1994; Ille et al., 2003). For each subject, eye calibration data and 3-D electrode locations were recorded. Source waveforms for artifact topographies were estimated in presence of a static fixed-dipole model of brain activity, thereby reducing the erroneous subtraction of spatially correlated brain activity. MSEC yields satisfying results, especially for horizontal saccades [2,3,4]. ! 62,767 reading fixations detected [6] ! Fixations occured in the first 800 ms after sentence onset and were discarded to avoid overlap with ERPs evoked by the sentence onset [3,4]. Sentence-final fixations were also not used. ! For each of 38,177 remaining fixations, EEG segments were cut time-locked to fixation and the outgoing saccade (100 ms baseline) ! Segments were averaged according to the direction and amplitude of the (outgoing) saccade About 10-15% of reading saccades are regressions that move the eyes back to earlier parts of the text. B In the present study, we investigated the neurophysiological correlates of regressive eye movements. Traditional r ecause regressions are difficult to induce experimentally, comparatively little is known about their determinants. eading experiments with event-related potentials (ERPs) do not allow for such analyses, because sentences are presented word-by-word and eye movements are strictly precluded. Consequently, readers cannot return to earlier words in the sentence. We used a recently developed method that allows for simultaneous recordings of eye movements and ERPs during natural, left-to-right reading [2,3,4] to establish a relationship between regressive saccades and the accompanying brain activity. Fixation-locked single epochs, sorted by outgoing saccade direction and amplitude Right. Grand-average saccade-locked ERP. For long regressions, the P600-like modulation began approx. 100 ms after saccade onset. Examples Den Ton gab der Künstler seinem Gehilfen (Exp. 1) Den ganzen Tag über konnte man die Raben krächzen hören. (Exp 1.) Er redete nur noch von seiner Glatze und nervte die anderen. (Exp 2.) Ocular Correction & Control of Overlap Claudia kann Salatsaucen mit viel Essig nicht ausstehen. 100 200 300 400 500 600 0 200 250 300 350 400 Horizontal eye position [pixel] 0 500 1000 1500 2000 2500 3000 3500 600 200 400 300 500 600 700 100 Eye position [pixel] Horizontal position Vertical position Left EOG Right EOG 387 202 176 282 193 164 193 143 176 408 42 100 µV Single Trial time [ms] Eye Position EOG [original] EEG [original] EOG [µV] EEG [µV] 100 µV EOG [corrected] EEG [corrected] Co-registration data for one sentence. An intra-word regression is highlighted. Co-registration setup Differential overlap. During normal vision, fixation-locked potentials are massively overlapped by potentials elicited by successive fixations (e.g. lambda waves). Naturally, the degree of overlap varies with systematic differences in fixation behavior. This may produce apparent condition differences. We used a simulation method that utilizes the temporal distribution of adjacent fixations to estimate the maximum impact of differential overlap on the ERP average. Saccade-locked Potential [µV] Large backward - one forward 400-800 ms * * * * * * * * * * * * * * * * Large backw. - forward Time window: 400-800 ms Large backw. - forward Time window: 400-800 ms 700 800 Pz Pz Saccade-locked ERP (EXP. 2)