Knowing Where and When to Look in a Complex Time-Critical Dual Task Anthony J. Hornof and Yunfeng Zhang Computer and Information Science University of Oregon Eugene, OR 97403 USA hornof@cs.uoregon.edu Tim Halverson Air Force Research Laboratory 6030 South Kent St. Mesa, AZ 85212 USA thalvers@cs.uoregon.edu ABSTRACT High-performance time-critical human-computer systems for use in multitasking environments need to be designed with an understanding of how human cognitive strategies can coordinate the interleaving of perceptual, memory, and motor processes. This paper presents eye movement data collected for a tightly-controlled, highly-motivated, practiced dual task. In the primary task, participants tracked a target with a joystick. In the secondary task, participants keyed-in responses to objects moving across a radar. Task manipulations included: whether the currently unattended task’s stimuli were visible; presence or absence of auditory cues; and the number of radar objects to classify in succession. Eye movement analysis demonstrates that participants developed fundamentally different strategies to perform the two tasks in parallel under the various task manipulations. The experiment provides a rich set of data to guide the development of models and theory of human performance in complex dual task scenarios. Author Keywords Auditory displays, cognitive strategies, eye tracking, time- critical tasks, multimodal interfaces, multitasking, visual displays. ACM Classification Keywords H.5.2. [Information Interfaces and Presentation]: User Interfaces – Evaluation/methodology, graphical user interfaces (GUI), screen design, theory and methods; H.1.2 [Models and Principles]: User/Machine Systems; I.2.0 [Artificial Intelligence]: General – Cognitive simulation. INTRODUCTION An important task domain for human-computer interaction (HCI) is high-performance, time-critical, real-time systems intended to support multiple tasks in parallel. This domain is of great interest to designers of devices that might be used in parallel with a life-critical task such as systems intended for air-traffic control, in-car navigation, and emergency evacuation. Such devices present users with the challenge of accomplishing a secondary task, such as providing manual responses to visual information as quickly as possible so that they can return to the primary critical task as quickly as possible. Psychological theories, such as those embedded in cognitive architectures—computational frameworks for building simulations of human performance—used for predicting performance in HCI tasks accommodate the potential for extensive overlapping of perceptual and motor processing across tasks and subtasks. Cognitive models built (often using these architectures) to simulate human information processing rely on such overlapping to explain aspects of human performance. Yet there are limited data available to measure and demonstrate the extent to which such overlapping occurs in high performance, time-critical, multitasking environments, and to develop, tune, and validate these architectures and models. A constraint-satisfaction approach to predicting skilled human-computer interaction posits that, with practice and motivation, people will develop elaborate strategies limited primarily based on a human’s ability to allocate perceptual and motor resources [13]. Rich data sets are needed to evaluate the theory across multiple tasks in parallel, and with fine-grained eye movement data. Work done to unify previous theories of multitasking for the benefit of human- computer interaction advances the notion of independent human processing modules used in an overlapped interleaved manner across multiple tasks, but more detailed data sets are needed to evaluate these theories as well [12]. Previous research [7, 9] developed some very specific cognitive models of human performance for a dual task experiment developed in the early 1990s at the Naval Research Laboratory (NRL), but these models were based entirely on reaction time data, with no eye tracking data available to inform the simulation of interleaved and overlapping processes. This paper revisits and extensively enhances the NRL dual task, both in experimental design and high fidelity data collection, to provide unique insights into how people develop complex interleaved cognitive strategies to coordinate and overlap perceptual processing and motor responses both within and across multiple tasks. Previous versions of the task contributed to insights for designing useful and usable interfaces for time-critical multitasking [2], guided the development of computational cognitive models [7, 9] of the task as well as general purpose cognitive architectures for predicting human performance across a wide range of tasks [8]. 1 Submitted to ACM CHI 2010. Please do not cite or distribute.