DHB: A Multiscale Framework for Analyzing Activity Dynamics James Hollan, Edwin Hutchins, and Javier Movellan University of California, San Diego What conditions can facilitate rapid advances and breakthroughs in behavioral science to rival those seen in the biological and physical sciences in the past century? The emergence of cognitive science and the converging view across multiple disciplines that human behavior is a complex dynamic interac- tion among biological, cognitive, linguistic, social and cultural processes are important first steps. While empirical and theoretical work is rapidly advancing at the biological end of this continuum, understand- ing such a complex system also necessitates data that capture the richness of real-world human activity and analytic frameworks that can exploit that richness. In the history of science, changes in technologies for capturing data, as well as those for creating and manipulating representations, have often led to sig- nificant advances. The human genome project, for example, would have been impossibly complex with- out automatic DNA sequencing [31, 43]. Recent ad- vances in digital technology present unprecedented opportunities for the capture, storage, analysis, and sharing of human activity data. Researchers from many disciplines are taking ad- vantage of increasingly inexpensive digital video and storage facilities to assemble extensive data collec- tions of human activity captured in real-world set- tings. The ability to record and share such data has created a critical moment in the practice and scope of behavioral research. The main obstacles to fully capitalizing on this opportunity are the huge time investment required for analysis using current meth- ods and understanding how to coordinate analyses focused at different scales so as to profit fully from the theoretical perspectives of multiple disciplines. We propose to integrate video and multiscale visu- alization facilities with computer vision techniques to create a flexible open framework to radically advance analysis of time-based records of human activity. We will combine automatic annotation with multiscale visual representations to allow events from multiple data streams to be juxtaposed on the same timeline so that co-occurrence, precedence, and other previ- ously invisible patterns can be observed as analysts explore data relationships at multiple temporal and spatial scales. Dynamic lenses and annotation tools will provide interactive visualizations and flexible or- ganizations of data. Our goals are to (1) accelerate analysis by employ- ing vision-based pattern recognition capabilities to pre-segment and tag data records, (2) increase anal- ysis power by visualizing multimodal activity and macro-micro relationships, and coordinating analysis and annotation across multiple scales, and (3) facil- itate shared use of our developing framework with collaborators, the wider NSF SIDGrid community, and internationally via our participation in the Rufae augmented environments network. The work we propose builds on our long term com- mitment to understanding cognition “in the wild” [30], developing multiscale visualizations [5], and re- cent experience automatically annotating video of freeway driving. We propose to extend the theory and methods developed in our earlier work and in- tegrate them with new web-based analysis tools to enable more effective analysis of human activity. As initial test domains we will focus on understanding activity in high-fidelity flight simulators and the ac- tivity histories of workstation usage and the process of writing. We will also evaluate a novel technique to assist in reinstating the context of earlier activities. Our long range objective is to better understand the dynamics of human activity as a scientific founda- tion for design. The multidisciplinary research team we have assembled, spanning cognitive and computer science and involving existing research collaborations with investigators in learning sciences and engineer- ing, is uniquely well qualified to conduct the proposed work, and has carefully defined a staged research ap- proach and milestone-based management plan. The intellectual merit of our project will derive from developing a multiscale analysis framework for representing and analyzing the dynamics of human behavior, integrating it with existing software tools for data collection, visualization, and analysis, and evaluating the augmented framework in selected real- world domains. The broader impact of the proposed activity is to provide critical analytic capabilities to support re- search networks in areas beyond our topical research areas. In general, any research using video or other time-based records in order to document or better understand human activity is a potential beneficiary of the proposed work. Our work will be disseminated through publi- cations and a community-oriented website in ac- cordance with University policy. Regular video- conferencing interactions will serve to communicate with current collaborators, extend interactions to a wider community, and encourage continued growth of a research community with shared interests in un- derstanding the dynamics of human activity. 1