Enabling Sensemaking for Intelligence Analysis in a Multi-user, Multimodal Cognitive and Immersive Environment Shannon Briggs 1 , Matt Peveler 2 , Jaimie Drozdal 2 Chelsea Sun 2 , Lilit Balagyozyan, Communications 1 , Computer Science 2 , CISL 3 Rensselaer Polytechnic Institute , Troy NY USA 12180 Email: briggs3@rpi.edu , email: pevelm@rpi.edu , email: drozdj3@rpi.edu , email: sunc4@rpi.edu , email: balagl@rpi.edu Hui Su 2 IBM Thomas J. Watson Research Laboratory Yorktown Heights NY USA 10598 email: huisuibmres@us.ibm.com Abstract — This research proposes a user interface design for a collaborative, multimodal, multi-user cognitive and immersive environment for intelligence analysis, informed by sensemaking theory and scenario- based design. We have created a prototype software in a cognitive and immersive environment, which is designed to facilitate the structured brainstorming process. Part of our research goal is to determine whether this environment will better enable sensemaking compared to traditional pen and paper tools. Keywords- HCI; immersive environment; intelligence analysis; sensemaking; scenario-based design; multimodal interaction I. INTRODUCTION The intelligence community needs software which can help them to make sense of the information they are receiving through their work as analysts. The research in this paper addresses an important aspect of how technology and the ever-increasing availability of massive amounts of data [1] impact the national security domain, specifically how intelligence analysts are challenged when sorting through information, gathering evidence, and proposing hypotheses. According to Hutchins et. al, "currently available technology is not always effective in helping the analyst assimilate the huge amount of information that needs to be synthesized." [1] Previous software for intelligence analysis has been discarded because of two major factors: the barrier of use of the software is too high, or the software does not meet the needs of the analyst. Our research takes a different approach to conceptualizing and designing software for analysts. We are developing software that will support analysts’ sensemaking by integrating educational and training materials that are already used by analysts in their own domain, such as those described by Beebe and Pherson [2] or Hall and Citrenbaum [3]. We believe that this will create a more accessible interface and will provide affordances that are already familiar to the user base. Providing analysts with a tool that provides a similar interface has proven to have some success with analysts as users, as has been discussed by Smallman [4]. Our team is focusing on the structured analytic technique of brainstorming and how we can leverage the cognitive process to create a digital tool. Through the lens of scenario-based design, we integrate the intended users’ cognitive processes into the development of our software; to do this, we will adopt cognitive tools, such as the brainstorming tool, and techniques already utilized by intelligence analysts [2], to create a cognitive and immersive environment that supports multiple users and multimodal interactions in a way that is useful for intelligence analysis. By leveraging the capabilities of the cognitive and immersive room, we are aiming to reduce cognitive load inherent in intelligence analysis. The specific research questions formulated for this stage of our research are: 1.) How can we leverage brainstorming tools to expedite the foraging process as described by Pirolli and Card [5], and 2.) How can we utilize multimodal, multi- participant input to address cognitive load ? Our application of structured analytic techniques (which can be seen in Figure 2) is a new approach to the development of digital tools and software for intelligence analysis. This paper will review historical trends among capabilities in previously released software and will suggest an application of sensemaking theory, participatory design, and scenario-based design to strengthen development of our software’s capabilities. The framework for our research is based on the intelligence analysis sensemaking process (see Figure 1) as explained by Pirroli and Card [5]. Thorough integration of the respective iterative loops of foraging for information and sensemaking will also inform how a user interface and the system tools should support the intelligence process. We propose that our software should be informed by direct feedback from analysts, scenario-based designs derived from existing structured analytic techniques, and associated resources that are currently used by intelligence analysts for training purposes, such as the tradecraft primer created by the CIA [6]. This is a novel approach to conceiving and designing software for intelligence analysis, as most existing software does not draw from the structured 7 Copyright (c) IARIA, 2019. ISBN: 978-1-61208-686-6 ACHI 2019 : The Twelfth International Conference on Advances in Computer-Human Interactions