A Context-Aware Personal Desktop Assistant (Demo Paper) Hung H. Bui, Federico Cesari, Daniel Elenius, David N. Morley SRI International Menlo Park, CA, U.S.A. first.lastname@sri.com Sriraam Natarajan Oregon State University Corvallis, OR, U.S.A. natarasr@eecs. oregonstate.edu Shahin Saadati, Eric Yeh, Neil Yorke-Smith SRI International Menlo Park, CA, U.S.A. first.lastname@sri.com ABSTRACT We demonstrate an intelligent personal assistant agent that has been developed to aid a busy knowledge worker in managing time com- mitments and performing tasks. The PExA agent draws on a di- verse set of AI technologies that are linked within the SPARK BDI agent framework. We focus on our agent’s ability to provide as- sistance within the context of current user activities, based on its recognition of user workﬂows and their progress, and on its context- sensitive proactive suggestions. We have instrumented a common suite of desktop applications so that, endowed with a sophisticated workﬂow tracker, PExA has the ability to pervasively monitor the user’s desktop activities. PExA follows and responds to the user’s progress on shared tasks, and is highly user-centric in its support for user needs and its adaptivity to user working style and preferences. Categories and Subject Descriptors I.2.11 [ARTIFICIAL INTELLIGENCE]: Distributed Artiﬁcial Intelligence—Intelligent agents; I.2.1 [ARTIFICIAL INTELLI- GENCE]: Applications and Expert Systems—Ofﬁce automation General Terms Algorithms, Design, Human Factors Keywords activity recognition, proactive assistance, integrated cognition, CALO 1. THE PEXA AGENT The vision of an agent that acts as your personal butler, attentive to your requests, aware of your goals and preferences, and antici- pating your needs, requires the agent to act appropriately according to context [4]. Our work on the PExA (Project Execution Assistant) agent is part of the CALO project, a large-scale effort to build an adaptive, interactive cognitive assistant situated in the ofﬁce envi- ronment [2]. The overall CALO system is designed to support its user in various ways including project and task management, infor- mation organization, and meeting preparation and summarization. A critical aspect of context in this setting is the user’s current ac- tivity within the larger scope of his or her current and future tasks. Cite as: A Context-Aware Personal Desktop Assistant (Demo Paper), H. H. Bui, F. Cesari, D. Elenius, D. N. Morley, S. Natarajan, S. Saadati, E. Yeh, N. Yorke-Smith, Proc. of 7th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2008), Padgham, Parkes, Müller and Parsons (eds.), May, 12-16., 2008, Estoril, Portugal, pp. XXX-XXX. Copyright c  2008, International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved. The agent must be able to understand what the user is working on in the present and what goals the current activities are directed to- wards. This demonstration exhibits aspects of PExA’s assistance, especially those capabilities which are enabled by the agent’s abil- ity to understand the user current activities and goals. Fig. 1 shows part of the PExA architecture. The Workﬂow Tracker recognizes current user activity, based on Logical Hidden Markov Models (Sect. 2). The Task Manager, based on the SPARK agent platform [5], deliberates and performs assistive actions (Sect. 3). Fig. 2 shows an example of suggestions made proactively by PExA. 2. WORKFLOW RECOGNITION In order to assist its user, a PExA agent requires an understand- ing of the user’s goals on the desktop, and knowledge of means by which the user and agent together can achieve these goals. PExA’s task library consists of a dual declarative/procedural representation. Declaratively, the task models describe the subtasks that the user, the agent, or other agents achieve to fulﬁll a goal, together with constraints between them (such as ordering of subtasks). Procedu- rally, the task models provide recipes that PExA can instantiate into a plan to achieve a given subtask. Some tasks in the library might be executed only by the user, only by the agent, or both. We call the declarative representation of tasks workﬂows. A workﬂow models a pattern of behaviour of the user (perhaps aided by other agents) in achievement of a user goal. For example, in the workﬂow journal paper review, the user ﬁrst downloads the paper and the review form attached to the review request email. Next, the paper is printed, and the review form is ﬁlled out. Finally, the review form is sent back as a reply to the request email. Knowledge of progress through the workﬂow (e.g., current step) enables PExA to volunteer information and suggestions (e.g., re- lated documents, emails, web links) speciﬁcally chosen for the right context, to provide summarization of progress (e.g., “waiting on Al- ice to complete this step”), and to itself act (e.g., offer to perform the next step, prepare for future steps). Keeping track of progress is challenging for steps being executed by the user. It would be burdensome for PExA to require the user to explicitly indicate commencement and completion of every step. We call the problem of automatically identifying the workﬂow and the user’s current step workﬂow recognition and tracking. We in- strumented the desktop (Windows Explorer) and common applica- tions such as email clients (Thunderbird), web browsers (Firefox and Internet Explorer), and ofﬁce applications (Word, PowerPoint, Excel) so that user-performed actions are captured and logged. A Workﬂow Tracker (Fig. 1) identiﬁes whether the stream of captured interaction events matches with any of the workﬂows in the task library, and if so, tracks its current progress.