O. A. Omojokun et al.: Towards Automatic Personalization of Device Controls Manuscript received January 15, 2009 0098 3063/09/$20.00 © 2009 IEEE 269 Towards Automatic Personalization of Device Controls Olufisayo A. Omojokun, Charles L. Isbell, Jr., and Prasun Dewan Abstract — People are increasingly using customizable remotes to interact with devices in new and interesting ways that are influenced by the idiosyncrasies of their behaviors and environments. With the growing use of advanced processors in small consumer electronics, it is becoming more practical to have such remotes execute machine-learning algorithms that can automatically specify the idiosyncrasies. This paper addresses two especially useful and common types of features of personalizable remotes: “task based button grouping” and macros. “Task-based button grouping” addresses clutter and frequent screen switching by only presenting the commands (or buttons) a user needs to accomplish a given active task. Macros allow users to efficiently invoke a sequence of commands across multiple devices that are used in the task. The contributions of this work include: (a) an identification of several usage patterns that show limitations of previous work in task and macro based commands, (b) a set of new algorithms that apply these patterns to address these limitations, and (c) an evaluation of each algorithm using real-world interaction data. We show that our algorithms, which uniquely apply fuzzy techniques and time-based heuristics, can offer a significant improvement from the state-of-the-art in automation and accuracy 1 . Index Terms — remote controls, personalization, tasks, macros. I. INTRODUCTION Our environments are becoming richly embedded with sensors, appliances, and other kinds of consumer electronics. Living rooms, for example, are being outfitted with new and unique kinds of devices for accessing various forms of Internet media without the need of a traditional computer. Furthermore, it is possible to purchase inexpensive off-the- shelf sensor systems that allow users to remotely monitor a wide range of interesting events at home, such as: (1) the opening and closing of doors—even those of cupboards, garages, and appliances (e.g. fridges and dryers), (2) whether a particular appliance is drawing electricity from a socket (i.e., if it is on or off), and (3) water leakages from washer devices. As the above trend continues, users will increasingly interact with devices in new and interesting ways that extend 1 This research was funded in part by NSF grants IIS 0312328, IIS 0712794, and IIS 0810861. Olufisayo A. Omojokun is with Georgia Institute of Technology, Atlanta, GA 30332 USA (e-mail: omojokun@cc.gatech.edu). Charles L. Isbell is with Georgia Institute of Technology, Atlanta, GA 30332 USA (e-mail: isbell@cc.gatech.edu). Prasun Dewan is with University of North Carolina, Chapel Hill, NC 27599 USA (e-mail: dewan@cs.unc.edu). from the idiosyncrasies of their behaviors and environments. Consequently, it is becoming increasingly important to focus on the controls for supporting interaction and ensure that the features they offer facilitate these idiosyncratic experiences. Unfortunately, specifying these idiosyncrasies in today’s customizable remotes is generally done manually, which can be tedious and prone to human error [1]. With the growing use of advanced processors in small consumer electronics, it is becoming more practical to have such remotes execute machine learning (ML) algorithms that can automatically specify the idiosyncrasies. To realize this vision, we have previously worked in ML-based support of two especially useful and common features offered by today’s personalizable remotes: “task-based button grouping” and macros [1],[2]. “Task-based button grouping” addresses clutter and frequent screen switching by only presenting the buttons a user needs to accomplish a given active task (e.g. listening to music). A macro, on the other hand, allows the user to efficiently invoke a sequence of commands across multiple devices that are used in the task (Fig. 1). For the two features, we have been pursuing a solution to the particularly important problem of how to automatically select and compose the commands that they require. Fig. 1. A task-based user-interface for listening to CDs using a DVD player and A/V receiver. A single screen includes the multi-device buttons needed while playing CDs, and a ‘Power All’ macro button for turning the two devices on and off with a single push. This paper represents a substantial update and extension of this previous work. Specifically, it makes the following contributions: (a) an identification of several usage patterns that show limitations of previous work in task and macro based commands, (b) a set of new algorithms that apply these patterns to address these limitations, and (c) an evaluation of each algorithm using real-world interaction data. We show that our algorithms, which uniquely apply fuzzy techniques and time-based heuristics, can offer a significant improvement from the state-of-the-art in automation and accuracy. The rest of this paper is organized as follows. Section II summarizes previous work in the area of personalizable device controls. Section III summarizes our research and results. Finally, Section IV discusses our conclusions and future work. PowerAll (DVD,RCVR) PREV NEXT PLAY PAUSE STOP VOL + VOL – MUTE DVD/LD