Extracting Knowledge from Usability Evaluation Databases Elena García 1 , Miguel A. Sicilia 2 , J. Ramón Hilera 1 & José. A. G. de Mesa 1 1 Alcalá University, Alcalá de Henares, Madrid, Spain {elena.garciab, jose.hilera, jantonio.gutierrez}@uah.es 2 DEI Laboratory, Carlos III University, Leganés, Madrid, Spain msicilia@inf.uc3m.es Abstract: We describe an approach to extract knowledge from large repositories of usability evaluation facts by using artificial intelligence techniques. Specifically, we have applied fuzzy linguistic aggregation, clustering, association mining and classifiers to different aspects of usability questionnaire analysis and design. The extracted knowledge has proven useful in computer-aided tool driven evaluation settings. Keywords: questionnaire-based usability evaluation, fuzzy linguistic aggregation, machine learning. 1 Introduction Cost-justifying usability is a well-known problem that can be addressed from different perspectives (Mayhew, 1994) depending on the various organizational contexts. An important hidden cost of many usability evaluation plans is the one derived from storing and processing potentially large amounts of data in computer systems. The approach we present attenuates these costs by building a computer-aided tool that attempts to give support for some usability inspection and evaluation techniques in an extensible manner, and provides some sort of “intelligent” value-added options. Our tool allows the usability expert to take advantage of automatically extracted findings, increasing this way the actual and perceived benefits of usability engineering. In this paper, we describe how artificial intelligent (AI) techniques can be used to exploit databases made up of usability evaluation facts obtained from questionnaires, in the following ways: (a) by using fuzzy operators, association rules and clustering algorithms to obtain helpful knowledge for the evaluation of external attributes of usability (Brajnik, 2000), and (b) by applying machine learning algorithms to predict Web page usability from internal attributes in an automated way. The rest of this paper is organized as follows: Section 2 describes our database schema, which stores usability evaluation facts. Section 3 describes the techniques used to exploit this repository, and Section 4 presents some conclusions and future work. 2 Automating Questionnaire- Based Usability Evaluations Usability evaluation questionnaires can be modeled in the following way. We consider a set O={o 1 ,o 2 ,…,o n } made up of evaluation subjects (e.g. software systems or web portals). These subjects are considered as objects that can be further decomposed in aspects (e.g. the user registration process in a web portal), belonging to set A. A number of evaluators E={e 1 , e2,…,e m } are asked for opinion about some of the aspects of the system (or about the system as a whole). Evaluators can be classified by expertise level or some other significant criteria, belonging to an evaluator profile denoted as e(e i ). Their opinions are collected through questionnaires, which are designed to evaluate some predefined criteria C={c 1 ,c 2 ,…,c k }, taken from a set of common ones that can be found in usability literature. A questionnaire is a hierarchically structured tree of questions in set Q. Each question can be internally associated with one or more criterion. Summing up, an evaluation fact is the response to a question, which internally can be characterized as a tuple (e i , o m , a k , q j , v ), that belongs to E x O x A x Q x V, where V represents the domain of evaluations. Our system implements