In Käkölä, T. (ed.), Proceedings of IRIS22, Jyväskylä, Finland, 7-10 August 1999 1 A Method for Intranet Search Engine Evaluations Dick Stenmark it.dixi@memo.volvo.se Volvo Information Technology Dept. 9530, HD1N SE-40508 Göteborg, Sweden IT & Organisation Viktoria Institute / GöteborgUniversity Box 620, SE-40530 Göteborg, Sweden Abstract Most comparative evaluations of Internet search engines provide soon outdated answers to a handful of pre-selected questions. Such an approach is too limited to be useful for practitioners wanting to evaluate information retrieval tools. Informed by an action research-oriented study of a large organisation conducting such an evaluation, this paper first discusses what questions to ask when analysing and evaluating intranet search tools. Applying Henderson and Cooprider’s approach and Saracevic’s theoretical foundation to the field data the author outlines a generic methodology for information system evaluation. The proposed methodology is refined through repeated interviews and discussions with people from a variety of organisational roles, thus taking a broader and more general approach. By suggesting that the selected features should be given different weighs according to their relative importance and summarised in two dimensions, the methodology helps organisations detecting the most useful tool for their specific requirements. Keywords: evaluation methodology, search engine, intranet, information systems 1. Introduction As the World Wide Web (hereafter, the web) started to explode in terms of numbers of users, servers, and pages, it became painfully obvious that search capabilities had to be added to this chaos of information in order for people to be able to find anything. Before long, a flourishing supply of publicly available search services emerged, and today these tools have become part of our everyday life as net citizens. Not surprisingly, a substantial body of research regarding search engines has therefore developed over the past years. A few years later, the number of intranets, i.e. intra-organisational webs hidden from the global public behind firewalls and proxies, had grown to become an important part of the web. Since the public search engines’ spiders are not allowed inside firewalls, organisations relying on such solutions must host their own engines in order to provide search capabilities. Though many commercial search engines and spiders are available on the market, it remains difficult for a particular organisation to know which solution best suits their needs and requirements. The computer-related press regularly publishes “stand-alone” evaluations of various tools, gadgets, and services - including search engines - but such evaluations are often highly subjective and the features are examined out of context. This fact calls for more targeted research on how such evaluations should be conducted.