Searchers' Criteria For Assessing Web Pages Anastasios Tombros Ian Ruthven Joemon M. Jose Department of Computing Science, University of Glasgow, Glasgow, U.K. Department of Computer and Information Sciences, University of Strathclyde, Glasgow, U.K. Department of Computing Science, University of Glasgow, Glasgow, U.K. tombrosa@dcs.gla.ac.uk Ian.Ruthven@cis.strath.ac.uk jj@dcs.gla.ac.uk ABSTRACT We investigate the criteria used by online searchers when assessing the relevance of web pages to information-seeking tasks. Twenty four searchers were given three tasks each, and indicated the features of web pages which they employed when deciding about the usefulness of the pages. These tasks were presented within the context of a simulated work-task situation. The results of this study provide a set of criteria used by searchers to decide about the utility of web pages. Such criteria have implications for the design of systems that use or recommend web pages, as well as to authors of web pages. Categories and Subject Descriptors H.3.3 [Information Search and Retrieval]: – search process H.5.4 [Hypertext/Hypermedia]: – user issues General Terms Design, Human Factors Keywords World Wide Web, user studies, evaluation, empirical studies 1. INTRODUCTION Studies on peoples’ perceptions of the relevance of information demonstrate that a range of factors affect human judgements of relevance. Barry [1], for example, studied factors that influenced searchers’ criteria for relevance when reading textual documents. However, such studies often only consider formal textual documents, such as journal and conference articles, rather than the wide range of formally and informally-produced multimedia documents found on the web. The motivation behind this study has been to gain a better understanding of what features make a web page useful for information seeking. We observe the decisions made by web users whilst searching on given information-seeking tasks. We gather, through think-aloud, system logging, and informal discussion, information on the relative utility of structural content (e.g. page layout, link structure), information content (e.g. use of text and multimedia) and aspects of quality (e.g. source of page or recency of information). Our study has important implications for the design of systems that recommend or use web pages. A good example is web page summarisation systems, e.g. [3]. Although such summaries have been shown to be useful for interactive searching, they generally only consider the textual content of the page. By utilising textual and non-textual aspects of pages, systems can provide a more complete picture of web pages. 2. METHODOLOGY Twenty four searchers participated in the study. Each was given three search tasks to complete. The searchers were asked to discuss, in the form of think-alouds, their perceptions of what features helped them decide on the usefulness (or non-usefulness) of the web pages they chose to view. There were no restrictions imposed on the search engines or web sites searchers could visit. User desktop actions (e.g. web pages accessed, mouse movements, etc.) and speech at the duration of each task were recorded in the form of video sessions using the Camtasia software package (Techsmith Corporation, www.techsmith.com). The sessions were subsequently analysed by one of the experimenters. 2.1 Search Tasks Each of the subjects was asked to complete three search tasks. The tasks were placed in the context of a simulated work task situation [2]. In this way, users are encouraged to treat the information seeking task as a personal task, searching as though the task was their own. The three tasks also simulated different types of searches. In the first task, a background search, searchers were asked to find as much general background information on the demographics of the Internet as possible. For the second task subjects were asked to decide on the best hi-fi speakers available in their own price range (decision task). The third task, a many items search, asked the subjects to compile a list of interesting things to do over a weekend stay in the city of Kyoto. The order of presentation of the tasks was rotated across subjects. 2.2 Analysing User Sessions We analysed each recorded user session, and extracted web page features that were mentioned by searchers as indicating a page’s usefulness (positive mention) or non-usefulness (negative mention) to a task (i.e. features were discovered through searchers’ mentions rather than predefined by the experimenters). We thus gathered data for features in two lists, depending on the type of pages (useful or not) they indicated. After the processing of all user sessions, features were grouped under broader feature categories. In total, 24 features were grouped under 5 feature categories (Table 1). Copyright is held by the author/owner(s). SIGIR’03, July 28–August 1, 2003, Toronto, Canada. ACM 1-58813-646-3/03/0007.