The Quality Indicators for an Information Retrieval System: User's Perspective Roslina Othman Faculty ofICT, International Islamic University Malaysia Abstract An IR system must be designed to satisfy a user's information need. To achieve quality results, the system must help users to construct quality searches. Thus the aim of this project was to identify a set of quality criteria and indicators to evaluate an IR system. Survey, observation, and self-reporting logs were used to compile a list of quality criteria and their indicators, and expected features from 250 users. The findings revealed quality criteria that include content, retrieval features (e.g. term density, term boosting, and fuzzy searching), user interface, thesaurus-enhanced search (e.g. visualization of search results), help and feedback mechanism, and administrative considerations. 1. Introduction An information retrieval (IR) system offers features and interfaces to help users find relevant information. However, users faced difficulties in finding the right information in an IR system [1]. The difficulties were related to application of retrieval features and search formulation. IR systems share many common features like Boolean operators, word/phrase search, proximity, stemming, truncation, and wildcard; however, the interpretation and implementation vary from one system to another [1]. Users struggled to construct search terms and include term variations [2]. Yet, these users have undergone training in searching. 1.1. Measures for quality searches In estimating the quality of searches conducted by users, measures like recall, precision, similarity, user effort, and usability have been applied to see if an IR system were able to satisfy a user's information need. 1.1.1. Recall, precision, and similarity Recall indicates a system's ability to retrieve all relevant items in the collection. Precision measures a system's ability to reject irrelevant items. Similarity [3] shows a system's ability to retrieve items having a close match to known items. In this case, the known items are representation of the user's information need. Recall, precision, and similarity yield levels of retrieval performance based on search results, which then prompt the evaluator to examine search strategies, retrieval features, and user-interfaces to identify known causes for such level of performance. Recall, precision, and similarity reveal success or failures related to search strategies and indexing languages. 1.1.2. Data Retrieval and Information Retrieval Search results are ranked according to system relevance, while these measures are computed based on user's relevance. The levels of performance produced by recall, precision and similarity indicate the extent to which system relevance match and mismatch user's relevance. Often the system's relevance did not correspond with user's relevance [4] The retrieval features and user interfaces are designed to help users construct a query that match the index terms assigned documents in the collection. A document is retrieved by the system when the index term matched with the query. However, a user may not agree that the document is relevant to his/her information need. The system in matching the query and the index terms is actually providing data retrieval. The system matched index terms and query, and not necessarily documents with information needs. Matching documents with information needs is information retrieval. Users expected that an IR system must provide the information or data with semantics that they have in mind. To achieve an effective match, users must be able to translate information needs into search terms, apply the appropriate retrieval features and user interfaces, and construct effective search strategies. This leads to the inclusion of measures that are concerned with user's searching ability. 1.1.3. User effort and usability Measures that focus on user's searching ability include user effort and usability. User effort [5] measures the time a user spent on searching in an IR system to fulfill his/her information need. User effort includes time spent learning to use the system since the user learns when he/she searches in the system. Usability [6] is concerned with the use of interfaces and retrieval features to achieve desired search results. Usability is a measure that focuses on both the search formulation and search results. Users need to perform effective searches in a quality IR system. Quality is defined here as finding the right information or information that satisfies the user's information need. 0-7803-9521-2/06/$20.00 §2006 IEEE. 1 738