6 Journal of Information Organization Volume 1 Number 1 March 2011 Improving Web Search Engines Evaluation By the Context Consideration Abdelkrim Bouramoul, Med Kholladi Computer Science Department, Misc Laboratory University of Mentouri Constantine B.P. 325, Constantine 25017, Algeria a.bouramoul@yahoo.fr ABSTRACT: Limitations and shortcomings of traditional approaches for evaluating information retrieval tools, particularly regarding to the consideration of the user, the manner in which these approaches measure the adequacy between the user’s query and the documents returned, and the characteristics consideration of the search tool itself, have motivated our reflections for the exploitation of contextual elements around the user, the query and the search tool during the evaluation process. This paper presents a new approach based on three complementary levels of context for evaluating information retrieval systems. The experiments gives at the end of this article has shown the applicability of the proposed approach to real research tools. The tests were performed with the most popular searching engine (i.e. Google, Bing and Yahoo) selected in particular for their high selectivity. The results have revealed that the evaluation of search engines at three different levels of context is a promising way to diagnose the performance, characteristics and behavior of these engines and the relevance of the results that they return. Keywords: Information retrieval, contextual evaluation, relevance judgments, web search engine Received: 17 October 2010, Revised 2 November 2010, Accepted 5 November 2010 1. Introduction Information Retrieval Systems (IRS) are tools for finding information in closed document collections or on the Web. The challenge is to finding, among the large volume of documents available, those that best fit our needs. Consequently several questions arise about these information retrieval tools, particularly in terms of their performance and the relevance of the results that they offer. It is therefore in the field of the evaluation of IRS and more specifically that of the contextual evaluation that our work falls. After a deep investigation around research and synthesis activities we realized that despite the abundant literature produced in this area dealing with both experimental results and methods that provide evaluation criteria and metrics of relevance, few of these methods are interested in the consideration of the context during the evaluation process. Our contribution is guided by two main reasons: firstly the lack that we observed around the context-based methodologies for measuring the quality of information retrieval tools, this finding is reinforced by the work of [1] and [2]. And secondly by the requirement to which we are confronted recently after conducting work in the field of the consideration of context in information retrieval systems [3]; Where we have failed to find a contextual evaluation protocol to validate of our proposal. This work will therefore be a logical continuation of what has been done before, and a promising way to cover the process of the contextual evaluation of IRS. This paper is organized as follows: we start first by giving a definition of the concept of context and its use in the field of information retrieval, we then present an overview of classical approaches for evaluating information retrieval systems and we focus on the limits and shortcomings faced by these approaches. In the next section we discuss our contribution by giving an overview of the contextual evaluation approach that we propose and describing its principle and its techniques. Finally, we present an experimentation of our approach to evaluate the performance of search engines Google, Yahoo and Bing, followed by a discussion of results and conclusion.