Evaluation of the Real and Perceived Value of Automatic and Interactive Query Expansion Yael Nemeth Ben-Gurion University Beer-Sheva 80145 Israel nemethy@bgumail.bgu.ac.il Bracha Shapira Ben-Gurion University Bee-Sheva 80145 Israel bshapira@bgumail.bgu.ac.il Meirav Taeib-Maimon Ben-Gurion University Bee-Sheva 80145 Israel meiravta@bgumail.bgu.ac.il ABSTRACT The paper describes a user study examining methods for improving users queries, specifically interactive and automatic query expansion and advanced search options. The user study includes subjective and objective evaluation of the effect of the above methods and a comparison between the real and perceived effect. Categories and Subject Descriptors H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval – query formulation, Search process General Terms Measurement, Performance, Experimentation, Human Factors. Keywords Query Expansion, Evaluation, User Study, Real Value, Perceived Value. 1. INTRODUCTION Many efforts for improving search engine performance have focused on developing tools to meet users query formulation needs rather than merely responding to their queries. It has been found that the main reason for low precision in search engines retrieval results stems from imprecise queries. Ambiguous, non- specific, or uncommon keywords used in a query result in non- relevant documents in the retrieved set. Some solutions to this problem include query expansion aiming at enhancing queries [1], and advance search options offered by several search engines. The present study evaluated the effect of these suggested solutions from the users' point of view. While earlier studies used simulations to evaluate the effectiveness of query expansion ([7], [3] , [2],[5]), our study attempts to examine the real and perceived value of automatic and interactive query expansion through user studies. We constructed a search engine that includes the automatic local- context query expansion algorithm [8], and an interactive (semi- automatic) version of this algorithm. The search engine can be activated in different modes applying three different search types, namely, Automatic Query Expansion (auto), Interactive Query Expansion (semi), and a regular search without query expansion (none). We also applied and evaluated advanced search options. The options were chosen according to results of a preliminary survey distributed to users and included file format and date of the page update. During the user studies, users were asked to perform tasks (answer questions) using our search engine in different search modes. Users achievements in performing the tasks when using each mode was evaluated to employ real-value analysis. Additionally, we asked users to evaluate the quality and express their satisfaction with the results to employ subjective perceived value analysis. 2. EXPERIMENTAL METHOD The experiment included three retrieval modes: 1) Retrieval with no Query Expansion (QE), 2) retrieval using Automatic QE, and 3) retrieval using Interactive QE. Each mode was tested with and without advanced search options (altogether, six modes). The six modes were: 1. Basic search engine (none). 2. Search Engine with automatic query expansion (auto). 3. Search Engine with interactive query expansion (semi). 4. Basic Search Engine with advance search options (none+). 5. Search Engine with automatic query expansion and with advance search options: (auto+). 6. Search Engine with interactive query expansion and with advance search options (semi+). Two aspects of the effectiveness of these modes were tested. One aspect was users objective performance, and the other was users subjective satisfaction when using the different retrieval modes. IQE called for additional interaction with users where they had to choose terms for expansion from the list of terms suggested by the system. We therefore examined another subjective aspect, relevant only to IQE: users satisfaction and attitude towards this interaction. Users objective performance was tested through their achievement in tasks and their effort using the Automatic and Interactive QE. Their achievement was evaluated by three expert evaluators who graded the users answers for their correctness. Users effort was measured by the no. of interactions they performed before they completed a task. We consider a query submitted to the system as an interaction. The users subjective opinion was tested through questions they had to answer after completing each task. The participants consisted of 72 undergraduate students from the department of Industrial Engineering and Management at Ben-Gurion University (41 males, 31 females) in their forth year of studies. Their ages ranged from 19 to 28. We used Lucene [4] as the basic search engine and implemented additional modules for QE. The engine could be activated in a basic mode (none) with no QE, or in the (auto) mode implementing the Local Context Analysis query expansion Copyright is held by the author/owner(s). SIGIR’04, July 25–29, 2004, Sheffield, South Yorkshire, UK. ACM 1-58113-881-4/04/0007. 526