The Appropriate (and Inappropriate) Use of Query Operators and Their Effect on Web Search Results zyxw Caroline M. Eastman Department zyxwvutsrq of Computer Science and Engineering, University of South Carolina, Columbia, zy SC zy 29208. Email: eastman@cse.sc.edu Bernard J. Jansen School of Information Sciences and Technology, The Pennsylvania State University, University Park, PA, 16801. Email: jjansen@acm.org In previous work, we showed that the use of Boolean and other operators in queries to Web search engines has little impact on the quality of the results retrieved. This earlier study used I00 queries containing operators selected from a search engine query log. The operators were removed from the queries, and both query versions (with and without operators) were submitted to three popular search engines. There was no significant difference in number of relevant results retrieved in the top 10 items. In this work, we revisit the queries used in this earlier study to examine the manner in which the operators were used. At that time the reason for the relatively low impact of the query operators was not addressed in detail. For each query, the operator used was classified based upon semantic appropriateness and anticipated impact. Most query operators were used in a semantically appropriate manner. This finding suggests that the reason for the relatively low impact of operators on retrieval results was not inappropriate or incorrect use of the operators. Introduction Previous work by the authors has shown that the use of Boolean and other operators in Web searching has little impact on the quality of the results retrieved, c.f., (Eastman, 2002; Eastman & Jansen, 2003; Jansen, 2003; Jansen & Eastman, 2003). A comprehensive study (Eastman & Jansen, 2003) used 100 queries from real users expressing real information needs selected from an Web search engine transaction log. Each query had Boolean or other operators. However, the research showed that the use of these operators had no significant effect on the results retrieved. However, this still leaves open the question of why the use of operators had so little impact. One possible explanation is that the searchers used the operators incorrectly. If this were the case, one might expect improvement in search results from improved education or assistance in the appropriate and effective use of query operators. However, if the searchers used the operators correctly, then additional educational or assistance efforts in query operator use are likely to be of little value. In this research, we investigate the nature of operator use in the queries used in the previous study to see if the searchers were using the query operators appropriately or not. To examine this, we placed the queries into four categories with respect to operator use: (1) appropriate, (2) neutral, (3) inappropriate, and (4) incomplete. Appropriate queries are ones in which the operator is used in a manner that, based upon the operator definition, might reasonably be expected to improve search results. Neutral queries are ones in which the operator is used in zy a semantically reasonable way but for which improved search results would not be expected. Inappropriate queries are ones in which the operator is used in a semantically unreasonable way. Incomplete queries are ones in which an operator is used in an apparently reasonable manner but used one or more implicit (and therefore) ambiguous phrases. Most of the queries fell into the neutral and appropriate categories; very few exhibited inappropriate operator usage. In the next three sections we discuss related work on the nature of operator use by end users of information retrieval and similar systems. We then summarize the methods and results of the previous study that serves as a foundation for the work presented here. The methods and results for this study are presented in the following two sections. Discussion and conclusions are in the final sections. Web Searching Studies There is a growing body of research examining the use of Web search engines. An overview is provided by Jansen & Pooch (2001). Some specific studies are described in (Hlilscher & Strube, 2000; Jansen & Pooch, 2001; Jansen, Spink, & Saracevic, 2000; Montgomery & Faloutsos, 2001; Silverstein, Henzinger, Marais, & Moricz, 1999). Jansen zy 2004 2 74 Proceedings zyxwvut of the 67th ASIS&TAnnual Meeting, vol, 41