Efficient Query Delegation by Detecting Redundant Retrieval Strategies Christian Scheel * DAI-Labor TU-Berlin scheel@dai-lab.de Nicolas Neubauer † NI Processing Group TU-Berlin neubauer@cs.tu-berlin.de Andreas Lommatzsch DAI-Labor * TU-Berlin andreas@dai-lab.de Klaus Obermayer NI Processing Group † TU-Berlin oby@cs.tu-berlin.de Sahin Albayrak DAI-Labor * TU-Berlin sahin@dai-lab.de ABSTRACT The task of combining the output of several retrieval strate- gies into a single relevance prediction per document is known as data fusion. The LETOR dataset provides three corpora with predictions of 25 or 44 strategies (depending on the corpus) per document/query pair. Given such a large num- ber of basic strategies, a point which is equally crucial as optimality of the combination, in our view, is its sparseness: Which strategies should be used in a real application when each strategy consumes resources? We hence focus on the question of “query delegation”, a special case of weighting strategies: Which strategies should be weighted greater than zero, i.e., asked in the first place? We propose several sim- ilarity measures between strategies like various correlation measures or precision@n. Assuming that similar strategies may not contribute much to each other’s results, we perform a clustering based on these similarities and only consider the best representative of each cluster. We show that this fusion strategy performs comparably to other fusion approaches like RankSVM or RankBoost, but only needs to consult a fraction of the available retrieval strategies. Categories and Subject Descriptors H.3.3 [Information Storage and Retrieval]: Informa- tion Search and Retrieval—Selection process, Information filtering ; I.5.3 [Clustering]: Similarity measures General Terms Algorithms, Performance, Experimentation Keywords Ranking, Correlation, Clustering, Fusion * Technische Universit¨ at Berlin, D-10587, Germany † Neural Information Processing Group,* Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Copyright 200X ACM X-XXXXX-XX-X/XX/XX ...$5.00. 1. INTRODUCTION Combining the results of various retrieval strategies into a single output is a frequent task in a number of scenar- ios: Quite naturally in meta-search engines and agent-based information retrieval communities, but also in stand-alone systems, as the combination of several retrieval strategies reliably improves retrieval performance. The general task of merging the results is known as fusion, or more concretely data fusion (as opposed to collection fusion) if the results refer to the same set of documents. Different fusion strate- gies have been discussed in prior work, e.g. [5, 11, 14], as well as the question what makes different retrieval strate- gies suitable for being combined: Some argue that data fu- sion candidates should be diverse and independent [15, 18], others claim they should be dependent [13]. A related dis- cussion is about whether a large overlap of relevant docu- ments is desirable [4] or whether the contribution of new relevant documents is more important [3]. We take the stance that given a choice out of a pool of retrieval strate- gies, we should minimize their similarity while maximizing their individual quality. We test this assumption on the LETOR dataset, which in our view – given predictions of 25 (OHSUMED corpus) / 44 (TREC corpora) strategies per document/query pair – imposes the additional constraint of finding a small, but suitable subset of retrieval strategies to fuse: each computation of these predictions, in a real application, would consume resources, so our focus is not so much on finding an optimal ratio between the individ- ual results, but rather on determining which strategies to ask in the first place (query delegation). We examine in how far we can fulfill the three constraints (maximum diver- sity, maximum quality, minimum number of strategies) men- tioned above by choosing the best strategy from a group of strategies created by similarity-based clustering. We exam- ine several similarity measures between retrieval strategies with regard to their suitability for this task. We do not use the information about the method behind each prediction as such meta-data might not be available in practice. One possibility instead is to evaluate and compare the strategies’ performances based on available relevance judgments. How- ever, similarity measures based on the correlation between the strategies’ rankings turn out to be particularly useful in this context, and have the additional advantage of not requiring user feedback. We show that using the sketched Presented at the SIGIR 2007 workshop on Learning to Rank for Information Retrieval (LR4IR-2007), Amsterdam, Netherlands, 2007. Copyright 2007 by the author(s)/owner(s).