Information Processing&Management Vol. 28, No. 3, pp. 291-315, 1992 03064573192 so0 + .oo Printed in Great Britain. Copyright 0 1992 Pergamon Press Ltd. ON PROBABILISTIC NOTIONS OF PRECISION AS A FUNCTION OF RECALL PETER BOLLMANN, ’ VIJAY V. RAGHAVAN,~ GWANG S. JUNG~ and LIH C. SHUT ‘Technische Universitat Berlin, Fachbereich Informatik, FR 5-11, FranklinstraRe 28129, D-1000 Berlin 10, Germany *The Center for Advanced Computer Studies, University of Southwestern Louisiana, P.O. Box 44330, Lafayette, LA 70504, U.S.A. 3Computer Science Department, Jackson State University, Jackson, MS 39217, U.S.A. 4Department of Computer Sciences, Purdue University, West Lafayette, IN 47907, U.S.A. Abstract -Two problems that arise when recall and precision are used to evaluate in- formation retrieval systems are due to the weak ordering of the documents generated by the system and evaluation with multiple queries. Although several alternative stopping criteria are available, our emphasis in this paper is on defining precision when recall is used as the stopping criterion. A number of different probabilistic notions of precision for handling the problem of weak ordering have been proposed in the past, including PRECALL, probability of relevance given retrieval (PRR), and expected precision (EP). Recently Raghavan et al. provided a comparative analysis of PRECALL, PRR, and EP They showed that previous usages of PRECALL for dealing with the problem of weak ordering and interpolation, which involved the application of ceiling operation, are in- consistent, and the results obtained are not easy to interpret. Consequently, they intro- duced an interpolation scheme, termed intuitive interpolation, that leads to consistent and meaningful handling of averaging results given by PRR over multiple queries. A sim- ple way of calculating PRR was also given. However, a comparable analysis of precision defined as EP has not been provided. Furthermore, given that several alternative ways of defining precision in a probabilistic sense are available, no theoretical basis for de- ciding which alternative to use in a specific situation exists. This paper initially investi- gates an efficient way of calculating EP and an interpolation scheme for averaging EP that are consistent with the intuitive interpolation scheme proposed for PRR. In addi- tion, PRECALL with intuitive interpolation is termed R-B Precision, and is shown to have interpretation as the value of PRR and EP, in the limit. From this result, PRR and EP are shown to be attractive in their ability to present experimental results in a descrip- tive sense. In contrast, in situations where experimental tests are intended for predictive use, R-B Precision is shown to be a better choice. 1. INTRODUCTION In information retrieval systems, recall and precision are two of the most popular perfor- mance evaluation criteria (Cleverdon, 1970; Salton & McGill, 1983). Recall is defined as the ratio of the number of relevant documents retrieved to the total number of relevant doc- uments. Precision, on the other hand, is the number of relevant documents retrieved di- vided by the number of retrieved documents. In particular, a recall-precision graph is often used as a combined measure of retrieval system performance. Such a graph, given an ar- bitrary recall point, is expected to tell us the corresponding precision value. In order for a retrieval system to locate the relevant items from a given collection with respect to a search request, a measure called the Retrieval Status Value (RSV) is often com- puted between each item in the collection and the search request. The RSVcan be viewed as an indicator of the degree of similarity between a document and a request. Many dif- ferent similarity or distance functions, in order to obtain the RSVs, have been proposed in the past. Among the commonly known examples are the simple matching function and the cosine similarity. None of them has been proved or observed to be optimal under all circumstances. Consequently, the choice of the function for computing the RSVs should 291