Studying the Use of Interactive Multilingual Information Retrieval Daqing He¹, Douglas W. Oard 2 , and Lynne Plettenberg 2 ¹School of Information Sciences, University of Pittsburgh, 135 North Bellefield Avenue, Pittsburgh, PA 15260 daqing@mail.sis.pitt.edu 2 College of Information Studies, University of Maryland, College Park, MD 20742 {oard, lpletten}@umd.edu ABSTRACT We often talk as if information retrieval systems were machines, but in reality the “systems” that we use to retrieve information are synergistic combinations of collections, machines, and processes that people use to search the collection(s) using the machine(s). Model-based evaluations such as those pioneered at Cranfield and now used in TREC, CLEF, and SIGIR focus on some functions of the machine (in particular, how best to build ranked lists). This paper expands that focus to examine what we have learned about the processes by which those machines will be used to perform Cross-Language Information Retrieval (CLIR), concluding with a brief description of how that perspective informs the nature of our research in DARPA’s new GALE program. Categories and Subject Descriptors H.1.2 [Information Systems]: User/Machine System – Human Information Processing General Terms: Design, Experimentation Keywords: Interactive Cross-Language Information Retrieval, User-Assisted Query Translation, Experiments, Search Behaviors 1. INTRODUCTION While it is true that Information Retrieval (IR) technology can be used as a component of some larger system (e.g., clustering, text classification, or question answering), in the normal use of the term “IR” we focus on cases in which the need for a search arises from some human need and the utility of the results will ultimately be judged by the person whose need must be satisfied. In this sense, the dichotomy between “batch” and “interactive” IR is a false one: all IR is ultimately interactive IR. What differs is not what we seek to achieve, but rather what we choose to evaluate. In the Cranfield tradition, we see to determine how well machines can identify documents in a collection that a searcher might wish to see. This abstract formulation covers a broad range of research questions, including query-based topic-oriented ranked retrieval in TREC/CLEF/NTCIR, recent work on query- based sentiment-oriented ranked retrieval in TREC and NTCIR, example-based event-oriented exact-match retrieval in TDT, and evolving simulations of explicit relevance feedback as a basis for exact-match retrieval in the TREC adaptive filtering task and the final year of the TDT topic tracking task. In the Cranfield tradition, we take the information need as fixed (or evolving in some easily modeled way) and we vary the design of the machine. The research tradition known as “relevance studies” adopts the opposite perspective: the available automated capabilities are (implicitly) taken as fixed, and the research focuses on understanding what factors would cause a user to value the content of a document. The term “relevance” in those studies is used in a broader sense than is typical at SIGIR—closer to what we would normally call “utility.” The research methods used to explore this broader notion of relevance are also different from the normal discourse at SIGIR, drawing heavily on cognitive psychology and often relying more on qualitative than quantitative methods of inquiry. The two research traditions intersect in at least one important way: relevance studies quite consistently indicate that the topical relevance that we focus on in the Cranfield tradition is often a dominant factor in the choices made by users (some others are recency, authority, availability, and comprehensibility). A third related research tradition focuses on the process of asking questions. An example of this that will be familiar to many SIGIR participants is Belkin’s “Anomalous States of Knowledge,” which observes that we design our machines to answer questions that are well formed, but that those machines are often used by searchers who bring an incomplete understanding of what they are really looking for [1]. Over the years, we have found Taylor’s four types of questions (what you really want to know, what you think you want to know, what you can articulate you want to know, and what you can formulate in your machine’s query language) to be a useful framework for thinking about this [2]. Two important lines of research emerge from this perspective: the “reference interview” process, and query (re-) formulation strategies. Both have been extensively studied in the context of training information professionals (e.g., reference librarians). The closest thing that we have to a unifying theory for these three disparate lines of research is sense-making, for which Dervin’s iterative situation-gap-bridge is perhaps the best known model [3]. Somewhat oversimplifying in order to draw the connections Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advent-age and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGIR’06 Workshop, August 11, 2006, Seattle, Washington, USA. Copyright 2006 ACM 1-58113-000-0/00/0004…$5.00.