An Analysis Framework for Search Sequences Qiaozhu Mei ∗ Kristina Klinkner Ravi Kumar Andrew Tomkins School of Information Yahoo! Research University of Michigan 701 First Avenue Ann Arbor, MI 48103. Sunnyvale, CA 94089 qmei@umich.edu {klinkner,ravikumar,atomkins}@yahoo-inc.com ABSTRACT In this paper we present a general framework to study sequences of search activities performed by a user. Our framework provides (i) a vocabulary to discuss types of features, models, and tasks, (ii) straightforward feature re-use across problems, (iii) realistic base- lines for many sequence analysis tasks we study, and (iv) a simple mechanism to develop baselines for sequence analysis tasks beyond those studied in this paper. Using this framework we study a set of fourteen sequence analysis tasks with a range of features and mod- els. While we show that most tasks benefit from features based on recent history, we also identify two categories of “sequence- resistant” tasks for which simple classes of local features perform as well as richer features and models. Categories and Subject Descriptors. H.3.m [Information Stor- age and Retrieval]: Miscellaneous General Terms. Algorithms, Experimentation, Measurements Keywords. Session analysis, Sequential analysis, Query logs 1. INTRODUCTION Consider a user who visits a search engine and queries for “mus- tang” then queries for “ford mustang” then queries for “nova.” To a human, it is immediately clear that the user is searching for cars, and that the final query of the sequence is for the car produced by automobile manufacturer Chevrolet. However, no major search engine provides a single reference to the Chevy Nova in the first fifty results. As search engines move beyond simple navigational queries to helping users with longer-running tasks, an ability to un- derstand the need behind a user’s query, using information about the query sequence, is becoming critical. Query processing for the “nova” query above could be improved by modeling the previous one or two queries to understand that the context of the session is automobiles. But it could also have been improved by a wider variety of sequence analysis techniques. Per- haps the user has been researching cars for the past month, so the ∗ Most of this work was done while the author was visiting Yahoo! Research. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CIKM’09, November 2-6, 2009, Hong Kong, China. Copyright 2009 ACM 978-1-60558-512-3/09/11 ...$10.00. query processing could be improved even if “nova” were the first query of the session. Perhaps the user never clicks on ads when performing automobile queries, so ads should be suppressed. Or perhaps the user often clicks on query suggestions after entering the name of a car, so “chevy nova” should be offered as a sug- gestion. Or perhaps other users who search for multiple cars in close proximity tend to click on results that offer model informa- tion rather than price information. Or perhaps the user entered this same query yesterday and is interested in picking up from where she left off. As these examples show, there are valuable improve- ments that require access only to the last one or two queries, and other improvements that may require a deeper lens into the user’s history, or even an aggregate view into the query-specific sequence behavior of other users. The body of work surrounding analysis of individual queries is both broad and mature, spanning multiple fields. The associated analysis of query sequences is at a much earlier point in its devel- opment. There have been a number of papers presenting ad hoc analysis of user sessions for understanding online behavior [12, 14, 13], improved search query processing [21], and understanding of reformulation behavior [19]. However, as yet there has been little formal work in developing sequential analysis frameworks for such problems; we undertake the development of such a framework. Our proposed framework (Section 3) captures sequences of user behavior at multiple levels of granularity from sessions to cohesive sub-tasks, blocks of related queries, individual queries, clicks, and eye-tracking fixations. The framework supports (i) a vocabulary to discuss types of features, models, and tasks, (ii) easy feature re-use across problems, (iii) realistic baselines for the sequence analysis problems we study, and (iv) a simple mechanism to develop base- lines for sequence analysis tasks beyond those studied in this paper. We selected 14 distinct search sequence analysis tasks spanning labeling, prediction, and sequence categorization (Section 4), and mapped each one into our framework. Some of these tasks have fully-automated labels such as predicting whether the next click will be on an ad, while others require editorially generated labels such as segmenting a sequence into missions or goals undertaken by the user. We present a set of 42 task-independent base features plus an additional set of “global” features aggregated across mul- tiple sessions by the current user or a broader population of users. These generic features are applicable to a broad range of tasks; we study them to prove the feasibility of feature re-use in such a frame- work — any individual task could be further improved by careful feature engineering. We compare approaches to our fourteen tasks using a log of 1.2M queries and 17K editorial judgments from a ma- jor search engine (Sections 5 and 6). Our results provide a charac- terization of the appropriate feature types and modeling approaches for a wide range of sequential analysis problems. 1991