IsNL? A Discriminative Approach to Detect Natural Language Like Queries for Conversational Understanding Asli Celikyilmaz, Gokhan Tur, Dilek Hakkani-T¨ ur Microsoft Silicon Valley, USA {asli,gokhan.tur,dilek}@ieee.org Abstract While data-driven methods for spoken language understanding (SLU) provide state of the art performances and reduce main- tenance and model adaptation costs compared to handcrafted parsers, the collection and annotation of domain-specific natural language utterances for training remains a time-consuming task. A recent line of research has focused on enriching the training data with in-domain utterances by mining search engine query logs to improve the SLU tasks. However genre mismatch is a big obstacle as search queries are typically keywords. In this paper, we present an efficient discriminative binary classifica- tion method that filters large collection of online web search queries only to select the natural language like queries. The training data used to build this classifier is mined from search query click logs, represented as a bipartite graph. Starting from queries which contain natural language salient phrases, random graph walk algorithms are employed to mine corresponding keyword queries. Then an active learning method is employed for quickly improving on top of this automatically mined data. The results show that our method is robust to noise in search queries by improving over a baseline model previously used for SLU data collection. We also show the effectiveness of detected natural language like queries in extrinsic evaluations on domain detection and slot filling tasks. Index Terms: natural language, keyword search, natural lan- guage understanding, web search, semantic parsing. 1. Introduction The goal of human to machine dialog systems is to provide the user with a seamless experience, in which users can speak to the machine naturally as if they are conversing with another hu- man. The spoken language understanding (SLU) component of the dialog systems plays a crucial role in extracting the re- quested information from the user input. A typical SLU engine employs several semantic parsing methods such as domain de- tection, user act (intent) determination or slot filling to better understand the user input [1]. Compared to a typical keyword- based web search query, the input to a dialog system is a natural language (NL) utterance, which usually contains verbs, phrases and clauses (see Table-1). Most state of the art approaches to SLU are based on su- pervised machine learning methods, which use training data from the corresponding application domain. Among these ap- proaches are generative models such as hidden Markov mod- els [2], discriminative classification methods [3, 4, 5] and prob- abilistic context free grammars [6, 7]. Although very effective in semantic parsing of utterances, they require a large number of in-domain NL sentences. Manually collecting NL sentences for training does not scale well because of the language vari- Natural Language (NL) Queries (S) what time do [lakers]team play in the [opening day] date (M) what are some [recent] date [funny]genre movies (G) [top-rated]review [wii]type games for [kids]genre (M) what are all of [channing tatum]artist ’s movies Keyword Queries · calories per day · wifi signal booster xbox 360 · oscar winners [2013] date · [jessica simpson]artist · [stolen honor : wounds that never heal]movie Table 1: Sample natural language and keyword queries mined from web search query click logs. Queries are labeled with selected semantic tags including domain labels; (M):Movie, (G):Game, (S):Sports, and slot tags, e.g., type, genre, artist, etc. ability issues of the NL interfaces. In particular, not only there is no limitation on what the user might say, but the models must generalize from a tractably small amount of training data. In a closely related research area, the information retrieval (IR) researchers have recently shown that the web search query click logs (QCL) are valuable resources that can be used as im- plicit supervision to improve the predictions of the future search results [8]. Specifically, the web search query-click log data in- cludes the queries issued by the users. The queries have cor- responding url-links that the users clicked from a list of urls returned by the search engine (see Figure 1). It is the the strong semantic relation between the queries issued by the users and the clicked urls that help to understand the queries. Only re- cently this relational but noisy data has been a valuable infor- mation source for building spoken dialog systems. For instance a recent study on the use of QCL data for building SLU models has shown improvements, in particular, on the domain and slot detection tasks [9, 10]. Typically, they mine the QCL data to extract additional NL-like queries, which are then used to build more robust and efficient SLU models. With the above improvements on SLU in mind, in this pa- per, we focus on rather efficient methods to mine NL queries for improving the SLU models. We start by summarizing the mining methods used in the previous SLU work, which sets the background of this paper. We then present a new feature-based NL classifier model, namely the IsNL to classify search queries into “NL” or “keyword” categories based on semantic, syntactic and structural features extracted from the queries and external resources. Using an active learning method, we select the train- ing data that best generalizes the SLU models. Specifically, we collect queries to: (i) extend the vocabulary; (ii) and capture NL patterns and phrases that did not exist in the training data. Our end goal is to improve the performance of the understand- ing tasks, specifically, domain detection and slot filling. In the empirical evaluations we show that the IsNL classifier is an ef- Copyright  2013 ISCA 25 - 29 August 2013, Lyon, France INTERSPEECH 2013 2569