Understanding Complex Multi-sentence Entity seeking Questions Danish Contractor 1, 2* , Barun Patra 1† , Mausam 1 , Parag Singla 1 1 Indian Institute of Technology, New Delhi 2 IBM Research AI, New Delhi dcontrac@in.ibm.com,barunpatra95@gmail.com, mausam,parags@cse.iitd.ac.in Abstract We present the novel task of understanding multi-sentence entity-seeking questions (MSEQs) i.e, questions that may be expressed in multiple sentences, and that expect one or more entities as an answer. We formulate the problem of under- standing MSEQs as a semantic labeling task over an open representation that makes minimal assumptions about schema or ontology specific semantic vocabulary. At the core of our model, we use a BiDiLSTM (bi-directional LSTM) CRF and to overcome the challenges of operating with low training data, we supplement it by using hand-designed features, as well as hard and soft constraints spanning multiple sentences. We find that this results in a 6-7pt gain over a vanilla BiDiL- STM CRF. We demonstrate the strengths of our work using the novel task of answering real-world entity-seeking ques- tions from the tourism domain. The use of our labels helps answer 53% more questions with 42 % more accuracy as compared to baselines. Introduction We introduce the novel task of understanding multi-sentence questions. Specifically, we focus our attention on multi- sentence entity-seeking questions (MSEQs) i.e., questions that expect one or more entities as answer. Such questions are commonly found in online forums, blog posts, discus- sion boards etc and come from a variety of domains includ- ing tourism, books and consumer products. Figure 1 shows an example MSEQ from a tourism forum, where the user is interested in finding a hotel that satisfies some constraints and preferences; an answer to this question is thus the name of a hotel (entity) which needs to satisfy some properties such as being a ‘budget’ option. A prelim- inary analysis of such entity-seeking questions from online forums reveals that almost all of them contain multiple sen- tences – they often elaborate on a user’s specific situation before asking the actual question. In order to understand and answer such a user question, we convert the question into a machine representation con- sisting of labels identifying the informative portions in a * This work was carried out as part of PhD research at IIT Delhi. The author is also a regular employee at IBM Research AI. † Work carried out when Barun was an undergraduate student at IIT Delhi. Copyright c  2019, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. question. We are motivated by our work’s applicability to a wide variety of domains and therefore choose not to re- strict the representation to use a domain-specific vocabu- lary. Instead, we design an open semantic representation, in- spired in part by Open QA (Fader, Zettlemoyer, and Etzioni 2014), in which we explicitly annotate the answer (entity) type; other answer attributes, while identified, are not fur- ther categorized. Eg. in Figure 1 ‘place to stay’ is labeled as entity.type while ‘budget’ is labeled as an entity.attr. We also allow attributes of the user to be represented. Domain specific annotations such as location for tourism questions are permitted. Such labels are then be supplied to a down- stream information retrieval (IR) or a QA component to di- rectly present an answer entity. We pose the task of understanding MSEQs as a se- mantic labeling (shallow parsing) 1 task where tokens from the question are annotated with a semantic label from our open representation. However, in contrast to related litera- ture on semantic role labeling (Yang and Mitchell 2017), slot filling tasks (Bapna et al. 2017) and query formula- tion (Wang and Nyberg 2016; Vtyurina and Clarke 2016; Nogueira and Cho 2017), semantic parsing of MSEQs raise several novel challenges. MSEQs express a wide variety of intents and requirements which span across multiple sen- tences, requiring the model to capture within-sentence as well as inter-sentence interactions effectively. In addition, questions can be unnecessarily belabored requiring the sys- tem to reason about what is important and what is not. Lastly, we find that generating training data for parsing MSEQs is hard due to the complex nature of the task, further requiring the models to operate in low training data settings. In order to address these challenges and label MSEQs, we use a bi-directional LSTM CRF (BiDiLSTM CRF) (Huang, Xu, and Yu 2015) and extend it in two ways: (i) We encode knowledge by incorporating hand-designed features as well as semantic constraints over the entire multi-sentence ques- tion during end-to-end training. (ii) We use partially labeled questions, that are easier to source, to improve training. In summary, our paper makes the following contributions: 1. We present the novel task of understanding multi-sentence entity-seeking questions (MSEQs). We define open se- 1 We use the phrases ‘semantic labeling’ and ’semantic parsing’ interchangeably in this paper.