Introspection and Adaptable Model Integration for Dialogue-based Question Answering Daniel Sonntag German Research Center for AI (DFKI) Stuhlsatzenhausweg 3, 66123 Saarbr¨ ucken, Germany sonntag@dfki.de Abstract Dialogue-based Question Answering (QA) is a highly complex task that brings together a QA sys- tem including various natural language processing components (i.e., components for question classi- fication, information extraction, and retrieval) with dialogue systems for effective and natural commu- nication. The dialogue-based access is difficult to establish when the QA system in use is complex and combines many different answer services with different quality and access characteristics. For example, some questions are processed by open- domain QA services with a broad coverage. Oth- ers should be processed by using a domain-specific instance ontology for more reliable answers. Dif- ferent answer services may change their charac- teristics over time and the dialogue reaction mod- els have to be updated according to that. To solve this problem, we developed introspective methods to integrate adaptable models of the answer ser- vices. We evaluated the impact of the learned mod- els on the dialogue performance, i.e., whether the adaptable models can be used for a more conve- nient dialogue formulation process. We show sig- nificant effectiveness improvements in the resulting dialogues when using the machine learning (ML) models. Examples are provided in the context of the generation of system-initiative feedback to user questions and answers, as provided by heteroge- neous information services. 1 Introduction You are visiting a football match in Berlin and you take a mobile mini computer with you which is able to answer ques- tions in real-time. If you ask, “Who was world champion in 1990?”, state-of-the-art question answering systems for this specific domain with a natural language understanding com- ponent and access to a knowledge base should be able to an- swer with great accuracy, “That was Germany”. Later, since you are new to the city, you are on a sightseeing tour. During the bus ride, you pass Castle Charlottenburg which arouses your curiosity, “I wonder who might have built Castle Char- lottenburg?” Unfortunately, most of the specific domain question an- swering systems would respond with “No Answer” after checking the knowledge base where the answer cannot be found (a task which might also consume a lot of time). In this situation, the user would be very dissatisfied with the system. Most existing approaches focus on improving the natural language understanding capability and/or the quality of the provided factual answers. Such improvements are im- portant, but do not enhance the robustness of the system on a large scale. For example, in order to enhance the range of possible questions to be answered, open domain access us- ing a search engine could be realised as a fallback strategy. This would potentially enhance recall, but also mean a loss of precision combined with problems of result provenance— the results are less reliable. Further problems occur if dif- ferent information sources have different access characteris- tics, e.g., Web Services answering questions such as “What’s the weather like tomorrow?” could be temporarily unavail- able. This leads to a situation where efficiency, effectiveness, and naturalness of the question answering dialogue is hard to achieve. 1 We focus on how to improve the QA system with a suitable dialogue within the QA dialogue and QA system’s capabili- ties. In the following user-system dialogue example, adequate question feedback is shown in bold: 1. U: “When was Brazil world champion?” 2. S: “In the following 5 years: 1958 (in Sweden), 1962 (in Chile), 1970 (in Mex- ico), 1994 (in USA), 2002 (in Japan).” (6000 ms) Later ... 3. U: “What can I do in my spare time on Saturday?” 4. S: “Sorry, services for such questions were unavailable a short while ago, shall I continue? This may take a minute or so ...” (600 ms) 5. U: “Ah okay, I can wait.” 6. S: “Unfortunately, this service only produces empty results at the moment.” (52000 ms) Later on the bus ride ... 1 Multi-strategy approaches use different QA subsystems when searching for answers to questions. An increasing number of open- domain QA systems have started using several retrieval approaches (e.g., by employing different search engines and different query expansions) and multiple answer extractors (e.g., keyword-based, concept-based, or based on user feedback, etc.). Particularly, the need for combining different data sources is of great importance. 1549