Baseline Systems for NTCIR-5 CLQA1: An Experimentally Extended QBTE Approach Yutaka Sasaki Department of Natural Language Processing ATR Spoken Language Communication Research Laboratories 2-2-2 Hikaridai, Keihanna Science City, Kyoto, 619-0288 Japan yutaka.sasaki@atr.jp Abstract This paper reports the performance of baseline sys- tems for the NTCIR CLQA1 Task. As a task orga- nizer for CLQA1 and hence a creator of both sam- ple and formal run questions of JE/EJ subtasks, I have deemed it ideal to completely exclude the effect of hu- man knowledge. Consequently, we have taken an ap- proach to statistically construct baseline CLQA sys- tems using only a QA data set. We employed the QBTE (Question-Biased Term Extraction) Model and a pre- liminary extended model of the QBTE Model, which statistically constructs QA systems only from question- answer pairs. The extended model uses word and POS dictionaries in addition to Q/A pairs. We constructed CLQA systems on the basis of a sample Q/A data set (300 Q/A pairs) that was provided by CLQA1 organiz- ers. It took only a week to develop baseline CLQA sys- tems, QATRO-JE/EJ/CE, but the results showed that 300 Q/A pairs is too small a number in a CLQA set- ting thus that only a small number of questions could be correctly answered by our system. An additional analysis after the formal run revealed that the cross- lingual setting makes it more difficult for the system to retrieve related documents and pin-point answers be- cause of discrepancies between translated words and words actually appearing in questions and target arti- cles. Keywords: NTCIR, Cross-Lingual QA, QBTE 1 Introduction One of the aims of the NTCIR CLQA1 (Pilot) Task is to evaluate the performance of Question Answering (QA) systems in the Cross-Lingual settings. The framework of Cross-Lingual QA is for find- ing answers to a question in language X (source lan- guage) from documents in another language Y (tar- get language). In NTCIR CLQA1, this is represented as the XY subtask. Since CLQA1 is a new pilot task in NTCIR, translating answers back to the source lan- guage is outside the scope of CLQA1. Topics to be investigated are as follows. • How to create Cross-Lingual QA (CLQA) sys- tems between Asian languages • The difference between a monolingual QA and cross-lingual QA • The extent to which a CLQA system for the XY subtask degrades from the monolingual QA sys- tem in language Y • The type of Machine Translation techniques that are effective for CLQA As an organizer of CLQA1, I created the base- line CLQA systems QATRO 1 for JE/EJ/CE subtasks. To exclude an effect of my knowledge about CLQA test data, QATRO was built on a purely statistical method based on QBTE (Question-Biased Term Ex- traction) [11] and an experimentally extended QBTE Model, which only relies on Q/A data sets. QBTE is a kind of Statistical Question Answer- ing approaches [3, 4, 6, 7, 12, 13, 14, 10, 2, 5]. We employed the machine learning technique Maximum Entropy Models (MEMs) [1] to extract answers from combined features of question features and document features. 2 Overview 2.1 Subtasks Participated We participated in the following two CLQA1 sub- tasks based on an extended QBTE Model. JE subtask: We submitted three official and three un- official runs. 1 QATRO stands for Question Answering system with TRansla- tiOn Proceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan