The Second PASCAL Recognising Textual Entailment Challenge Roy Bar-Haim * , Ido Dagan * , Bill Dolan ** , Lisa Ferro † , Danilo Giampiccolo ‡ , Bernardo Magnini ⋆ , Idan Szpektor * * Computer Science Department, Bar-Ilan University, Ramat Gan 52900, Israel ** Microsoft Research, Redmond, WA 98052, USA † The MITRE Corporation, 202 Burlington Rd., Bedford, MA 01730, USA ‡ CELCT, Via dei Solteri 38, Trento, Italy ⋆ ITC-irst, Istituto per la Ricerca Scientifica e Tecnologica 38050 Povo, Trento, Italy Abstract This paper describes the Second PASCAL Recognising Textual Entailment Chal- lenge (RTE-2). 1 We describe the RTE- 2 dataset and overview the submissions for the challenge. One of the main goals for this year’s dataset was to pro- vide more “realistic” text-hypothesis ex- amples, based mostly on outputs of ac- tual systems. The 23 submissions for the challenge present diverse approaches and research directions, and the best results achieved this year are considerably higher than last year’s state of the art. 1 Introduction 1.1 Textual entailment recognition Textual entailment recognition is the task of decid- ing, given two text fragments, whether the mean- ing of one text is entailed (can be inferred) from another text (see section 2.2 for the specific oper- ational definition of textual entailment assumed in the challenge). This task, introduced by Dagan and Glickman (2004), captures generically a broad range of inferences that are relevant for multiple applica- tions. For example, a Question Answering (QA) system has to identify texts that entail the expected answer. Given the question “Who is John Lennon’s widow?”, the text “Yoko Ono unveiled a bronze statue of her late husband, John Lennon, to com- plete the official renaming of England’s Liverpool 1 http://www.pascal-network.org/Challenges/RTE2 Airport as Liverpool John Lennon Airport.” entails the expected answer “Yoko Ono is John Lennon’s widow”. Similarly, semantic inference needs of other text understanding applications such as Infor- mation Retrieval (IR), Information Extraction (IE), and Machine Translation evaluation can be cast as entailment recognition (Dagan et al., 2006). Textual entailment may serve as a unifying generic framework for modeling semantic inference, which so far have been addressed independently by separate application-specific research communities. Its formulation as a mapping between texts makes it independent of concrete semantic interpretations, which then become a mean rather than the goal. For example, in word sense disambiguation, it is not al- ways easy to define explicitly the right set of senses to choose from. In practice, however, it is suffi- cient for most applications to determine whether a word meaning in a given context lexically entails another word. For instance, the occurrence of the word “chair” in the sentence “IKEA announced a new comfort chair” entails “furniture”, while its oc- currence in the sentence “MIT announced a new CS chair position” does not. Thus, proper modeling of lexical entailment in context may alleviate the need to interpret word occurrences into explicitly stipu- lated senses. Eventually, we hope that research on textual en- tailment will lead to the development of entailment “engines”, which will be used as a standard module in many applications (similar to the role of part-of- speech taggers and syntactic parsers in current NLP applications).