Entailment-based Fully Automatic Technique for Evaluation of Summaries Pinaki Bhaskar, 1 Partha Pakray, 2 Alexander Gelbukh, 3 Sivaji Bandyopadhyay 1 1 Department of Computer Science and Engineering, Jadavpur University, Kolkata – 700032, India 2 Department of Computer and Information System, Norwegian University of Science and Technology, Sem Sælandsvei 7-9, NO-7491, Trondheim, Norway 3 Center for Computing Research, National Polytechnic Institute, Mexico City, Mexico {pinaki.bhaskar, parthapakray}@gmail.com, gelbukh@gelbukh.com, sivaji_cse_ju@yahoo.com Abstract. We propose a fully automatic technique for evaluating text summaries without the need to prepare the gold standard summaries man- ually. A standard and popular summary evaluation techniques or tools are not fully automatic; they all need some manual process or manual reference summary. Using recognizing textual entailment (TE), automat- ically generated summaries can be evaluated completely automatically without any manual preparation process. We use a TE system based on a combination of lexical entailment module, lexical distance module, Chunk module, Named Entity module and syntactic text entailment (TE) module. The documents are used as text (T) and summary of these doc- uments are taken as hypothesis (H). Therefore, the more information of the document is entailed by its summary the better the summary. Com- paring with the ROUGE 1.5.5 evaluation scores over TAC 2008 (for- merly DUC, conducted by NIST) dataset, the proposed evaluation tech- nique predicts the ROUGE scores with a accuracy of 98.25% with respect to ROUGE-2 and 95.65% with respect to ROUGE-SU4. Keywords: Automatic text summarization, summary evaluation, recognizing textual entailment. 1 Introduction Automatic summaries are usually evaluated using human generated reference summar- ies or some manual efforts. Summaries generated automatically from the documents are difficult to evaluate using completely automatic evaluation process or tool. 11 Research in Computing Science 65 (2013) pp. 11–23