User-focused task-oriented MT evaluation for wikis: a case study Federico Gaspari, Antonio Toral and Sudip Kumar Naskar School of Computing Dublin City University Dublin 9, Ireland {fgaspari, atoral, snaskar}@computing.dcu.ie Abstract This paper reports on an evaluation ex- periment focusing on statistical machine translation (MT) software integrated into a more complex system for the syn- chronization of multilingual information contained in wiki sites. The experiment focused on the translation of wiki entries from German and Dutch into English carried out by ten media professionals, editors, journalists and translators work- ing at two major media organizations who post-edited the MT output. The in- vestigation concerned in particular the adequacy of MT to support the transla- tion of wiki pages, and the results in- clude both its success rate (i.e. MT ef- fectiveness) and the associated confi- dence of the users (i.e. their satisfaction). Special emphasis is laid on the post-editing effort required to bring the output to publishable standard. The results show that overall the users were satisfied with the system and regarded it as a potentially useful tool to support their work; in particular, they found that the post-editing effort required to attain translated wiki entries in English of pub- lishable quality was lower than translat- ing from scratch. 1 Introduction 1.1 Background to the evaluation The evaluation reported in this paper was con- ducted as part of the three-year CoSyne project, funded by the EU under the FP7 scheme. 1 The project consortium includes seven partners: three academic institutions, i.e. University of Amsterdam (UvA, The Netherlands) as coordi- nator, Fondazione Bruno Kessler (FBK, Italy) and Dublin City University (DCU, Ireland); one research organization, the Heidelberg Institute for Theoretical Studies (HITS, Germany); and three end-user partners, Deutsche Welle (DW, Germany), the Netherlands Institute for Sound and Vision (NISV, The Netherlands) and Wiki- media Foundation Netherlands (WMF). The aim of the technology developed within the CoSyne project is to facilitate the synchro- nization of the contents of wiki sites across dif- ferent languages. In this context, machine trans- lation (MT) is used as part of an integrated sys- tem which includes other modules that take care of textual entailment, document structure mod- eling, overlap synchronization (to check how similar or different wiki entries on the same top- ic in multiple languages are), insertion point de- tection (to establish where new machine-trans- lated information should be added or replaced following some edits), etc. The statistical MT software incorporated within the CoSyne system was developed by UvA (Martzoukos and Monz, 2010). Toral et al. (2011) conducted a comparative evaluation of the CoSyne MT software against four free web- based MT systems over data sets from the news domain based on a range of state-of-the-art au- tomatic metrics. In its first year up to February 2011, the project has covered three language pairs, i.e. German, Dutch and Italian from and into English. By the end of the project, Turkish and Bulgarian will also be included to extend 1 More details of the project are available at www.- cosyne.eu 9HQWVLVODY =KHFKHY HG 3URFHHGLQJV RI WKH 7KLUG -RLQW (0&1*/ :RUNVKRS ｴ%ULQJLQJ 07 WR WKH 8VHU 5HVHDUFK 0HHWV 7UDQVODWRUVｵ -(& ｷ SS ｲ /X[HPERXUJ  2FWREHU  