(Linguistic) Science Through Web Collaboration in the ANAWIKI Project Udo Kruschwitz University of Essex udo@essex.ac.uk Jon Chamberlain University of Essex jchamb@essex.ac.uk Massimo Poesio University of Essex and Universit`a di Trento poesio@essex.ac.uk Abstract Despite the impressive progress made in recent years in all areas of natural lan- guage processing there are still tasks that do not perform well enough to be used in everyday applications. One example is anaphora resolution. The most promising approach to get significant improvements in this area is to create sufficiently large linguistically annotated resources which can then be used to train, for example, machine learning systems. Annotated cor- pora of the size needed for modern compu- tational linguistics research cannot how- ever be created by small groups of hand- annotators; but ESP and similar games have demonstrated how it might be pos- sible to do this through Web collabora- tion. This paper reports on the ongoing work on Phrase Detectives, a game devel- oped in the ANAWIKI project designed for collaborative linguistic annotation on the Web. Of particular concern here are the measures that assure high-quality an- notations. 1 Introduction The statistical revolution in natural language processing (NLP) has resulted in the first NLP systems and components really usable on a large scale, from part-of-speech (POS) taggers to parsers [7]. But it has also raised the problem of creating the large amounts of annotated linguistic data needed for training and evaluating such sys- tems. Potential solutions to this problem include semi-automatic annotation, and machine learn- ing methods that make better use of the avail- able data. Unsupervised or semi-supervised tech- niques hold great promise, but for the foreseeable future at least, the greatest performance improve- ments are still likely to come from increasing the amount of data to be used by supervised training methods, which crucially rely on hand-annotated data. Traditionally, this requires trained anno- tators, which is prohibitively expensive both fi- nancially and in terms of person-hours (given the number of trained annotators available) on the scale required. Recently, however, web collaboration has started to emerge as a viable alternative. Wikipedia and similar initiatives have shown that a surprising number of individuals are willing to help with resource creation and scientific ex- periments. The Open Mind Common Sense project [12] demonstrated that such individuals are also willing to participate in the creation of databases for Artificial Intelligence (AI), and von Ahn showed that web games are an effective way of motivating subjects to annotate data for ma- chine learning purposes [16, 17]. The goal of the ANAWIKI project 1 is to experiment with Web collaboration as a solu- tion to the problem of creating large-scale lin- guistically annotated corpora, both by develop- ing tools through which members of our scien- tific community can participate in corpus cre- ation through annotation tools with a Web in- terface and through the use of game-like inter- faces. We will present ongoing work on Phrase Detectives 2 , a game designed to collect judgments about anaphoric annotations. We will also report results which include a substantial corpus of an- notations that has already been collected. The paper will be structured as follows. We will start with a brief discussion of some related work (Section 2) and address the main issues arising (Section 3). We will then introduce the Phrase Detectives game (Section 4) followed by a discus- sion of how we enforce quality control (Section 5). Two further sections will discuss implemen- tational issues and first results before we outline future work. 1 http://www.anawiki.org 2 http://www.phrasedetectives.org