Stumping e-rater:challengingthevalidityof automatedessayscoring DonaldE.Powers*,JillC.Burstein,MartinChodorow, MaryE.Fowles,KarenKukich Educational Testing Service, Rosedale Road, Princeton, NJ 08541, USA Abstract For this study, various parties were invited to ‘‘challenge’’ e-rater—an automated essay scorerthatreliesonnaturallanguageprocessingtechniques—bycomposingessaysinresponse toGraduateRecordExaminations(GRE 1 )WritingAssessmentpromptswiththeintention ofunderminingitsscoringcapability.Specifically,usingdetailedinformationabout e-rater’s approachtoessayscoring,writerstriedto‘‘trick’’thecomputer-basedsystemintoassigning scoresthatwerehigherorlowerthandeserved. E-rater’sautomatedscoresonthese‘‘problem essays’’werecomparedwithscoresgivenbytwotrained,humanreaders,andthedifference betweenthescoresconstitutedthestandardforjudgingtheextenttowhich e-rater wasfooled. Challengersweredifferentiallysuccessfulinwritingproblematicessays.Asawhole,theywere moresuccessfulintricking e-rater intoassigningscoresthatweretoohighthaninduping e- rater into awarding scores that were too low. The study provides information on ways in which e-rater,andperhapsotherautomatedessayscoringsystems,mayfailtoprovideaccu- rateevaluations,ifusedasthesolemethodofscoringinhigh-stakesassessments.Theresults suggestpossibleavenuesforimprovingautomatedscoringmethods. # 2002ElsevierScience Ltd.Allrightsreserved. Keywords: Writing assessment; Graduate Record Examinations (GRE); Validity; Automated scoring; Essayscoring;Computer-assisted 1. Introduction Complexperformanceassessmentsthattypicallyrequiretesttakerstoperformor produce (instead of to recognize or select) are becoming increasingly popular (Aschbacher, 1991). Although such constructed-response measures offer distinct advantages over traditional multiple-choice measures (see Bennett & Ward, 1993, ComputersinHumanBehavior18(2002)103–134 www.elsevier.com/locate/comphumbeh 0747-5632/02/$-seefrontmatter # 2002ElsevierScienceLtd.Allrightsreserved. PII:S0747-5632(01)00052-8 * Correspondingauthor.Tel.:+1-609-734-5573;fax:+1-609-734-1755. E-mail address: dpowers@ets.org(D.E.Powers).