Assessing Writing 12 (2007) 26–43 Re-training writing raters online: How does it compare with face-to-face training? Ute Knoch ∗ , John Read 1 , Janet von Randow 2 Department of Applied Language Studies and Linguistics, University of Auckland, New Zealand Available online 21 June 2007 Abstract The training of raters for writing assessment through web-based programmes is emerging as an attractive and flexible alternative to the conventional method of face-to-face training sessions. Although some online training programmes have been developed, there is little published research on them. The current study aims to compare the effectiveness of online and face-to-face training in the context of a large-scale academic writing assessment for students entering a major English-medium university. A team of 16 raters, divided into two groups of 8, all initially rated a set of 70 scripts. In the training phase, the online group rated 15 benchmark scripts online and received immediate feedback, whereas the face-to-face group received individual feedback on their pre-training performance, rated the 15 scripts at home and then met for a face-to-face session. After the training, both groups re-rated the initial 70 scripts and then reported their attitudes towards the different forms of training by means of questionnaires and interviews. According to the statistical results, using multi-faceted Rasch measurement, both types of training were effective overall, but the self-report data revealed various responses favouring one type or the other. The findings are discussed in terms of the factors influencing rater responsiveness and the refinements that are needed for future rater training programmes. © 2007 Elsevier Inc. All rights reserved. Keywords: Writing assessment; Rater training; Online rater training; Multi-faceted Rasch analysis; DELNA; Rater effect 1. Background Although there have been substantial advances in automated rating of writing in recent years (Jamieson, 2005), it is still the norm in writing assessment to use human raters. Unfortunately, their ∗ Corresponding author. Tel.: +64 9 3737599x84697; fax: +64 9 3082360. E-mail addresses: u.knoch@auckland.ac.nz (U. Knoch), ja.read@auckland.ac.nz (J. Read), j.vonrandow@auckland.ac.nz (J. von Randow). 1 Tel.: +64 9 3737599x87673; fax: +64 9 3082360. 2 Tel.: +64 9 3737599x82427; fax: +64 9 3082360. 1075-2935/$ – see front matter © 2007 Elsevier Inc. All rights reserved. doi:10.1016/j.asw.2007.04.001