Peer evaluation in online anchored discussion for an increased local relevance of replies J. van der Pol a, * , W.F. Admiraal a,b , P.R.J. Simons a a IVLOS Institute of Education, Utrecht University, P.O. Box 80127, 3508 TC Utrecht, The Netherlands b ILO Graduate School of Teaching and Learning, University of Amsterdam, P.O. Box 19268, 1000 GG Amsterdam, The Netherlands article info Article history: Available online 5 November 2007 Keywords: Collaborative learning Anchored discussion Peer-feedback Dialogue analysis Coherence Relevance abstract This study investigates the use of an evaluation function to increase the local relevance of replies in online anchored discussion. Being implemented in a university course on French linguistics, a regular sys- tem for anchored discussion is compared with two versions that are enhanced with an integrated eval- uation function. The function asks students to evaluate the relevance of each others’ replies. To compare between experimental and control conditions, the collaboration protocols are analysed with a newly developed coding scheme for the local relevance of replies. Statistical processing of the data is done with a multilevel approach and results indicate that an evaluation function can effectively increase the local relevance of students’ replies, but only if it is actually used to a certain extent. Using the evaluation func- tion is hypothesized to increase students’ awareness of the importance of writing relevant replies. Ó 2007 Elsevier Ltd. All rights reserved. 1. Introduction Computer supported collaborative learning (CSCL) offers stu- dents possibilities for a deep and active processing of their subject matter, especially when complementing face-to-face (F-2-F) inter- action (Dietz-Uhler & Bishop-Clark, 2001; Järvelä & Häkkinen, 2002). As stated by Lapadat (2002), asynchronous online discus- sion in particular facilitates reflection, conceptual change and the collaborative construction of meaning, making it especially suit- able for the collaborative processing of academic literature. This study – which aims to facilitate students’ collaborative processing of literature – will use a specialised form of online discussion called ‘‘anchored discussion” (Bernheim Brush, Bargeron, Grudin, Bor- ning, & Gupta, 2002), that integrates students’ online discussion with the subject matter that is being discussed. Van der Pol, Admi- raal, and Simons (2006) have demonstrated that anchored discus- sion is better suited for supporting the early stages of collaborative text processing than regular forum discussion, in that it affords a more efficient and meaning-oriented collaboration by relating the discussion more closely with the subject matter. However, anchored discussion might still have some constraints with regard to the coherence of students’ interaction and the learn- ing potential of their peer-directed replies. In this article, we devel- op and investigate an enhanced tool for anchored discussion that aims to increase the quality of students’ replies. 2. Constructive learning conversations and the relevance of replies While the level of interactional or ‘‘cross-turn” coherence of on- line learning conversations seems to be an important factor in determining the effectiveness of students’ online collaboration (Hoadley & Enyedy, 1999; Hsi, 1997), it is also often identified as a problematic element (Herring, 1999; Reyes & Tchounikine, 2003; Van der Meij, De Vries, Boersma, Pieters, & Wegerif, 2005). Students’ difficulties with preserving the interactional coherence are understandable, because students – who do not yet fully mas- ter the subject matter – act as both the creators and the receivers of feedback. On the one hand, students’ limited level of understand- ing sets high demands for the quality of feedback they require, as it might otherwise be difficult for them to correctly interpret and respond to it. Yet, on the other hand, providing clear, relevant and detailed feedback will be difficult for students as it normally requires a thorough understanding of the subject matter. As found by Webb and Mastergeorge (2003) effective helping behaviour in peer-directed groups entails the posing of precise questions and the providing of detailed explanations. In previous research on using online discussion for the collaborative processing of texts Van der Pol (2002) found that (undergraduate) students do experi- ence difficulties with providing each other with specific feedback, especially when they are processing difficult texts in a foreign lan- guage. Students’ responses were often found to be only associa- tively linked to each other, leading rather to a collection of different ideas than to the building and deepening of ideas. As stated by Bellamy (1997), a constructive theoretical learning 0747-5632/$ - see front matter Ó 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.chb.2007.09.005 * Corresponding author. Tel.: +31 30 2531613; fax: +31 30 2534262. E-mail address: j.vanderpol@vu.nl (J. van der Pol). Computers in Human Behavior 26 (2010) 288–295 Contents lists available at ScienceDirect Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh