Proceedings of the 2011 AAEE Conference, Fremantle, Western Australia, Copyright Willey, et. al. 2011 Gatekeeping or filtering?: Investigating the connection between peer review and research quality Keith Willey 1 , Lesley Jolly 2 , Gregory Tibbits 2 , Anne Gardner 1 1 University of Technology, Sydney; 2 University of Queensland keith.willey@uts.edu.au; ljolly@bigpond.net.au; gregory.tibbets@uqconnect.edu.au; Anne.Gardner@uts.edu.au Abstract: Arguably, the most important opportunity to acquire the standards and norms of the discipline and develop researchers’ judgement is the peer review process – but this depends on the quality of the reviews. ‘Good’ feedback - which we take to mean feedback that has the capacity to improve subsequent practice - has been identified as being timely, specific and relevant. Yet often reviews lack these basic qualities. In this paper we report an investigation of the peer review process at the 2010 Australasian Association of Engineering Education (AAEE) conference. Authors at the conference were given the chance to rate their reviews and we subsequently analysed both the nature of the reviews and authors’ responses. Findings suggest that the opportunity to use the peer review process to induct people into the field and improve practice is being missed. As in other disciplines there is also ample evidence that the review process does little or nothing to ensure the standard and relevance of conference presentations. It is therefore legitimate to ask whether there may not be better processes to attain these ends and we conclude with some discussion of how the review process may be made more helpful for everyone involved. Introduction If we are to talk of inclusive engineering we must consider our own practices in venues such as this one. What diversity of ideas is able to emerge through the normative practices of setting conference themes, reviewing of papers and the potentially confrontational conference presentation itself? How are neophyte engineering education researchers able to be developed within formal structures that reward the familiar and the well-established? While conference presentation is sometimes regarded as a preliminary step to mature publishing in journals (the gold standard for academic practice), are conferences such as ours really venues that can foster scholars and their ideas? We wish here to raise questions about one aspect of scholarly practice which we usually take for granted, the peer review system, which might actually work to discriminate against innovation and impede the development of the field. While we acknowledge that there are external pressures which are likely to maintain the need for peer review, we would like to open debate on how we might diversify our practices in productive ways. It has been well demonstrated that there are many ways in which peer review fails (Goodstein 2000) including through restricting the dissemination of new ideas, excessive in-group gatekeeping and inconsistency (Fitzpatrick 2010, Lipworth and Kerridge 2011). Origgi (2010) has argued that an important function of the review process is a “conversation in slow motion” amongst scholars in order to improve, disseminate and develop their ideas and we argue that such a conversation is much need in the emerging and as yet immature field of engineering education research. Academics engaging in this field frequently have to argue for the rigour and significance of their work(JEE 2011). While some of the necessity for this argument is a result of institutional resistance to the pursuit of educational research by engineers, some of it arises from a perception (sometimes accurate) that conference presentations and published work in this still emerging field is not of the highest quality. Given that the majority of people who undertake this research were trained in quite different paradigms than those that underpin educational research, bridging the epistemological divide between technical training and the more social science of education continues to be problematic for some(Borrego 2007). 241