© Juhani Iivari, July 9, 2014 1 Juhani Iivari Department of Information Processing Science University of Oulu Oulu, Finland How to improve the quality of peer reviews? – Three suggestions for system-level changes Abstract Peer reviewing is critical in the process of legitimizing new scientific knowledge. Yet, there are concerns about its quality, especially if one considers developmental review- ing as an ideal. The present essay suggests three ways to improve review quality: pro- vide reviewers with systematic feedback about their performance, reward active and good reviewers, and make reviewers more accountable by revealing their identities to the authors on certain conditions. Introduction Scholarly or scientific peer review is the evaluation of research findings for compe- tence, significance and originality, by qualified experts who do research in the same field (Brown 2004, Benos et al. 2007). It is critical in the process of legitimizing new scientific knowledge and assuring its quality. A piece of research that has not passed scholarly review and has not been published cannot be regarded as scientific, since its findings have not been accepted by the scientific community in question and are not trustworthy in that sense. Just as democracy peer reviewing it is not perfect, but nobody has found a better al- ternative to it. Benos et al. (2007) summarize its weaknesses such as various biases (status and gender biases, and biases because of ideological differences, unconven- tional ideas and conflicts of interest), its inability to identify major flaws and scien- tific misconduct, and delays in the publication process. Yet, scholarly reviewing pro- vides an opportunity to authors to respond to the points of criticism raised by peers before publishing and consequently to improve the articles. That alone is a sufficient reason to preserve peer reviewing (Benos et al. 2007). In the field Information Systems (IS) not only the quality of peer reviews is a concern (Gray et al. 2006), but there are also opinions that the quality of reviews has recently been deteriorating, as evidenced with the discussion on the AISWorld forum during Fall 2013. One reason is the increasing number of journal submissions, leading to a constant shortage of (good) reviewers. If the quality of reviews is getting lower, there is a higher risk that journal editors make Type I errors, in which papers of low quality are accepted, or still more alarmingly Type II errors, in which papers with great po- tential are rejected (Straub 2008).