Measuring the Crowd – A Preliminary Taxonomy of Crowdsourcing Metrics Eoin Cullina NUI Galway, J.E Cairnes School of Business & Economics Newcastle, Galway, Ireland. +353 61 213028 e.cullina2@nuigalway.ie Kieran Conboy NUI Galway, J.E Cairnes School of Business & Economics Newcastle, Galway, Ireland. +353 91 492988 kieran.conboy@nuigalway.ie Lorraine Morgan NUI Galway, J.E Cairnes School of Business & Economics Newcastle, Galway, Ireland. +353 61 213028 lorraine.morgan@nuigalway.ie ABSTRACT Crowdsourcing initiatives benefit from tapping into diversity. A vast plethora of disparate individuals, organizations, frameworks and skillsets can all play a role in sourcing solutions to a challenge. Nevertheless, while crowdsourcing has become a pervasive phenomenon, there is a paucity of research that addresses how the crowdsourcing process is measured. Whereas research has advanced various taxonomies of crowdsourcing none to date have specifically addressed the issue of measuring either specific stages of the crowdsourcing process or the process as a whole. As a first step towards achieving this goal, this research-in- progress paper examines crowdsourcing at the operational level with a view towards (i) identifying the parts of the process (ii) identifying what can be measured and (iii) categorising operational metrics to facilitate deployment in practice. The taxonomy advanced is overarching in nature and can be deployed across disciplines. Furthermore, the preliminary taxonomy presented will offer practitioners a comprehensive list of metrics that will enable them to facilitate comparison across various crowdsourcing initiatives. Categories and Subject Descriptors C.4 [Computer Systems Organization]: Performance of Systems - Measurement Techniques; H.4.3 [Information Systems Applications]: Communications Applications. General Terms Measurement, Performance, Design. Keywords Crowdsourcing, taxonomy generation, metrics. 1. INTRODUCTION With the move towards seeking solutions outside of the traditional four walls of organizations, the information systems (IS) community is actively seeking to identify antecedents to open innovation and its subsets including crowdsourcing. Crowdsourcing as a contemporary means of problem solving is drawing mass attention from the Internet community. In 2014 big brands such as Procter and Gamble, Unilever, and Pepsi Co increased their investment in crowdsourcing in ranges from 50% to 325% [73]. In the case of Unilever, crowdsourcing presents itself as a means to achieving a target of doubling the business by 2020 in a sustainable manner. The Unilever Foundry IDEAS platform is a key part of the businesses innovation and idea generation strategy for the organization [83]. While an initial definition for the term crowdsourcing was advanced in 2006 [44], historical non-socio technical examples exist of calling upon the wisdom of crowds including the Longitude Prize of 1714 [77]. Jeff Howe defines crowdsourcing as the “act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call. [44]. Recent research has extended the definition to also include situations outside of pure “business” outsourcing such as charity, philanthropy and government initiatives. Since 2006 a plethora of other definitions have been advanced from research [19; 27; 53]. Moreover, specific definitions for crowdsourcing have been put forward in certain industry specific contexts such as the software development industry [79]. Within the context of different sectors, such as innovation and scientific research, crowdsourcing has been utilised as an effective innovation and problem-solving tool [30; 60; 61; 71; 82]. Diversity in itself can have a major effect on organizations and their revenue streams [47]. However, with this great diversity come the challenges of establishing metrics and benchmarking. According to Kerzner you cannot correct or improve something that cannot be effectively identified and measured [48]. The preliminary taxonomy presented herein proposes to address one part of this challenge (i.e. the identification of metrics) at an operational level. Taxonomy generation represents a means by which relationships amongst concepts can be understood through structure [35]. With the advent of crowdsourcing organizations can now use information systems technologies to source ideas and solutions from the crowd. Crowdsourcing is used for many different types of challenges. Numerous examples exist of successful crowdsourcing initiatives including iCancer UK, Dell Idea Storm, IBM Jams and Lego Ideas to name but a few. Furthermore, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. OpenSym '15, August 19 - 21, 2015, San Francisco, CA, USA © 2015 ACM. ISBN 978-1-4503-3666-6/15/08…$15.00 DOI: http://dx.doi.org/10.1145/2788993.2789841