Proceedings of the ASME 2013 International Design Engineering Technical Conferences IDETC 2013 August 4-7, 2013, Portland, Oregon, USA DETC2013- 13020 A SIMULATION BASED ESTIMATION OF CROWD ABILITY AND ITS INFLUENCE ON CROWDSOURCED EVALUATION OF DESIGN CONCEPTS Alex Burnap * Yi Ren Panos Y. Papalambros Optimal Design Laboratory University of Michigan Ann Arbor, MI Richard Gonzalez Department of Psychology Department of Statistics University of Michigan Ann Arbor, MI Richard Gerth National Automotive Center TARDEC-NAC Warren, MI ABSTRACT Crowdsourced evaluation is a promising method for evalu- ating attributes of design concepts that require human input. One factor in obtaining good evaluations is the ratio of high-ability to low-ability participants within the crowd. In this paper we introduce a Bayesian network model capable of finding partic- ipants with high design evaluation ability, so that their evalua- tions may be weighted more than those of the rest of the crowd. The Bayesian network model also estimates a score of how well each design concept performs with respect to a design attribute without knowledge of the true scores. Monte Carlo simulation studies tested the quality of the estimations on a variety of crowds consisting of participants with different evaluation ability. Re- sults suggest that the Bayesian network model estimates design attribute performance scores much closer to their true values than simply weighting the evaluations from all participants in the crowd equally. This finding holds true even when the group of high ability participants is a small percentage of the entire crowd. Keywords: Crowdsourcing, Design Concept Evaluation, Machine Learning. 1 Introduction Suppose we wish to evaluate a set of military vehicle design concepts with respect to a set of mission performance attributes. For many attributes, detailed engineering simulations are used to obtain accurate evaluations, such as finite-element analysis to * Address all correspondence to this author. Email: aburnap@umich.edu evaluate blast resistance or human mobility modeling to evaluate ergonomics. However, for some attributes, physics-based simu- lation is difficult and evaluation requires human input. To obtain human evaluations on these perceptual design at- tributes [10], one may ask a number of specialists to evaluate the vehicle concepts. The ability to make an evaluation is likely scattered over the “collective intelligence" of a large number of people with diverse backgrounds [5] and viewpoints. Crowdsourced evaluation, or the delegation of an evalua- tion task to a large and unknown group of people, is a promis- ing approach to obtain evaluations on perceptual attributes. This approach draws on lessons from online communities, like Wikipedia, which have shown that accuracy and comprehensive- ness is possible in large crowdsourced settings. An important lesson from such community efforts is the need to implement a consistent method of filtering "signal" from "noise;" namely, obtaining valid contributions from those that are not. This general “signal to noise problem" manifests itself in any large crowd with heterogenous abilities, and in a crowd- sourced evaluation it is desirable to identify high ability partici- pants from the rest of the crowd. Consequently, identifying eval- uations from high-ability participants will make the crowdsourc- ing process more effective and efficient. In this paper, we explore the identification of high-ability participants through simulation of the crowdsourced evaluation process. The goal is to test the identification process prior to conducting experiments and collecting data with an actual hu- man crowd. Clearly, simulation results depend on the modeling 1