Empirical Mechanism Design: Methods, with Application to a Supply-Chain Scenario Yevgeniy Vorobeychik, Christopher Kiekintveld, and Michael P. Wellman University of Michigan Computer Science & Engineering Ann Arbor, MI 48109-2121 USA { yvorobey, ckiekint, wellman }@umich.edu ABSTRACT Our proposed methods employ learning and search techniques to estimate outcome features of interest as a function of mechanism parameter settings. We illustrate our approach with a design task from a supply-chain trading competition. Designers adopted sev- eral rule changes in order to deter particular procurement behavior, but the measures proved insufficient. Our empirical mechanism analysis models the relation between a key design parameter and outcomes, confirming the observed behavior and indicating that no reasonable parameter settings would have been likely to achieve the desired effect. More generally, we show that under certain condi- tions, the estimator of optimal mechanism parameter setting based on empirical data is consistent. Categories and Subject Descriptors I.6 [Computing Methodologies]: Simulation and Modeling; J.4 [Computer Applications]: Social and Behavioral Sciences—Eco- nomics General Terms Algorithms, Economics, Design Keywords Empirical Mechanism Design, Game Theory 1. MOTIVATION We illustrate our problem with an anecdote from a supply chain research exercise: the 2003 and 2004 Trading Agent Competition (TAC) Supply Chain Management (SCM) game. TAC/SCM [1] defines a scenario where agents compete to maximize their profits as manufacturers in a supply chain. The agents procure components from the various suppliers and assemble finished goods for sale to customers, repeatedly over a simulated year. 1 1 Information about TAC and the SCM game, including specifi- cations, rules, and competition results, can be found at http: //www.sics.se/tac. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. EC’06, June 11–15, 2006, Ann Arbor, Michigan, USA. Copyright 2006 ACM 1-59593-236-4/06/0006 ...$5.00. As it happened, the specified negotiation behavior of suppliers provided a great incentive for agents to procure large quantities of components on day 0: the very beginning of the simulation. During the early rounds of the 2003 SCM competition, several agent devel- opers discovered this, and the apparent success led to most agents performing the majority of their purchasing on day 0. Although jockeying for day-0 procurement turned out to be an interesting strategic issue in itself [19], the phenomenon detracted from other interesting problems, such as adapting production levels to varying demand (since component costs were already sunk), and dynamic management of production, sales, and inventory. Several partici- pants noted that the predominance of day-0 procurement overshad- owed other key research issues, such as factory scheduling [2] and optimizing bids for customer orders [13]. After the 2003 tourna- ment, there was a general consensus in the TAC community that the rules should be changed to deter large day-0 procurement. The task facing game organizers can be viewed as a problem in mechanism design. The designers have certain game features un- der their control, and a set of objectives regarding game outcomes. Unlike most academic treatments of mechanism design, the objec- tive is a behavioral feature (moderate day-0 procurement) rather than an allocation feature like economic efficiency, and the allowed mechanisms are restricted to those judged to require only an in- cremental modification of the current game. Replacing the supply- chain negotiation procedures with a one-shot direct mechanism, for example, was not an option. We believe that such operational re- strictions and idiosyncratic objectives are actually quite typical of practical mechanism design settings, where they are perhaps more commonly characterized as incentive engineering problems. In response to the problem, the TAC/SCM designers adopted several rule changes intended to penalize large day-0 orders. These included modifications to supplier pricing policies and introduction of storage costs assessed on inventories of components and finished goods. Despite the changes, day-0 procurement was very high in the early rounds of the 2004 competition. In a drastic measure, the GameMaster imposed a fivefold increase of storage costs midway through the tournament. Even this did not stem the tide, and day-0 procurement in the final rounds actually increased (by some mea- sures) from 2003 [9]. The apparent difficulty in identifying rule modifications that ef- fect moderation in day-0 procurement is quite striking. Although the designs were widely discussed, predictions for the effects of various proposals were supported primarily by intuitive arguments or at best by back-of-the-envelope calculations. Much of the dif- ficulty, of course, is anticipating the agents’ (and their develop- ers’) responses without essentially running a gaming exercise for this purpose. The episode caused us to consider whether new ap-