4 th International Conference on Production Research - ICPR Americas’ 2008 AN EXAMPLE ON THE UNRELIABILITY OF MACBETH APPLICATIONS Valério A. P. Salomon (UNESP) Abstract AHP and MACBETH are two of the MCDM methods most often applied to solve Production Management problems in Brazil. However, a MACBETH application can generate unreliable results. An example of a MACBETH application resulting in a wrong ranking of the alternatives, in terms of money, is presented. The AHP application, on the other hand, gives the same ranking as one obtains with cash-flow analysis. Reasons for the unreliability of the MACBETH application are also discussed. Keywords: AHP, MACBETH, MCDM. 1 INTRODUCTION Multiple-Criteria Decision-Making (MCDM) is the study of the inclusion of conflicting criteria in decision-making, as defined by the International Society on MCDM [1]. According to a recent survey [2], the Analytic Hierarchy Process (AHP) and the Measuring Attractiveness by a Categorical Based Evaluation Technique (MACBETH) are two of the MCDM methods most applied to solve Production Management problems in Brazil. Since its first publication [3], the AHP has been widely applied and studied. From 1988 to 2007, there have been nine international conferences on the AHP, where academics and professionals have discussed the theory and the practical aspects of this method. Four of these conferences took place in North America and the other five in countries like China, Chile, Indonesia, Japan and Switzerland. One poor reason given for the MACBETH’s appeal is that it is a more recent method. However, MACBETH has so many elements in common with the AHP, that one can say it is not much more than another version of AHP. Both methods can incorporate tangible and intangible criteria to rank the alternatives. Both AHP and MACBETH measure intangible attributes of the alternatives with judgments inserted in pairwise comparison matrices. The quality of the input data in the pairwise comparisons in both methods is established by checking the consistency among the judgments [4]. Finally, the relative values for the alternatives, for each criterion, must be weighted by the criterion’s priority. The priority of the criteria is obtained by making pairwise comparisons, in both methods. Differences between these methods include the scale used for judgments and the process of synthesize the judgments into the priorities. This note shows that a MACBETH application can provide unreliable results, even for simple decision- making problems, with small numbers of alternatives and criteria. The research method adopted is the mathematical modeling, which is a quantitative approach of research [5]. A personal decision problem is presented in Section 2. In this example, quantitative values for the alternatives are known. Both AHP and MACBETH were applied (as presented in Sections 3 and 4) to determine the relative priorities of the alternatives and the results were compared to the actual data (Section 5). Judgments were used only to obtain the priorities of the criteria and actual data was used to determine the performance of the alternatives on each criterion. The alternative ranking from the AHP application is the same as that obtained with a flow-cash analysis. However, the MACBETH application results in a higher attractiveness for an alternative which is not the most attractive one. Some features intrinsic to MACBETH method are discussed. These features may cause the unreliable results: the use of an unjustified and unclear scale of numbers to represent judgments; the inclusion of a virtual element in the judgments on the priorities of the criteria. In the last section, another unfavorable feature of the MACBETH applications is pointed out: the use of black box software that checks the quality from the input data judgments and calculates the priorities. 2 A PERSONAL DECISION PROBLEM Table 1 presents data for three different new cars (A1, A2 and A3), with different cost considerations and similar features otherwise: body type, comfort & convenience, interior specifications and