ISDSS’99 1 A Method For Multiple Criteria Evaluation Of DSS In A Multiple- Constituency Environment Sean Maynard Department of Information Systems, University of Melbourne, Australia Email: s.maynard@dis.unimelb.edu.au David Arnott, Frada Burstein School of Information Management and Systems Monash University, Melbourne, Australia Email david.arnott@sims.monash.edu.au frada.burstein@sims.monash.edu.au Abstract Many studies discuss approaches for the evaluation of DSS. Few, however, provide a method to allow the context of the evaluation to be explicitly considered. This paper proposes a DSS evaluation method based on multiple criteria and incorporating a multiple constituency perspective. The approach proposed can be used for any group - constituencies, involved in DSS project at any stage. With multiple-constituency DSS evaluation many criteria may, at any time, be valuable to a constituency group, whilst others may be unimportant, or inconsequential. This paper introduces the concept of multiple-constituencies and discusses some methods for multiple criteria decision making and evaluation. A prototype tool implementing the approach is also described. 1. Introduction The evaluation of a decision support system (DSS) is usually based on the opinions of a single reference group, or constituency, rather than on all relevant groups involved with the DSS project. Consequently, many methods of evaluation focus on a single group, usually the users or decision-makers, depending on the terminology used. Including each relevant constituency in the evaluation process may produce a more context-sensitive evaluation that could be used to improve the DSS from each group’s perspective (Maynard, Arnott and Burstein, 1995). An example of this type of approach is the multiple-constituency approach (Connolly, Conlon and Deutsch 1980). The evaluation process must measure the success of the system from each constituency’s perspective to be considered useful. To accomplish this, a number of evaluation criteria will be required, with many being valid criteria for one, or many constituency groups. Any method used in such an evaluation environment must adequately utilise those criteria that are valid for the particular constituency evaluating the DSS. The context of the evaluation may also strongly influence the evaluation results. Thus, this is important to have a flexible tool for DSS evaluation that takes into consideration all these issues. Maynard and Arnott (1994) adopt a multiple-constituency approach for DSS evaluation. This approach allows evaluation to be performed from multiple, and some times very different, perspectives of the DSS project. The method was extended with multiple-criteria evaluation (Maynard; Arnott and Burstein, 1995). A comprehensive set of criteria useful for measuring a success of DSS project was identified. The criteria were presented as a hierarchy classified by the different perspectives they measure DSS from; for example effectiveness, efficiency, satisfaction and use, are the classes of the upper level of this hierarchy. The approach allows various groups of people concerned with the DSS to