European Journal of Information Systems (2000) 9, 36–46  2000 Operational Research Society Ltd. All rights reserved 0960-085X/00 $15.00 www.stockton-press.co.uk/ejis An analysis of expert systems for business decision making at different levels and in different roles JS Edwards 1 , Y Duan 2 and PC Robins 1 1 Aston Business School, Aston University, Birmingham, B4 7ET, UK; 2 Luton Business School, University of Luton, LU1 3JU, UK This paper begins by analysing decision making activities and information requirements at three organiza- tional levels and the characteristics of expert systems (ESs) intended for the two different roles of support- ing and replacing a decision maker. It goes on to review the evidence from many years of commercial use of ESs at different levels and in different roles, and to analyse the evidence obtained from a pilot experiment involving developing ESs to fulfil two different roles in the same domain. The research finds that ESs in a replacement role prove to be effective for operational and tactical decisions, but have limi- tations at the strategic level. ESs in a support role, as advisory systems, can help to make better decisions, but their effectiveness can only be fulfilled through their users. In the experiments, an expert advisory system did not save a user’s time, contrary to the expectations of many of its users, but an ES in a replacement role did improve the efficiency of decision making. In addition, the knowledge bases of the ESs in the different roles need to be different. Finally, the practical implications of the experience gained from developing and testing two types of ESs are discussed. Introduction Expert systems (ESs) were among the earliest branches of artificial intelligence (AI) to be commercialised and still constitute a growing segment of computer-based information systems (Moody et al, 1998). Many organi- zations have leveraged the technology to increase pro- ductivity and profits through better business decisions (Durkin, 1996), even if the impact of ESs has not achieved the levels predicted in the 1980s. Although there have been some failures (O’Keefe & Rebne, 1993; Wong, 1996), recent research (Yoon et al, 1995; Kunnathur et al, 1996) shows that there are many companies who remain enthusiastic proponents of the technology and continue to develop important ES applications. As Gill (1995) has found, successes in several ESs have amply demonstrated the technology’s capability both to gener- ate huge financial returns and to contribute to the stra- tegic goals of the firm. However, Tsai et al (1994) argue that the benefits of ESs are often oversold to the business community by supporters and vendors. The performance of implemented ESs frequently does not measure up to the unrealistic expectations of the users and managers; thus, they view claims for ES benefits with scepticism and disappointment. Clearly there remains a divergence of attitude towards the use of expert systems. In these circumstances, the need to assess the value of ESs and assure implementation success becomes even more important (Yoon et al, 1995). At the start of the decade, the ES field was characterised as having a wide range of theoretical viewpoints, but an absence of gener- ally accepted theory in many important areas (Bramer, 1990). This is still particularly true when discussing the use of ESs in the business decision making area. This paper addresses two key issues regarding the application of ESs, which may be stated as follows: (1) At which organizational level? A commonly adopted framework suggests that the decision making activi- ties in an organization can be divided into three lev- els: strategic, tactical and operational. The problems encountered generally vary from unstructured to structured between strategic and operational levels. A source of confusion is whether the level is that of the decision or that of the decision maker. (2) In what roles? ESs can be designed for two different purposes, to support or replace a decision maker. An ES in a support role is also known as an expert advisory system; the two terms will be used inter- changeably here. Two ESs with different roles were developed for this research, using a business game as a simulated organiza- tion. One (EXGAME) was designed to replace a human in decision making, while the other (ADGAME) was an advisory system to support human decision makers. The two ESs were built to mimic decision making in a busi- ness game environment which simulates a manufacturing company. The evaluation of EXGAME provides some insight into the ES’s ability to work successfully at dif- ferent organizational levels. The tests of ADGAME shed some light on the effectiveness of an expert advisory