C. Hofmeister et al. (Eds.): QoSA 2006, LNCS 4214, pp. 9 – 26, 2006. © Springer-Verlag Berlin Heidelberg 2006 MEMS: A Method for Evaluating Middleware Architectures Yan Liu 1 , Ian Gorton 1 , Len Bass 2 , Cuong Hoang 3 , and Suhail Abanmi 1 1 National ICT Australia (NICTA), Australia 1 , & School of Computer Science and Engineering, University of New South Wales, Australia {jenny.liu, ian.gorton, suhail.abanmi}@nicta.com.au 2 Software Engineering Institute, USA ljb@sei.cmu.edu 3 Engineering Faculty, University of Technology Sydney vietcuong.hoang@student.uts.edu.au Abstract. Middleware architectures play a crucial role in determining the overall quality of many distributed applications. Systematic evaluation meth- ods for middleware architectures are therefore important to thoroughly assess the impact of design decisions on quality goals. This paper presents MEMS, a scenario-based evaluation approach. MEMS provides a principled way of evaluating middleware architectures by leveraging generic qualitative and quantitative evaluation techniques such as prototyping, testing, rating, and analysis. It measures middleware architectures by rating multiple quality at- tributes, and the outputs aid the determination of the suitability of alternative middleware architectures to meet an application’s quality goals. MEMS also benefits middleware development by uncovering potential problems at early stage, making it cheaper and quicker to fix design problems. The paper de- scribes a case study to evaluate the security architecture of grid middleware architectures for managing secure conversations and access control. The re- sults demonstrate the practical utility of MEMS for evaluating middleware architectures for multiple quality attributes. 1 Introduction Middleware refers to a broad class of software infrastructure technologies that use high-level abstractions to simplify construction of distributed systems. This infra- structure provides a distributed environment for deploying application-level compo- nents. These application components rely on the middleware to manage their life cycles and execution, and to provide off-the-shelf services such as transactions and security. Consequently, the application component behavior and the middleware architec- ture are tightly coupled, and the middleware plays a critical role in achieving the 1 National ICT Australia is funded through the Australian Government’s Backing Australia’s Ability initiative, in part through the Australian Research Council.