A Performance Assessment Support System for Advanced Software Systems Fred Kwakkel and Martin L. Kersten Database research group, CWI, kruislaan 413 NL-1098 SJ Amsterdam, The Netherlands {kwakkel,mk}@cwi.nl Abstract Performance measurement involves running a set of representative workloads, such as benchmarks, and quality assurance, which involves extensive testing. Most present-day benchmarks provide the user and system engineer with a few performance figures only (e.g. TPS: throughput in number of transaction per second). To compare two or more different software systems or evaluate a newly designed system, such figures are often not sufficient. A performance relationship between experimental factors and response variables is much harder and cumbersome to obtain, because more measurements have to be performed. Moreover, expertise in the area of experimental design is required to derive the performance relationships at minimal cost (time). The novel performance assessment sup- port system (PASS), called “SOFTWARE TESTPILOT”, described in this paper greatly simplifies performance-as- sessment and -analysis of advanced database management systems by enabling a compact specification of the workload search space, a flexible mechanism to interact with a system under study, and a fast algorithm to expose the performance bottlenecks or software instabilities. 1 Introduction Performance evaluation is an art. To perform an art, proper tools are indispensable. Could Rembrandt have created his “Nachtwacht” without brush and easel? Many computer-system users nowadays have to perform the art of performance evaluation and thus require tools to support assessment and analysis of measurements taken. In this paper we discuss a system that greatly simplifies performance assessment of advanced database management systems. Database technology for single processor architectures has matured from file processing by a large number of individual programs to sound data models and data manipulation schemes ap- plicable in many environments. The DBMSs are delivered as portable, single processor imple- mentations with many techniques to obtain good performance. However, new (prototype) sys- tems often exhibit low quality and low performance due to lack of extensive field tests in the application areas they intend to support. Inclusion of new techniques into existing architectures also hamper from unpredictable side-effects on system performance and stability. Yet, from an economic point of view it is mandatory to predict their performance and to assess their stability long before the hardware and software are installed. For this reason the Pythagoras project was called into existence. Its objectives are to develop the means to predict, assess and tune the performance of Advanced Information Servers (AIS). To do this it aims to develop a tool-kit and methodology for performance quality evaluation and prediction of parallel database servers intended to sup- port an AIS. In addition, it is targeted at both buyers and designers. The Pythagoras project has three principal facets: (1) The project delivers a performance suite aimed at quantifying the effectiveness of AIS in geographic, case, and business applications. (2) These exemplars are complemented by in-depth studies to improve the knowledge of the key software technology required and to assess the impact of hardware technology on the software choices, such as its architecture, logical query optimization and data placement.