An Automated Component-Based Performance Experiment Environment Van Bui Mathematics and Computer Science Division 9700 S. Cass Ave Argonne, IL 60439 vbui@mcs.anl.gov Boyana Norris Mathematics and Computer Science Division 9700 S. Cass Ave Argonne, IL 60439 norris@mcs.anl.gov Lois Curfman McInnes Mathematics and Computer Science Division 9700 S. Cass Ave Argonne, IL 60439 curfman@mcs.anl.gov ABSTRACT Performance experiments can involve multiple iterative runs where parameters such as the execution platform, measure- ment tools, methods of measurement, application parame- ters, and analysis techniques can vary. In order to man- age the layers of complexity involved in setting up and run- ning experiments, data management, and post analysis, a degree of abstraction and automation is necessary for each phase. We present an integrated component-based environ- ment that leverages existing performance measurement and analysis technologies and automates the process of running multiple performance experiments and parameter selection of parallel scientific applications. Our component toolkit will enable application scientists to easily modify the ex- perimental parameters over multiple execution runs and to selectively retrieve the data for analysis and generation of performance models. 1. INTRODUCTION Performance experiments typically involve several phases and often require many runs of an application with different configurations and inputs. In order to simplify the process, a more uniform interface is needed across different tools, technologies, and configuration settings. We outline here a common approach to performance experimentation, along with the tools we developed to simplify and automate this process. The first step in this process is determining what to measure. Next, the performance experiment must be set up and then executed using different tools whose selection depends on the underlying platform’s capabilities as well as the application. Once the data is collected, it can be stored to a database for later retrieval. Finally, analysis can be performed on the data to extract useful information. Figure 1 depicts the common scenario that we describe in running performance experiments. We employ a component-based approach 1 to provide uni- form infrastructure for managing performance experiments. Common interfaces enable us to encapsulate the handling of many disparate aspects of different hardware platforms and existing performance tools with similar functionality. The ultimate goal is to design a portable and extensible tool infrastructure that can make performance experiments with both component and non component applications much more streamlined and easier to perform routinely and auto- matically. Database PerfDMF, MetaDB, etc… Analysis Visualization PerfExplorer, CUBE, PyLab, etc… Application Binary 1 2 3 BlueGene, Ranger, etc… Application Metadata Performance Measurement SCALASCA, TAU, etc… PETSc, FLASH, etc… Figure 1: Performance experiment workflow. 1.1 The Common Component Architecture High-performance computing (HPC) is encountering increas- ing software development challenges caused by the grow- ing complexity and size of libraries and applications and by the size and diversity of high-performance hardware plat- forms. The productivity costs of implementing, porting, and tuning the performance of large-scale scientific codes are rapidly rising. Component technology (e.g., [27]), which is now widely used in mainstream computing but is relatively new in HPC, builds on the ideas of object-oriented design to improve software interoperability, robustness, maintainabil- ity, and performance. Furthermore, component-based soft- ware engineering (CBSE) approaches present opportunities 1 Project website: http://trac.mcs.anl.gov/projects/ cca/wiki/performance.