IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. 8, NO. 4, AUGUST 2004 405 Statistical Exploratory Analysis of Genetic Algorithms Andrew Czarn, Cara MacNish, Kaipillil Vijayan, Berwin Turlach, and Ritu Gupta Abstract—Genetic algorithms have been extensively used and studied in computer science, yet there is no generally accepted methodology for exploring which parameters significantly affect performance, whether there is any interaction between param- eters, and how performance varies with respect to changes in parameters. This paper presents a rigorous yet practical statistical method- ology for the exploratory study of genetic and other adaptive al- gorithms. This methodology addresses the issues of experimental design, blocking, power calculations, and response curve analysis. It details how statistical analysis may assist the investigator along the exploratory pathway. As a demonstration of our methodology, we describe case studies using four well-known test functions. We find that the effect upon performance of crossover is pre- dominantly linear, while the effect of mutation is predominantly quadratic. Higher order effects are noted but contribute less to overall behavior. In the case of crossover, both positive and nega- tive gradients are found suggesting the use of a maximum crossover rate for some problems and its exclusion for others. For mutation, optimal rates appear higher compared with earlier recommenda- tions in the literature, while supporting more recent work. The sig- nificance of interaction and the best values for crossover and mu- tation are problem specific. Index Terms—Adaptive algorithms, experimental design, ge- netic algorithms (GAs), methodology, statistical analysis. I. INTRODUCTION A DAPTIVE algorithms such as genetic algorithms (GAs) [1] work by iteratively adapting members of a population of potential solutions. The individuals interact either through the adaptation operators themselves, or through competitive selec- tion mechanisms for determining subsequent generations. If the adaptation strategy is successful, the population (or part thereof) will converge on an optimal (or at least “good”) solution. While the mechanics of each individual adaptation are quite straightforward, the way individual changes affect the success of the population as a whole is more difficult to determine. This is also true of the many parameters that are used to fine tune, or improve the success of adaptive algorithms. Examples include population size, mutation and crossover rates, elite group sizes, acceleration constants, step sizes, and so on. Values for these Manuscript received September 8, 2003; revised April 30, 2004. A. Czarn and C. MacNish are with the School of Computer Science and Software Engineering, University of Western Australia, Crawley 6009, Western Australia (e-mail: aczarn@csse.uwa.edu.au; cara@csse.uwa.edu.au). K. Vijayan and B. Turlach are with the School of Mathematics and Statis- tics, University of Western Australia, Crawley 6009, Western Australia (e-mail: vijayan@maths.uwa.edu.au; berwin@maths.uwa.edu.au). R. Gupta is with the Department of Mathematics and Statistics, Curtin University of Technology, Bentley 6102, Western Australia (e-mail: ritu@ maths.curtin.edu.au). Digital Object Identifier 10.1109/TEVC.2004.831262 parameters are most commonly set through a process of trial and error, or based on recommendations from related problems in the literature, rather than through statistically sound analysis of their effects on algorithm performance. In this paper, we propose a rigorous yet practical statistical methodology for assessing the impact of parameter settings. The methodology addresses issues of experimental design, blocking, power calculation, and response curve analysis. We demonstrate the approach with a case study applying GAs to benchmark problems from De Jong’s [2] and Schaffer’s [3] test suites. In Section II, we provide some background to the problem of analyzing GA performance. This is followed in Section III by a discussion of nonstatistical exploratory work in this area. Section IV examines work which has used a statistical con- struct, recognizing the appropriateness of statistical analysis to this problem. However, a number of limitations are found. In Section V, we discuss a range of factors that must be considered in developing a suitable methodology and outline our approach. The results of applying this methodology to the GA in our case study are reported in Section VI. This includes some unexpected outcomes, particularly on the use of crossover. A discussion in Section VII concludes the paper. II. BACKGROUND A GA works by encoding potential solutions to a problem as a series of bits or genes on a bit-string or chromosome. The me- chanics of a GA are straightforward: in its simplest form new so- lutions are generated using crossover, where genes are crossed over between pairs of chromosomes, and mutation, where the binary value of a gene is inverted. In contrast, the way in which a GA population converges on solutions has been more complex to describe [1]. Holland put forward the idea of schemata [4]: similarity templates de- scribing a subset of strings with similarities at certain positions [5]. When the chromosome possesses these schemata its fitness improves. Operators such as crossover and mutation work by al- tering chromosomes to contain more good schemata. Goldberg elaborated by conceptualizing building blocks (highly fit, short- defining-length schemata) and implicit parallelism [5]. How- ever, the increase in sophistication and differences in implemen- tations of GAs, such as quantum-inspired GAs [6] and the use of transposition [7], has made it increasingly difficult to propose newer models of convergence. In addition, previously accepted aspects of GAs are being debated. For example, while it has been traditionally main- tained that crossover is a necessary inclusion, the conjecture of naive evolution (using selection and mutation only) places 1089-778X/04$20.00 © 2004 IEEE