Exploiting Open-Endedness to Solve Problems Through the Search for Novelty Joel Lehman and Kenneth O. Stanley School of Electrical Engineering and Computer Science University of Central Florida, Orlando, FL, 32816 {jlehman,kstanley}@eecs.ucf.edu Abstract This paper establishes a link between the challenge of solv- ing highly ambitious problems in machine learning and the goal of reproducing the dynamics of open-ended evolution in artificial life. A major problem with the objective function in machine learning is that through deception it may actu- ally prevent the objective from being reached. In a similar way, selection in evolution may sometimes act to discourage increasing complexity. This paper proposes a single idea that both overcomes the obstacle of deception and suggests a sim- ple new approach to open-ended evolution: Instead of either explicitly seeking an objective or modeling a domain to cap- ture the open-endedness of natural evolution, the idea is to simply search for novelty. Even in an objective-based prob- lem, such novelty search ignores the objective and searches for behavioral novelty. Yet because many points in the search space collapse to the same point in behavior space, it turns out that the search for novelty is computationally feasible. Furthermore, because there are only so many simple behav- iors, the search for novelty leads to increasing complexity. In fact, on the way up the ladder of complexity, the search is likely to encounter at least one solution. In this way, by de- coupling the idea of open-ended search from only artificial life worlds, the raw search for novelty can be applied to real world problems. Counterintuitively, in the deceptive maze navigation task in this paper, novelty search significantly out- performs objective-based search, suggesting a surprising new approach to machine learning. Introduction The problem of overcoming deception and local optima to find an objective in machine learning is not often linked to the goal of creating a truly open-ended dynamic in artificial life. Yet this paper argues that the same key idea addresses both challenges. The concept of the objective function, which rewards get- ting closer to the goal, is ubiquitous in machine learning [22]. However, objective functions come with the pathol- ogy of local optima; landscapes from objective (e.g. fitness) functions are often deceptive [9, 21]. As a rule of thumb, the more ambitious the goal, the more likely it is that search can be deceived by local optima. The problem is that the objec- tive function does not necessarily reward the stepping stones in the search space that ultimately lead to the objective. For example, it is difficult to train a simulated biped with- out first suspending it from a string because it simply falls down on every attempt, obfuscating to the objective function any improvements in leg oscillation [30]. For these reasons, ambitious objectives are often carefully sculpted through a curriculum of graded tasks, each chosen delicately to build upon the prior [8, 10, 30]. Yet such in- cremental training is difficult and ad hoc, requiring intimate domain knowledge and careful oversight. In contrast to the focus on objective optimization in ma- chine learning, researchers in artificial life often study sys- tems without explicit objectives, such as in open-ended evo- lution. An ambitious goal of this research is to reproduce the unbounded innovation of natural evolution. A typical approach is to create a complex artificial world in which there is no final objective other than survival and replication [4, 32]. Such models assume that biologically-inspired evo- lution supports creating an open-ended dynamic that leads to unbounded increasing complexity [3, 4, 16]. However, a growing yet controversial view in biology is that the drive towards complexity is a passive force, i.e. not driven primarily by selection [15, 18, 19]. In fact, in this view, the path towards complexity in natural evolution can sometimes be inhibited by selection pressure. Thus although open-endedness is often framed as an adaptive competition in artificial life worlds [3, 16], this paper decouples the idea of open-endedness from the domain by capitalizing on a simpler perspective: An open-ended evolutionary system is simply one that continually produces novel forms [25]. This perspective leads to a key idea that addresses the problems in both artificial life and machine learning: In- stead of modeling natural evolution with the hope that novel individuals will be continually discovered, it is possible to search directly for novelty. Thus this paper introduces the novelty search algorithm, which searches with no objective other than continually finding novel behaviors in the search space. By defining novelty in this domain-independent way, novelty search can be applied to real world problems as di- rectly as artificial life worlds. In fact, because there are only so many ways to behave, some of which must be more com- Artificial Life XI 2008 329