Avoiding Local Optima with User Demonstrations and Low-level Control Shane Celis Dept. of Computer Science University of Vermont Email: shane.celis@uvm.edu Gregory S. Hornby UARC, U.C. Santa Cruz NASA Ames Research Center Email: gregory.s.hornby@nasa.gov Josh Bongard Dept. of Computer Science University of Vermont Email: josh.bongard@uvm.edu Abstract—Interactive Evolutionary Algorithms (IEAs) use human input to help drive a search process. Traditionally, IEAs allow the user to exhibit preferences among some set of individuals. Here we present a system in which the user directly demonstrates what he or she prefers. Demonstration has an advantage over preferences because the user can provide the system with a solution that would never have been presented to a user who can only provide preferences. However, demonstration exacerbates the user fatigue problem because it is more taxing than exhibiting preferences. The system compensates for this by retaining and reusing the user demonstration, similar in spirit to user modeling. The system is exercised on a robot locomotion and obstacle avoidance task that has an obvious local optimum. The user demonstration is provided through low-level control. The system is compared against a high-level fitness function that is susceptible to becoming trapped by a local optimum and a mid-level fitness function designed to remove the local optimum. We show that our proposed system outperforms most variants of these completely automatic methods, providing further evidence that Evolutionary Robotics (ER) can benefit by combining the intuitions of inexpert human users with the search capabilities of computers. I. I NTRODUCTION Citizen science, wherein inexpert users contribute to scien- tific research, has demonstrated that non-expert users can assist researchers on tasks that appear to require a great deal of train- ing, such as Foldit, the crowd-sourced protein-folding game [12]. Interactive evolutionary algorithms seem like a natural means of integrating user input and computational algorithms for certain problem domains [7, 10, 19, 20] because input from a user warps and, if done correctly, can smooth the fitness landscape for the evolutionary algorithm. However, it is not yet clear how, when, and how much the user should influence the evolutionary algorithm. How should one incorporate user input? Should an algorithm present complete solutions that the user ranks according to preference? Should an algorithm present solutions that the user may directly manipulate? This paper explores the latter idea whereby the user may take a solution and directly manipulate it to suit his or her preference. Some similar problems crop up in approaches that require the user to indicate preferences between solutions and approaches that require the user to provide a demonstration, so it is worth surveying work on user preferences. Traditional IEAs rely on the user to effectively become the fitness function of an EA. The user is presented with a set of individuals. The user chooses one or more individuals according to their preference. The EA then selects the favored individuals to generate the next generation. This can be es- pecially effective in domains that rely on human perception [9, 10, 18, 19]. However, driving search by user input does have a number of issues. One limitation is that the fitness function is costly, degrades over time, and is imprecise. The degradation over time is known as user fatigue [20]. Typical non-interactive evo- lutionary algorithms require thousands of fitness function eval- uations to find useful solutions. This is particularly problematic if the user is required to demonstrate preferred solutions rather than simply judge the relative merits of automatically- generated solutions. One thread of research seeking to alleviate the user fa- tigue problem is user modeling, originally presented in the evolutionary algorithms domain by Schmidt and Lipson [17], inspired by the Estimation-Exploration Algorithm [4, 6]. The basic scheme is to collect the user responses and automatically build a user model that may be used with impunity. Subsequent work by Hornby and Bongard has shown that user modeling can produce solutions of better quality to those produced by a traditional IEA and with far fewer user interactions [13, 14]. Although the user input differs in this paper from the work on user preferences in that it allows the user to demonstrate what they prefer, it needs to overcome the user fatigue problem to even be feasible. It must have a means of retaining and reusing the user demonstrations similar in spirit to the way user modeling retains, reuses, and learns the user preferences. Demonstration is meant broadly, i.e., if the domain involves creating images as in Picbreeder [18], the user might demon- strate their preference by painting a portion of an image red, signaling to the search algorithm that it should favor images with red coloring. If the domain involves creating 3D shapes as in Endless Forms [9], the user might demonstrate by extruding or molding the form. If the problem is a robotics task, the user might demonstrate by moving the joints of the robot directly, an idea pursued in this paper. Precursors to this idea are known as behavioral cloning [2], programming by demonstration [8], also known as learning by imitation. This paper uses a robotics task to evaluate the proposed system: a robot must locomote around a barrier to reach a target (see Fig. 1). This task was chosen in part to compare with concurrent work on user preferences [5]. In this paper we consider three levels of control: high-level, mid-level, and low-level. Consider the quadruped robot attempting the task as shown in Fig. 2. A high-level controller might command