A filtering method for algorithm configuration based on consistency techniques Ignacio Araya ⇑ , María-Cristina Riff Depto. Informática, Universidad Técnica Federico Santa María, Av. España 1680, V región, Chile article info Article history: Received 9 January 2013 Received in revised form 2 January 2014 Accepted 6 January 2014 Available online 11 January 2014 Keywords: Parameter tuning Constraint satisfaction problems Consistency techniques Algorithm configuration Algorithm design abstract Heuristic based algorithms are typically constructed following an iterative process in which the designer gradually introduces or modifies components or strategies whose performance is then tested by empir- ical evaluation on one or more sets of benchmark problems. This process often starts with some generic or broadly applicable problem solving method (e.g., metaheuristics, backtracking search), a new algorith- mic idea or even an algorithm suggested by theoretical considerations. Then, through an iterative process, various combinations of components, methods and strategies are implemented/improved and tested. Even experienced designers often have to spend substantial amounts of time exploring and experiment- ing with different alternatives before obtaining an effective algorithm for a given problem. In this work, we are interested in assisting the designer in this task. Considering that components, methods and strategies are generally associated with parameters and parameter values, we propose a method able to detect, through a fine-tuning process, ineffective and redundant components/strategies of an algorithm. The approach is a model-free method and applies simple consistency techniques in order to discard values from the domain of the parameters. We validate our approach with two algorithms for solving SAT and MIP problems. Ó 2014 Elsevier B.V. All rights reserved. 1. Introduction When we design an algorithm for problem solving, generally we address several important decisions related to its components, such as which variable-ordering heuristic to use, which transfor- mation algorithm to use, and whether to include some specific methods. These decisions can be handled by algorithm parameters. Then, we can apply a fine-tuning process to determine which com- ponents are crucial for the algorithm performance. There exist several automated procedures to find the optimal instantiation of the algorithm parameters (called configuration here) of a given algorithm. The problem of finding the optimal configura- tion is known as the algorithm configuration problem (AC) [17,3,9,16]. Automated procedures for solving this problem are called configuration algorithms or configurators, while algorithms whose parameters are tuned are called target algorithms. Usually, finding near-optimal configurations allows good per- formance on large data sets. However, this does not directly assist with the design phase, specifically, with discarding, simplifying or just better understanding the components of the target algorithm. Several configuration algorithms exist that supply information about the parameters/components of the target algorithm in addi- tion to finding near-optimal configurations. Sampling methods (e.g., Latin-Square [21] and Taguchi Orthogonal Arrays [26]) take a representative sample of the configuration space using a full factorial design. The different con- figurations are analyzed to predict which parameter values work best and which are the most robust. CALIBRA [2] and meta-GA [21] reduce the search area from which new configurations are sampled at each iteration. Thus, they can be only used to find good configurations and are not well suited to analyzing the parameters. Model-based methods construct a model based on a reduced set of configurations. This model predicts the performances of new configurations. A common approach is to use a regression method to predict the utility of a configuration [12,22,14]. Based on the model, it is possible to identify some features of the parameters. Sequential model-based methods allow for a better exploration of the configuration space. Coy et al. [11] proposes a procedure consisting of a model-based method followed by a local search pro- cedure to optimize the parameter values. SPO [5,6] goes further. In each iteration, it generates a new set of configurations and predicts their utilities using the current model. The vectors with the highest predicted performance are used to update the model for the next iteration. Finally, the procedure returns an accurate model of the most promising areas. Model-based methods commonly use 0950-7051/$ - see front matter Ó 2014 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.knosys.2014.01.005 ⇑ Corresponding author. Tel.: +56 322654962. E-mail addresses: iaraya@inf.utfsm.cl (I. Araya), mcriff@inf.utfsm.cl (M.-C. Riff). Knowledge-Based Systems 60 (2014) 73–81 Contents lists available at ScienceDirect Knowledge-Based Systems journal homepage: www.elsevier.com/locate/knosys