Automatic Recurrent ANN Rule Extraction with Genetic Programming JuliÆn Dorado, Juan R. Rabuæal, Daniel Rivero, Antonino Santos, Alejandro Pazos University of A Coruæa, Campus Elviæa s/n, 15192, A Coruæa julian@udc.es, juanra@udc.es, infdrc00@ucv.udc.es , nino@udc.es, ciapazos@udc.es Abstract- Various rule-extraction techniques using ANNs have been used so far, most of them being applied on multi-layer ANNs, since they are more easily handled. In many cases, extraction methods focusing on different types of networks and training have been implemented, however, there are virtually no methods that view the extraction of rules from ANNs as systems which are independent from their architecture, training and internal distribution of weights, connections and activation functions. This paper proposes a rule-extraction system of ANNs regardless of their architecture (multi-layer or recurrent), using Genetic programming as a rule-exploration technique. I. INTRODUCTION Artificial Neural Networks (ANNs) are systems which are easily implemented and handled. These and other features make them optimal for problem-solving in various areas. However, many developers and researchers avoid using them, since they consider them as black boxes, that is, they are systems which, produce certain response outputs from a series of inputs, while the process through which those outputs are produced remains unknown. For that reason, in fields such as medicine, where their use is highly recommended, people do not consider ANNs to be accountable, due to the fact that the reason for their right functioning and the solutions they contribute cannot be explained. An example could be medical diagnosis. A computational system designed for diagnosis should be able to explain the reason for that diagnosis and how it was reached. Nevertheless, Expert Systems (ES) are able to explain the solution or response achieved, which is their main core and also their guarantee of success. Therefore, this paper tries to develop a system which carries out an automatic extraction of rules from already trained ANNs, thus obtaining the knowledge that an ANN obtains from the problem it solves. Different rule-extraction techniques using ANNs have been used so far, always applied to multi-layer ANNs due to the fact that they are more easily handled. These networks also have a limited capacity with regard to the knowledge which can be distributed among their connections. As may be inferred, the extraction of rules from recurrent ANNs is more complicated, due to the fact that past states intervene in neural activation, and that their capacity of distributed knowledge is considerably higher than that of multi-layer ANNs, since there are no restrictions to neural connectivity. If, besides, recurrent ANNs are used in dynamic problems where certain time characteristics such as the prediction of time series intervene, the task of extracting by means of the methods developed so far becomes harder, if not impossible for most of them. Therefore, the system presented can be applied to every ANN kind. For that reason, the system should comply with a series of requirements [1]: 1) It should have no ANN architecture requirements: A rule-extraction mechanism which can work with every type of neural network, including those which are highly interlinked or recurrent. 2) It should have no ANN training requirements: Many of the proposed algorithms are based on a given ANN training process for which rule-extraction is designed. Therefore, they are not valid for other training sets. 3) It should be correct: Many rule-extraction mechanisms only generate functioning approaches. It is desirable that the rules describe the ANN as accurately as possible. 4) It should have a highly expressive level: Rule language (syntax) characterises the compactness of the extracted symbolic language. Usually, powerful languages are desirable, due to the fact that a very compact and easy to understand rule language can be produced. II. STATE OF THE ART A. Genetic Programming Some people thinks that Cramer and Fuji, who published on evolving programs in 1985 and 1987 at the very first ICGA conference [2][3], are the pioneers of Genetic Programming (GP). But still others think that Friedberg from 1958 and 1959, who evolved machine language programs [4][5], is really the pioneer. John Koza created the term which titles the book Genetic Programming [6]. This book establishes formally the bases of GP used nowadays. Later, the same author published Genetic Programming II [7], and, recently, Genetic Programming III [8]. Both explore new possibilities of GP. Different branches derive form GP. One of the most promising ones with regard to Knowledge Discovery (KD) is that of fuzzy rules [9][10]. This branch derives form the union between fuzzy logic and systems based on rules (SBR). Fuzzy rules can be obtained by means of Evolutionary Computation (EC) with the technique known as Automatically Defined Functions (ADF) [7], which represent an evolution of the concept called Classical genetic Computing. 0-7803-7278-6/02/$10.00 ©2002 IEEE