Evolving Weights and Transfer Functions in Feed Forward Neural Networks M.Annunziato 1 , I.Bertini 1 , M.Lucchetti 2 , S.Pizzuti 1,3 1 ENEA – Energy, New technologies, Environment Agency ‘Casaccia’ R.C., Via Anguillarese 301, 00060 Rome Italy Phone: +39-0630484411, Fax: +39-0630484811 email:{mauro.annunziato, ilaria.bertini, stefano.pizzuti}@casaccia.enea.it 2 University of Rome “La Sapienza” Dept. of Computer and Systems Science, Via Eudossiana 18, 00184 Rome Italy Phone: +39-06-44585938, Fax: +39-06-44585367 email: lucchetti@dis.uniroma1.it 3 CS - Communication Systems s.r.l. Piazza della Repubblica 32, Milan Italy ABSTRACT: In this paper we show different evolutionary algorithms applied to the simultaneous off-line evolution of weights and transfer functions of feed-forward neural networks. Experimentation has been carried out with classical benchmarks when weights and both weights and transfer function are evolved and a comparison of the proposed evolutionary methods with classical methodologies (the back-propagation algorithm) are shown. Results are very promising and show the effectiveness of the addressed evolutionary methodologies to solve the problem of simultaneously finding the optimal weights and transfer functions of a neural network. KEYWORDS: Evolutionary Neural Networks, Evolutionary Algorithms, Feed Forward Neural Networks, Smart Adaptive Systems 1. INTRODUCTION Considerable research on the off-line evolution of Artificial Neural Networks (ANN) using Evolutionary Algorithms (EAs) has been carried out in recent years giving rise to a new branch of ANN known as Evolutionary Neural Networks [1]. In this context most of the research has concentrated on the evolution of weights and topological structures but relatively little has been done on the evolution of node transfer functions and the simultaneous evolution of both weights and node transfer functions. Often the transfer function of each node in the architecture has been assumed to be fixed and predefined by human experts but the transfer function has been shown to be an important part of an ANN architecture which has significant impact on ANN's performance. Moreover the transfer function is often assumed to be the same for all the nodes in an ANN, at least for all the nodes in the same layer. In this way the application of EAs to the evolution of node transfer functions to get the optimal network’s architecture seems to be a promising research field. Stork et al. [2] were, to our best knowledge, the first to apply EAs to the evolution of both topological structures and node transfer functions even though only simple ANNs with seven nodes were considered. The transfer function was specified in the structural genes in their genotypic representation. It was much more complex than the usual sigmoid function because they tried to model a biological neuron in the tailflip circuitry of crayfish. White and Ligomenides [3] adopted a simpler approach to the evolution of both topological structures and node transfer functions. For each individual (i.e., ANN) in the initial population, 80% nodes in the ANN used the sigmoid transfer function and 20% nodes used the Gaussian transfer function. The evolution was used to decide the optimal mixture between these two transfer functions automatically. The sigmoid and Gaussian transfer function themselves were not evolvable. No parameters of the two functions were evolved. Liu and Yao [4] used EP to evolve ANNs with both sigmoidal and Gaussian nodes. Rather than fixing the total number of nodes and evolve mixture of different nodes, their algorithm allowed growth and shrinking of the whole ANN by adding or deleting a node (either sigmoidal or Gaussian). The type of node added or deleted was determined at random. Hwang et al. [5] went one step further. They evolved ANN topology, node transfer function, as well as connection weights for projection neural networks.