Neural Network Training with Constrained Integer Weights V.P. Plagianakos University of Patras Department of Mathematics, U.P. Artificial Intelligence Research Center (UPAIRC), GR-26500 Patras, Greece. e-mail: vpp@mat h.upatras. gr Abstract- In this contribution we present neural network training algorithms, which are based on the differential evolution (DE) strategies intro- duced by Storn and Price [Journal of Global Op- timization 11, 341-359, 19971. These strategies are applied to train neural networks with small integer weights. Such neural networks are bet- ter suited for hardware implementation than the real weight ones. Furthermore, we constrain the weights and biases in the range [-2k + 1, 2k - 11, for k = 3,4,5. Thus, they can be represented by just k bits. These algorithms have been designed keeping in mind that the resulting integer weights require less bits to be stored and the digital arithmetic operations between them are easier to be imple- mented in hardware. Obviously, if the network is trained in a constrained weight space, smaller weights are found and less memory is required. On the other hand, as we have found here, the network training procedure can be more effec- tive and efficient when large weights are allowed. Thus, for a given application a trade off between effectiveness and memory consumption has to be considered. Our intention is to present results of evolu- tion algorithms on this difficult task. Based on the application of the proposed class of methods on classical neural network benchmarks, our ex- perience is that these methods are effective and reliable. 1 Introduction Artificial Feedforward Neural Networks (FNNs) have been widely used in many application areas in recent years and have shown their strength in solving hard problems in Artificial Intelligence. Although many dif- ferent models of neural networks have been proposed, multilayered FNNs are the most common. FNNs consist of many interconnected identical simple processing units, called neurons. Each neuron calculates the dot product of the incoming signals with its weights, adds the bias to M.N. Vrahatis University of Patras Department of Mathematics, U.P. Artificial Intelligence Research Center (UPAIRC), GR-26500 Patras, Greece. e-mail: vrahatis@math.upatras.gr the resultant, and passes the calculated sum through its activation function. In a multilayer feedforward network the neurons are organized into layers with no feedback connections. FNNs can be simulated in software, but in order to be utilized in real life applications, where high speed of ex- ecution is required, hardware implementation is needed. The natural implementation of an FNN - because of its modularity - is a parallel one. The problem is that the conventional multilayer FNNs, which have continuous weights, are expensive to implement in digital hardware. Another major implementation obstacle is the weight storage. FNNs having integer weights and biases are easier and less expensive to implement in electronics as well as in optics and the storage of the integer weights is much easier to be achieved. Another advantage of the FNNs with integer weights is their immunity to noise in the training data, Such networks only capture the main feature of the training data. Low amplitude noise that possibly contaminates the training data cannot perturb the discrete weights, because those weights require relatively large variations to jump from one integer value to another. In recent publications [6, 51 we have studied neural networks with integer weights. Here, we proceed further by studying neural networks having integer weights con- strained in the ranges [-2k + 1, 2k - 11, k = 3,4,5 which correspond to k-bit integer representation of the weights. This property reduces the amount of memory required for weight storage in digital electronic implementations. Additionally, it simplifies the digital multiplication oper- ation, since multiplying any number with a k-bit integer requires only the following number of basic instructions: one sign change, (k - l)(k - 2)/2 one-step left shifts and k - 2 additions. Finally, if inputs are restricted to the set { -1, l} (bipolar inputs), the neurons in the first hid- den layer require only sign changes during multiplication operations, and only integer additions. the incremental adaptation of the connection weights that propagate information between the neurons, is a sub- ject of considerable ongoing research and numerous al- gorithms have been proposed to this end. The majority The efficient supervised training of FNNs, i.e. 0-7803-5536-9/99/$10.00 01999 IEEE 2007