Minimization of Quantization Errors in Digital Implementations of Multi Layer Perceptrons Jakob Carlström Department of Computer Systems, Uppsala University, Box 325, S-751 05 Uppsala, Sweden, URL: http://www.docs.uu.se E-mail: jakobc@docs.uu.se Abstract Quantization errors appear in digital implementations of Multi Layer Perceptrons (MLPs), whose weight and signal values are represented as binary integers of limited length. This paper discusses how to select the representation of weights and how to modify the training in a way that minimizes such errors. Experiments on MLPs for sonar data classifica- tion and ATM link admission control show that backpropagation train- ing with clipping of the weights during training can be a useful learning algorithm in a constrained weight space. Other experiments show that a representation of weights which optimizes an MLP’s performance can be found by minimizing the sum of errors caused by clipping of the weights in the training phase and by discretization in the feed-forward phase. The shorter binary words used, the more should the weights be clipped during training to minimize the quantization error. A hardware motivated model of discretization noise is used in the simulations. 1 Introduction In digital hardware implementations of neural networks, the binary word length required to store data is of great importance. Generally, longer binary words for the data gives better accuracy, but augments the required complexity and cost of the hardware, and/or increases the response time of the neural network. To be able to reduce the word length, we must know how to select a representation of data which gives as little degradation of the neural network’s performance as possible, given a fixed word length. This paper discusses the effects of restricting the binary word length in the com- putations of Multi Layer Perceptrons (MLPs). Experiments on two near-real world classification problems illustrate how the degradation of classification per- formance can be reduced by a good representation of weight and signal values in the MLP, given a fixed binary word length. 2 Quantization means clipping and discretization Binary numbers of a fixed word length are quantized into a fixed number of levels. These levels can be distributed either uniformly or nonuniformly. In [YP91], sev- eral nonuniform quantization methods were compared to uniform quantization of weight values. The uniform quantization showed the best results, when applied