Complex Sy st ems 6 (1992) 479-494 Refined Pruning Techniques for Feed-forward Neural Networks Anthony N. Burkitt " Computer Sciences Laboratory, Research School of Physical Sciences, Au stralian National University, GPO Box 4, Canberra, A CT 2601, Au stralia Peer Ueberholz Physics Department, University of Wupp ertal, Gauss-StraBe 20, D-5600 Wupp ertal1 , Germany Abstract. Pruning algorit hms for feed-forward neural networks typ- ically have the undesirable side effect of interfering with the learning procedure. The network reduction algorithm presented in this paper is implement ed by considering only directions in weight space t hat are ort hogonal to those required by the learning algorithm. In this way, the network redu ction algorithm chooses a minimal network from amon g the set of networks with const ant E-function values. It thus avoids introducing any inconsistency with learning by explicitly using t he r edundancy inherent in an oversize network. Themethod is tested on boolean problems and shown to be very useful in practice. 1. Introduction The problem of choosing the optimal network size and architecture for a given situation has remained intractable. Once a network architecture has been decided upon, learning procedures such as back-propagation [l J enable the network to be tr ained, provided that there are enough degrees of freedom (hidden units, weights and biases) . One of the principal methods for solving the network size problem in practice is pruning an oversize network; that is, elimin ating some of the weights according to some appropriate criteria. A variety of such algorithms have been described in the literatur e [2, 3, 4, 5, 6, 7, 8J . The primary reason for theextensive interest in such network opt imiza- tion algorit hms is the fact t hat minimal nets tend to have the best generaliza- tion propert ies [9, 10], as demonstrated both by recent rigorousresults [11, 12] and empiri cal investigations [4]. Networks that are too large are able to fit "Electronic mail address: tonyiUnimbus. anu. edu. au