~~ zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA J. Phys. zyxwvutsr A Math. Gen. zyxwvut 26 (1993) 3165-3185. Printed in the UK zyxw The effect of correlations in neural networks A Wendemutht, M Oppert and W Kinzelt Institut f~ Thearetische F'hpik, Justus-Liebig UniversiW Heinrich-Buff-Ring 16, W-6300 Giekn. Germany Received 7 Sepkmkr 1992, in final form zyxw 4 March 1993 Abstract The effect of conelations in neural networks is investigated by wnsider@g biased input and output palm'" Statistical mechanics is applied @ study training times and intend potentials of the MINOVER and ADALME leaming algorithms. For the latter, a direet extension to generalization abilify is obtained. Comparison with computer simulations shows good apemen1 with theoretical predictions. With biased pattems, we find a decrease in Uaining times and internal potentials for the MINOVER algorithm, which, however. does not lead to faster storage zy of a given information measure. In ADALME training. characteristic times undergo a transition from order I to order N at any finite bias, for the leaming of patterns as well as for the decay of the generalization error. This leads to a rescaling of the gain parameters. 1. Introduction .The methods of statisticalm6chanics have been extensively used in the quantitative analysis of neural networks. An interesting feature is the network's perfomance during the leaming phase. We shall consider here two training algorithms in particular. For the ADALINE algorithm, the dynamical evolution of the G n i n g error and the generalization, error have been studied 18, IO]. For the MINOVER algorithm, the distribution of leaming times has been computed [Ill. However, the dynamics were obtained for the simplest case of randomly chosen, uncorrelated patterns only. A question first put fonvard by Gardner [6] in the context of storage capacities is the effect of correlation between pattems. Gardner found that the storage capacity of the optimal network rises monotonically and continuously from cr, = 2 for random patterns to a, = 00 for fully correlated patterns. Here, we shall investigate output and generalization ermm for correlated pattems in the ADALINE algorithm.' In contrast zyxwv to ~ardner's result for the (static) storage capacity, for the ADALINE algorithm we will find a discontinuous jump in the dynamical behaviour of the eiror decay for any finite coirelatiou. This will lead us to a recalculation of typical time constants. For the MINOVER algorithm, we^ shall calculate the distribution of leaming times. Furthermore, we^ investigate the effect of redundancy of information. The information capacity of a set ofpatterns decreases as the pattems become more correlated; as a result, the errors decrease faster. We will show, however, that agiven information cannot be leamt any faster by introducing redundancy, i.e. by spreading it over a larger set of correlated pattems. t Resent address: Theoretical Physics, Oxford University, 1 Keble Road, Oxford zyxw OX13NF'. UK. $ Present address: Physikalisches Instituf~ Julius-Maximilians Universit21, Am Hubland. W-8700 WOnburg, Federal Republic of Germany. 0305-4470/93/133165+221$07.S0 @ 1993 IOP Publishing Ltd 3165