Neural Networks, Vol. 4, pp. 643-055. 1991 0893-6080/91 $3.00 + .00 Printed in the USA. All rights reserved. Copyright ,c 1991 Pergamon Press pie ORIGINAL CONTRIBUTION Representation and Processing in a Stochastic Neural Network: An Integrated Approach M. M. VAN HULLE 12 AND G. A. ORBAN 1 'Laboratorium w~or Neuro-en Psychofysiologie, Katholieke Universitcit Leuvcn, Lcuven, Belgium ~'Departmcnt of Electroteehnology, Katholicke Universiteit Leuven, t leverlcc, Belgium (Received 1 September 1989: revised and accepted 23 January 1991 ) Abstract--To relate the stochastic properties of individual neurons to the representational and processing abilities of a network built with these neurorts, a generic framework is introduced. Herein, representational abilities become expressed in terms of the type of probability density function the network uses to encode the environment from which it receives input. Processing abilities are characterized by the type of processing procedure used by the network, an analogue of simulated annealing, and are shown to be uhimately related to representational abilities. The condition under which the analogue converges is established. Keywords--Subjective probability density function, Stochastic networks, Biological networks. 1. INTRODUCTION Conceptualization of those properties of individual neurons which form the underpinning of the repre- sentational and processing abilities of a network built with these neurons, constitutes a leading topic of research among network researchers. However, both computational abilities are rarely approached in an integrated fashion. In few cases (Koenderink, 1984a, 1984b) have the representational abilities of artificial neural networks (ANNs) been related to the pro- cessing properties of the networks" neurons without referring to some adaptational paradigm or learning rule, modifying connection weights. Analysis of pro- cessing in biological neural networks is usually re- stricted to the representational properties of individ- ual neurons such as receptive field properties. This paper attempts to derive a framework for generic stochastic ANNs wherein these computational abil- Acknowledgements: The authors would like to thank S. Rai- guel and I. Tollenaere, Laboratorium voor Neuro- en Psycho- fysiologic, Katholieke Universiteit Leuven, for their valuable comments on earlier drafts of this paper. The helpful comments of the reviewers arc also gratefully acknowledged, especially the one pointing out the work of V. I. Kryukov, previously unknown to us. The first author holds a postdoctoral fellowship of the Katholiekc Universiteit Leuven. This work was supported by a grant AI/RF/01 from the Belgian Ministry of Science to G.A.O, Requests for reprints should be sent to Dr. G. A. Orban, Laboratorium voor Neuro-en Psychofysiologie, Katholieke Univ- ersiteit Leuven, Campus Gasthuisberg, Herestraat, B-3000 Leu- yen, Belgium. 6~ ities are regarded as two expressions of the same stochastic mechanism, underpinned by the inherent noisiness of neurons and synapses. Representational abilities of ANNs are studied in terms of memory states in content-addressable mem- ories (Hopfield, 1982; McEliece, Posner, Rodemich, & Venkatesh, 1987), in terms of metastable states as in statistical physics (Kirillov, G. N. Borisyuk, R. M. Borisyuk, Kovalenko, Ye, & Kryukov, 1986; Kryu- kov, 1984), or in terms of mapping of the environ- ment's structure onto the network's internal state. Mapping has invariably been associated with adap- tational paradigms such as learning and competitive behaviour emerging from network dynamics. Map- pings have been studied for deterministic networks such as laterally inhibited networks (Kohonen, 1984), for competitive learning paradigms (Linsker, 1986; Rumelhart & Zipser, 1986) and for stochastic net- works like the Boltzmann machine (Ackley, Hinton, & Sejnowski 1985; Sejnowski & Kienker, 1986), Harmony theory (Smolensky, 1986), and the Gibbs sampler (S. Geman & D. Geman, 1984). In this con- tribution, we will concentrate on a nonlearning, but fully stochastic and generic network model. Repre- sentational abilities will be determined in terms of the type of probability density function (p.d.f.) of possible network states the network uses to represent the environment to be perceived: the p.d.f, will be termed subjective p.d.f. (Cox, 1946). To determine this p.d.f., a real scalar-valued function will be in- troduced to summarize, in a compact way, critical information about network dynamics. Such a func-