Chapter 3 Foundations of Artificial Neural Networks The rapid growth of computational power of computers is one of the basic qualities in the development of computer science. Therefore, informatics is applied to solving more and more complex problems and, what follows, the demand for bigger and more complex software occurs. It is not always possible, however, to use classical algorithmic methods to create such a type of software. There are two reasons for it. First of all, a good model of the relation between the input and output parameters often either does not exist at all or it cannot be created at the present level of scientific knowledge. It is worth of mentioning that the algorithmic approach requires the knowledge of the explicit form of the mapping between the aforementioned sets of parameters. Secondly, even if the model is given, the algorithmic approach can be impossible regarding its over-complexity. It can be both complexity of the task on the stage of the algorithm creating, and too slow working of the implemented system. The latter one is a critical parameter especially in the on-line systems. Therefore, the alternative approaches, in comparison with the classical algorithmic approach, are developed intensively. Artificial neural networks are included into this group of methods. The neurophysiological studies of functional properties of nervous systems enabled researchers at the beginning of the 1940’s to formulate the cybernetic model of the neuron [58] which, slightly modified, is commonly used up to present. At the turn of 1950’s and 1960’s the first artificial neural systems - PERCEPTRON and ADALINE were constructed. They were electromechanical systems. The first algo- rithms for setting of the synaptic weights in such a type of systems were worked out. Those pioneering attempts attracted attention to the possibilities of such systems. At the same time, however, significant limits were discovered. Nowadays, from the perspective of the time, it is known that on the one hand, the limits were caused by the lack of proper mathematical models of neural networks. On the other hand, they were caused by the application of just one type of artificial neural networks - the multilayer © Springer International Publishing AG, part of Springer Nature 2019 A. Bielecki, Models of Neurons and Perceptrons: Selected Problems and Challenges, Studies in Computational Intelligence 770, https://doi.org/10.1007/978-3-319-90140-4_3 15