Applied Soft Computing 35 (2015) 52–65 Contents lists available at ScienceDirect Applied Soft Computing j ourna l ho me page: www.elsevier.com/locate /asoc A novel hybrid learning algorithm for full Bayesian approach of artificial neural networks Ozan Kocada˘ glı * Department of Statistics, Mimar Sinan Fine Arts University, Silahsör Cad., No: 71, Bomonti/Sisli, Istanbul, Turkey a r t i c l e i n f o Article history: Received 16 November 2013 Received in revised form 24 April 2015 Accepted 2 June 2015 Available online 25 June 2015 Keywords: Bayesian neural networks Bayesian learning Hierarchical Bayesian models Genetic algorithms Markov chain Monte Carlo Hybrid Monte Carlo a b s t r a c t The Bayesian neural networks are useful tools to estimate the functional structure in the nonlinear sys- tems. However, they suffer from some complicated problems such as controlling the model complexity, the training time, the efficient parameter estimation, the random walk, and the stuck in the local optima in the high-dimensional parameter cases. In this paper, to alleviate these mentioned problems, a novel hybrid Bayesian learning procedure is proposed. This approach is based on the full Bayesian learning, and integrates Markov chain Monte Carlo procedures with genetic algorithms and the fuzzy member- ship functions. In the application sections, to examine the performance of proposed approach, nonlinear time series and regression analysis are handled separately, and it is compared with the traditional training techniques in terms of their estimation and prediction abilities. © 2015 Elsevier B.V. All rights reserved. 1. Introduction Within Bayesian perspective, it is assumed that uncertainty of any quantity of interest can be expressed and measured by prob- abilistic distributions. This framework provides a natural way to estimate the functional structure in the nonlinear systems. For this reason, the Bayesian learning is mostly treated in the regression, the time series, the classification and the density estimation applica- tions of the artificial neural networks (ANNs). Bayesian learning in the ANNs are typically based on Gaussian approximation, ensem- ble learning and Markov chain Monte Carlo (MCMC) simulations known as full Bayesian approach. For ANNs, Gaussian approxima- tion was introduced by Buntine and Weigend [1] and MacKay [2] in which one well-established procedure for approximating the integrals over parameter space, known as Laplace’s method. This approach is to model the posterior distribution by a Gaussian dis- tribution, centered locally at a mode of posterior distribution of parameters [2]. The ensemble learning was introduced by Hinton and Camp [3] in which the approximating distribution is fitted globally by minimizing a Kullback–Leibler divergence rather than locally. In the context of Full Bayes approach, Neal [4] introduced advanced Bayesian simulation methods in which MCMC simula- tions are used to generate samples from the posterior distribution. * Tel.: +90 212 246 0011x5511; fax: +90 2122611121. E-mail address: ozankocadagli@gmail.com However, MCMC techniques can be computationally expensive, and also suffer from assessing the convergence. For this reason, Neal [4] integrated Bayesian learning with Hybrid Monte Carlo (HMC) method introduced by Duane et al. [5] to overcome the mentioned shortcomings. Afterwards, the Bayesian applications to ANNs was reviewed thoroughly in [6–8]. In the literature, there are the remarkable studies that deal with the specific problems related to ANNs from Bayesian per- spective. For instances, Insua and Müller [9], Marrs [10], Holmes and Mallick [11] worked on the issue of selecting the number of hidden neurons with the growing and the pruning algorithms for the dimensionality problem in the ANNs. In these studies, they applied the reversible jump MCMC algorithm introduced by Green [12], Richardson and Green [13]. Freitas [14] incorporated the particle filters and the sequential Monte Carlo (MC) methods in the BNNs. Liang and Wong [15] proposed to the evolutionary MC algorithm which samples the parameters in the ANNs from the Boltzmann distribution using the mutation, the crossover and the exchange operations defined in the Genetic Algorithms (GAs). Chua and Goh [16] used the Gaussian approach for the multivariate modeling in the ANNs. They used the stochastic gradient descent algorithm integrated with the evolutionary operators to produce the parameters in the ANNs. Liang [17] and Xie et al. [18] used the truncated Poison priors to determine the neuron numbers in the hidden layers, and estimated the parameters in the ANNs via the evolutionary MC algorithms proposed by Liang and Wong [15]. Lampinen and Vehtari [19], Vanhatalo and Vehtari [20] improved http://dx.doi.org/10.1016/j.asoc.2015.06.003 1568-4946/© 2015 Elsevier B.V. All rights reserved.