Vol.:(0123456789) 1 3 Evolving Systems https://doi.org/10.1007/s12530-020-09345-2 ORIGINAL PAPER Automatic tuning of hyperparameters using Bayesian optimization A. Helen Victoria 1  · G. Maragatham 1 Received: 21 December 2019 / Accepted: 15 May 2020 © Springer-Verlag GmbH Germany, part of Springer Nature 2020 Abstract Deep learning is a feld in artifcial intelligence that works well in computer vision, natural language processing and audio recognition. Deep neural network architectures has number of layers to conceive the features well, by itself. The hyperpa- rameter tuning plays a major role in every dataset which has major efect in the performance of the training model. Due to the large dimensionality of data it is impossible to tune the parameters by human expertise. In this paper, we have used the CIFAR-10 Dataset and applied the Bayesian hyperparameter optimization algorithm to enhance the performance of the model. Bayesian optimization can be used for any noisy black box function for hyperparameter tuning. In this work Bayes- ian optimization clearly obtains optimized values for all hyperparameters which saves time and improves performance. The results also show that the error has been reduced in graphical processing unit than in CPU by 6.2% in the validation. Achiev- ing global optimization in the trained model helps transfer learning across domains as well. Keywords Hyperparameters · Optimization · CIFAR-10 · Black box function 1 Introduction Due to the immense growth of data from various sources, digital data can be used for more exciting applications. Deep learning has the scope to use the deluge of data to build sophisticated and intricate deep neural network models. Such customization engenders a large number of layers by increasing the number of hyperparameters. Since deep learn- ing is considered to be a black box approach the researcher doesn’t have much scope in hand tuning the parameters as the layers are hidden and there are many hyperparameters related to network structure and training algorithms as well. Bayesian optimization is most useful while optimizing the hyperparameters of a deep neural network, where evaluat- ing the accuracy of the model can take few days for train- ing. The aim of optimizing the hyperparameters is to fnd an algorithm that returns best and accurate performance obtained on a validation set. The optimizer fnds the best hyperparameters which yield the best score on the test set. Bayesian optimization is also widely used in diverse design problems in diferent felds, such as gait recognition (Feurer et al. 2019), environmental sustainment and moni- toring (Marchant and Ramos 2012), algorithm confguration using optimization (Hutter et al. 2014), automatic machine learning (Snoek et al. 2012; Bergstra et al. 2011; Hofman et al. 2014), reinforcement learning (Brochu et al. 2009), and big data applications (Shahriari et al. 2016). Bayesian optimization works fairly well when the dataset for clas- sifcation is non-linear, complex and noisy, since the com- putation for identifying the hyperparameters is expensive thereby afecting model performance. The main motivation of this work is to reduce the training time of deep neural network using Bayesian optimization without sparing the model performance. 2 Related work 2.1 Multiple parameter optimization Deep learning architectures has various layers hence before ftting into a model we have to confgure all the hyperparam- eters that leads to proper learning of the model which can yield better classifcation. The choice of hyperparameters * A. Helen Victoria helenvia@srmist.edu.in G. Maragatham maragatg@srmist.edu.in 1 Department of Information Technology, SRM Institute of Science and Technology, Kattankulathur, Chennai, Tamil Nadu, India