Adv Data Anal Classif DOI 10.1007/s11634-016-0253-y REGULAR ARTICLE A new approach for determining the prior probabilities in the classification problem by Bayesian method Thao Nguyen-Trang 1,2 · Tai Vo-Van 3 Received: 8 July 2015 / Revised: 12 April 2016 / Accepted: 10 May 2016 © Springer-Verlag Berlin Heidelberg 2016 Abstract In this article, we suggest a new algorithm to identify the prior probabilities for classification problem by Bayesian method. The prior probabilities are determined by combining the information of populations in training set and the new observations through fuzzy clustering method (FCM) instead of using uniform distribution or the ratio of sample or Laplace method as the existing ones. We next combine the deter- mined prior probabilities and the estimated likelihood functions to classify the new object. In practice, calculations are performed by Matlab procedures. The proposed algorithm is tested by the three numerical examples including bench mark and real data sets. The results show that the new approach is reasonable and gives more efficient than existing ones. Keywords Classification · Bayes error · BayesC · Prior probability Mathematics Subject Classification 62H30 · 68T10 1 Introduction Classification is assigning an object to an appropriate population. It is an important problem in statistical discriminant analysis and applied in many fields, such as eco- nomics, medicine, sociology, etc. Recently, there have been many different proposed B Thao Nguyen-Trang nguyentrangthao@tdt.edu.vn 1 Division of Computational Mathematics and Engineering, Institute for Computational Science, Ton Duc Thang University, Ho Chi Minh City, Vietnam 2 Faculty of Mathematics and Statistics, Ton Duc Thang University, Ho Chi Minh City, Vietnam 3 College of Natural Science, Can Tho University, Can Tho City, Vietnam 123