ORDER SELECTION IN FINITE MIXTURE MODELS 1 Jiahua Chen, Abbas Khalili Department of Statistics and Actuarial Science University of Waterloo Abstract A fundamental and challenging problem in the application of finite mixture models is to make inference on the order of the model. In this paper, we develop a new penalized likelihood approach to the order selection problem. The new method deviates from the information-based methods such as AIC and BIC by introducing two penalty functions which depend on the mixing proportions and the component parameters. The new method is shown to be consistent and have other good properties. Simulations show that the method has much better performance compared to a number of existing methods. We further demonstrate the new method by analyzing two well known real data sets. Short Title: ORDER SELECTION 1. Introduction. Making inference on the number of components of the model is a fundamental and challenging problem in the application of finite mixture models. A mixture model with a large number of components can provide a good fit to the data, but has poor interpretive values. Complex models as such are not favoured in applications in the name of parsimony, and for the sake of preventing over-fitting of the data. A large number of statistical methods for order selection have been pro- 1 AMS 2000 subject classifications. Primary 62G05; secondary 62G07. KEY WORDS: E-M algorithm, finite mixture model, LASSO, penalty method, SCAD. 1