Peter Grünwald May 2007 The Catch-Up Phenomenon 1 The Catch-Up Phenomenon Peter Grünwald www.grunwald.nl Joint work with Tim van Erven and Steven de Rooij The AIC-BIC Dilemma • Two main types of model selection methods: 1. AIC-type – Akaike Information Criterion (AIC, 1973) – leave-one-out cross-validation – DIC, C p 2. BIC-type – Bayesian Information Criterion (BIC, 1978) – prequential validation – Bayes factor model selection – standard MDL The AIC-BIC Dilemma • Two main types of model selection methods: 1. AIC-type – Akaike Information Criterion – leave-one-out cross-validation – DIC, C p 2. BIC-type – Bayesian Information Criterion – prequential validation – Bayes factor model selection – standard MDL inconsistent consistent asymptotic overfitting The AIC-BIC Dilemma • Two main types of model selection methods: 1. AIC-type – Akaike Information Criterion – leave-one-out cross-validation – DIC, C p 2. BIC-type – Bayesian Information Criterion – prequential validation – Bayes factor model selection – standard MDL inconsistent consistent optimal rate slower rate asymptotic underfitting The AIC-BIC Dilemma • Which method to use in practice? • Fierce discussion in statistical, psychological and biological literature, going on for more than 20 years now • Googling “AIC and BIC”: 355000 hits. The Best of Both Worlds • We give a novel analysis of the slower convergence rate of BIC-type methods: the catch-up phenomenon • This allows us to define a model selection/averaging method that, in a wide variety of circumstances, 1. is consistent 2. achieves optimal convergence rates 3. satisfies Dawid’s weak prequential principle (some have claimed that such a combination is impossible!) • Method is “MDL/prequential, but not as we know it”