Commentary on Gronau and Wagenmakers Suyog H. Chandramouli 1 & Richard M. Shiffrin 1 Published online: 21 November 2018 Abstract The three examples Gronau and Wagenmakers (Computational Brain and Behavior , 2018; hereafter denoted G&W) use to demonstrate the limitations of Bayesian forms of leave-one-out cross validation (let us term this LOOCV) for model selection have several important properties: The true model instance is among the model classes being compared; the smaller, simpler model is a point hypothesis that in fact generates the data; the larger class contains the smaller. As G&W admit, there is a good deal of prior history pointing to the limitations of cross validation and LOOCV when used in such situations (e.g., Bernardo and Smith 1994). We do not wish to rehash this literature trail, but rather give a conceptual overview of methodology that allows discussion of the ways that various methods of model selection align with scientific practice and scientific inference, and give our recommendation for the simplest approach that matches statistical inference to the needs of science. The methods include minimum description length (MDL) as reported by Grünwald (2007); Bayesian model selection (BMS) as reported by Kass and Raftery (Journal of the American Statistical Association, 90, 773–795, 1995); and LOOCV as reported by Browne (Journal of Mathematical Psychology , 44, 108–132, 2000) and Gelman et al. (Statistics and Computing, 24, 997–1016, 2014). In this commentary, we shall restrict the focus to forms of BMS and LOOCV. In addition, in these days of BBig Data,^ one wants inference procedures that will give reasonable answers as the amount of data grows large, one focus of the article by G&W. We discuss how the various inference procedures fare when the data grow large. Keywords Model Selection . Overlapping Model Classes . Cross-Validation . Bayes Factor At the most conceptual level, our commentary is motivated by the desire to have statistics serve science not science serve statistics. To help make this happen, we propose Bayesian inference be carried out with the following components. A chief motivation for these proposals is the assumption that the Btrue^ model is never among and is always more complex than any instance in the models being considered: 1) Discretize all instance hypotheses and probabilities into sufficiently small intervals (multidimensional volumes). 2) Represent all model classes as combinations of instance intervals. 3) Approximate each instance interval by a point hypothesis within the interval. 4) Start Bayesian inference by finding the posterior proba- bility of every one of these point hypotheses (see Gelman and Carlin 2017). 5) When desired and desirable, form a posterior probability for each class by summing the posterior probabilities for the instances in the class, and use those class posteriors to make class inferences. 6) Carry out model class comparisons for model classes that do not overlap. To make this proposal generally applica- ble, one must allow certain types of shared instances that are Bdistinguishable^ to be separated into distinct hypotheses. We start with terminology: a model instance is fully specified—it has all parameter values defined and predicts a fully specified distribution of outcomes for the experiment in question. A model class is a collection of model instances, often defined by a given parametric form (e.g., a seven degree polynomial). Different model classes sometimes overlap by sharing instances. This is commonly seen when model classes are nested hierarchically (e.g., a two degree polynomial is nested in a seven degree polynomial), in which case, all the instances in the smaller class are also in the larger class. It is of course possible for model classes to overlap in non-nested fashion; e.g., class 1 for probability of success, p, might spec- ify p to lie in (0, .7) while class 2 might specify p to lie in (.4, * Suyog H. Chandramouli suchandr@iu.edu 1 Indiana University, Bloomington, IN, USA Computational Brain & Behavior (2019) 2:12–21 https://doi.org/10.1007/s42113-018-0017-1 # Springer Nature Switzerland AG 2018