[2] Walach H. There is no hierarchy in the first placeda comment on J€ urgen Windeler. J Clin Epidemiol 2016;75:128e9. [3] Walach H, Loef M. Using a matrix-analytical approach to synthesiz- ing evidence solved incompatibility problem in the hierarchy of evidence. J Clin Epidemiol 2015;68:1251e60. [4] Filk T. It is the theory which decides what we can observe (Einstein). In: Dzhafarov E, Jordan S, Zhang R, Cervantes V, editors. Contex- tuality from quantum physics to psychology. Singapore: World Scientific Publishers; 2016:77e92. [5] Aerts D. Quantum theory and human perception of the macro-world. Front Psychol 2014;5:554. http://dx.doi.org/10.1016/j.jclinepi.2016.02.029 We should not be so quick to abandon the use of domain experts and full models (letter commenting: J Clin Epidemiol. 2015;71C:76–85.) The recent tutorial by Bagherzadeh-Khiabani et al. [1], on variable selection for clinical prediction models, is of very high importance. This type of research has been sorely missing from the literature. Given the possible effects on the field from this study, it seems reasonable to have a bit more discussion surrounding some key points raised by this study. First, it would be helpful to have additional evidence to support the authors’ point that it is no longer acceptable to believe that more variables produce better predictive perfor- mance when applied to future patients. Many would believe that full models, with variables chosen by domain experts, predict more accurately than do models that have had data- driven variable reduction methods applied with the use of the same data used to build the models. It would be nice to see more literature showing that full models did not pre- dict as accurately as reduced models to be comfortable this fact is well documented. These head-to-head comparisons appear to be quite rare. Second, and closely related to the first, is the issue of us- ing domain experts to choose the variables to include in a prediction model. Although experts might be biased, and they surely will struggle with the actual weighting of the predictors, it would seem that models with predictors cho- sen by experts would fare well compared to models with predictors chosen purely by the data. This would seem particularly important when concern is raised regarding transportability [2]. Testing a model’s performance using a single split of the initial data, into training and testing, is not as stringent as when testing performance across a different data collection system; a spurious predictor is more likely to survive in the former versus the latter. As such, the argument in favor of having experts choose the predictors is largely based on concerns about performance when the prediction model is tested in a very different sys- tem (e.g., geographically different). Again it seems that transportability has been rarely evaluated with respect to the value of domain experts choosing the predictors. Third, also related to the second, is a concern about the generalizability of the present results. A single-split sample approach is strongly discouraged in the literature [3], so it would be comforting to know the results of Bagherzadeh-Khiabani et al. were robust to more stringent validation procedures than this antiquated approach. Moreover, it would be comforting to see that the findings held beyond this single data set. Fourth, the results in Table 1 of Bagherzadeh-Khiabani et al. could use some further interpretation. Namely, it was not mentioned how the winners were chosen. There would appear to be numerous ties here in the performances, making it difficult to judge which model is truly superior to which. (Minor point: there appears to be a typo for the AIC for SBP value). In summary, the tutorial by Bagherzadeh-Khiabani et al. is important work, the type of which we need more. It is nice to see studies like this given the importance of the topic. However, it may be premature to abandon the use of domain experts for selection of covariates in prediction models, especially when the list of predictors is short, and they are routinely collected. Michael W. Kattan* Department of Quantitative Health Sciences Cleveland Clinic 9500 Euclid/JJN3-01 Cleveland, OH 44195, USA Frank E. Harrell Jr. Department of Biostatistics School of Medicine Vanderbilt University, 2525 West End, Ste. 11000 Nashville, TN 37203, USA *Corresponding author. Tel.: þ1-216-444-0584; fax: þ1-216-445-7659. E-mail address: kattanm@ccf.org (M.W. Kattan) References [1] Bagherzadeh-Khiabani F, Ramezankhani A, Azizi F, Hadaegh F, Steyerberg EW, Khalili D. A tutorial on variable selection for Clinical prediction models: feature selection methods in data mining could improve the results. J Clin Epidemol 2016;71:76e85. [2] Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med 1999;130:515e24. [3] Steyerberg EW, Harrell FE Jr. Prediction models need appropriate internal-external, and external validation. J Clin Epidemiol 2016;69: 245e7. http://dx.doi.org/10.1016/j.jclinepi.2016.02.010 The authors’ reply to letter to the editor re: Bagherzadeh-Khiabani et al., J Clin Epi, 2015 We do appreciate the comments on our article [1] made by Kattan & Harrell [2]. Concerning the first comment, we acknowledge the importance of domain knowledge. The 55 DOI of original article: http://dx.doi.org/10.1016/j.jclinepi.2015.10.002. 131 Letters to the Editor / Journal of Clinical Epidemiology 75 (2016) 126e132