Analysis of small sample size studies using nonparametric bootstrap test with pooled resampling method Alok Kumar Dwivedi, a,b * † Indika Mallawaarachchi b and Luis A. Alvarado b Experimental studies in biomedical research frequently pose analytical problems related to small sample size. In such studies, there are conflicting findings regarding the choice of parametric and nonparametric analysis, especially with non-normal data. In such instances, some methodologists questioned the validity of parametric tests and suggested nonparametric tests. In contrast, other methodologists found nonparametric tests to be too conservative and less powerful and thus preferred using parametric tests. Some researchers have recommended using a bootstrap test; however, this method also has small sample size limitation. We used a pooled method in nonparametric bootstrap test that may overcome the problem related with small samples in hypothesis testing. The present study compared nonparametric bootstrap test with pooled resampling method corresponding to parametric, nonparametric, and permutation tests through extensive simulations under various conditions and using real data examples. The nonparametric pooled bootstrap t-test provided equal or greater power for com- paring two means as compared with unpaired t-test, Welch t-test, Wilcoxon rank sum test, and permutation test while maintaining type I error probability for any conditions except for Cauchy and extreme variable lognormal distributions. In such cases, we suggest using an exact Wilcoxon rank sum test. Nonparametric bootstrap paired t-test also provided better performance than other alternatives. Nonparametric bootstrap test provided benefit over exact Kruskal–Wallis test. We suggest using nonparametric bootstrap test with pooled resampling method for comparing paired or unpaired means and for validating the one way analysis of variance test results for non-normal data in small sample size studies. Copyright © 2017 John Wiley & Sons, Ltd. Keywords: bootstrap test; nonparametric test; parametric test; resampling method; small sample size; experimental studies Introduction Common designs in biomedical research include experimental designs in laboratory studies and pilot randomized controlled designs in clinical studies. Data from these studies are often analyzed using simple univariate statistical tests. In such studies, some methodologists have suggested using parametric tests, whereas others have preferred the use of nonparametric tests [1,2]. Generally, these studies are based on small sample sizes, thus conventional statistical guidelines would recommend using nonpara- metric approaches for analyzing data from these studies [3]. There are two ways of obtaining p-values in nonparametric tests. One way calculates exact probability of obtaining observed or more extreme results under the null hypothesis, which is suitable for small sample size studies, and referred to as exact non- parametric test. The other way calculates p-value based on asymptotic property, which is suitable for large sample size studies, and referred to as asymptotic nonparametric test. Asymptotic and exact proce- dures for computing p-values for most of the nonparametric tests are available. These tests are useful when the assumptions of parametric tests are under suspicion [4,5]. However, standard exact or asymptotic nonparametric methods do not perform well in many conditions with small sample size a Division of Biostatistics and Epidemiology, Department of Biomedical Sciences, Paul L. Foster School of Medicine, Texas Tech University Health Sciences Center, El Paso, Texas, U.S.A. b Biostatistics and Epidemiology Consulting Lab, Office of Research Resources, Texas Tech University Health Sciences Center, El Paso, Texas, U.S.A. *Correspondence to: Division of Biostatistics and Epidemiology, Department of Biomedical Sciences, Paul L. Foster School of Medicine, Texas Tech University Health Sciences Center, El Paso, Texas, U.S.A. † E-mail: alok.dwivedi@ttuhsc.edu Copyright © 2017 John Wiley & Sons, Ltd. Statist. Med. 2017 Research Article Received: 12 January 2016, Accepted: 31 January 2017 Published online in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/sim.7263