REVIEW PAPER A Sensitivity Analysis of Three Nonparametric Treatment Effect Scores for Single-Case Research for Participants with Autism Monica E. Carr Received: 14 September 2014 /Accepted: 15 October 2014 /Published online: 29 October 2014 # Springer Science+Business Media New York 2014 Abstract The widely used percentage of nonoverlapping da- ta (PND) treatment effect calculation was compared to more recently developed methods which, it has been argued, better account for outlying variables and trend in single-case design (SCD) intervention studies. Percentage of all nonoverlapping data (PAND) and nonoverlap of all pairs (NAP) were selected for comparison as both are amenable to hand calculation, making them widely accessible to clinicians and teachers as well as researchers. A data set was developed through a systematic search of peer-reviewed literature on self- management interventions conducted with participants with autism spectrum disorder (ASD). Treatment effect sizes de- rived from each method were compared for studies that pro- vided sufficient data. Results indicated that PND provided a conservative measure of strength of treatment effect when compared to PAND and NAP scores. Interpretation scales for treatment effect scores derived from each method were reviewed. Implications for selecting a calculation method for participants with ASD are discussed. Keywords Autism . Treatment effect . Nonparametric . Hand calculation . PND Introduction In 1994, the American Psychological Association (APA) pub- lication manual encouraged the inclusion of a treatment effect size in research reports. Since that time, the APA Taskforce on Statistical Inference (1999) argued that a treatment effect size permits the evaluation of the stability of findings across sam- ples and is important to future meta-analyses. Subsequently, in some quarters, treatment effect scores have been considered a requirement for research publication (Leland Wilkinson and the Taskforce on Statistical Inference 1999). The initial report developed by the APA Taskforce on Statistical Inference (1996) warned that, with advances in state-of-the-art statistical analysis software, statistics are com- monly reported without understanding of the computational methods or necessarily even an understanding of what the statistics mean. Parker and colleagues have also argued that treatment effect calculations reported in meta-analysis of single-case design (SCD) research should be interpretable by various different stakeholder groups (Parker et al. 2005), a point well illustrated with autism intervention-related research where parents, educators and policy makers as well as clinicians and researchers all need to understand reports on treatment effects. Given the increasing demand to develop an evidence base in educational psychology, quality assessment guidelines have been developed by the US Department of Education What Works Clearinghouse (WWC) (Kratochwill et al. 2010; Kratochwill et al. 2013a; Kratochwill et al. 2013b). Various methods for determining treatment effects have been proposed for use in meta-analyses of SCD research, although the merits of these different computation methods remain a matter of debate (Horner and Kratochwill 2012; Horner et al. 2012; Kratochwill et al. 2013b; Scruggs and Mastropieri 2013). Unlike group research designs, a generally accepted method for the calculation of treatment effect size for SCD research has yet to be established. The initial version of the WWC SCD guidelines indicated a preference for regression based proce- dures for calculating effect sizes though the WWC panel subsequently suggested conducting a sensitivity analysis of treatment effect scores using several indices. Most recently, the WWC has moved away from the use of a treatment effect score and reverted to visual analyses until a general consensus on the most appropriate method has been reached (Kratochwill et al. 2013b). M. E. Carr (*) Monash University, Melbourne, VIC 3800, Australia e-mail: mebar4@student.monash.edu Rev J Autism Dev Disord (2015) 2:67–78 DOI 10.1007/s40489-014-0037-2