Yes, Psychologists Must Change the Way They Analyze Their Data: Clarifications for Bem, Utts, and Johnson (2011) Eric–Jan Wagenmakers, Ruud Wetzels, Denny Borsboom, Rogier Kievit, & Han L. J. van der Maas University of Amsterdam Abstract Does psi exist? In a widely publicized article featuring nine experiments with over one thousand participants, Bem (in press) claimed that future events retroactively affect people’s responses. In a response, we pointed out that Bem’s analyses were partly exploratory. Moreover, we reanalyzed Bem’s data using a default Bayesian t-test and showed that Bem’s evidence for psi is weak to nonexistent. A robustness analysis confirmed our skeptical con- clusions. Recently, Bem, Utts, and Johnson (2011) question several aspects of our analysis. In this brief reply we clarify our analysis procedure and demonstrate that our arguments still hold. Keywords: Confirmatory Experiments, Bayesian Hypothesis Test, ESP. The History and the Hype In a recent article for Journal of Personality and Social Psychology, Bem (in press) presented nine experiments that test for the presence of psi. Specifically, Bem’s experi- ments were designed to assess the hypothesis that future events affect people’s thinking and people’s behavior in the past (henceforth precognition). Bem argued that in eight out of the nine experiments, the data supported the presence of precognition, that is, one-sided p values were smaller than .05. Bem’s findings—and, perhaps more importantly, the fact that they were going to be published in a major journal—created a storm of media attention. In the New York Times, several researchers voiced strong opinions: Dr. Ray Hyman, a long-time critic of ESP research, questioned the quality of the refereeing process as he believed that the publication of Dr. Bem’s article was “(...) pure craziness (...) an embarrassment for the This version was last updated with minor changes on February 18th, 2011. This research was supported by Vidi grants from the Dutch Organization for Scientific Research (NWO). Correspondence concerning this article may be addressed to Eric–Jan Wagenmakers, University of Amsterdam, Department of Psychology, Roetersstraat 15, 1018 WB Amsterdam, the Netherlands. Email address: ej.wagenmakers@gmail.com.