Making Effects Manifest in Randomized Experiments Jake Bowers * July 19, 2010 Experimentalists desire precise estimates of treatment effects and nearly always care about how treatment effects may differ across subgroups. After data collection, concern may focus on ran- dom imbalance between treatment groups on substantively important variables. Pursuit of these three goals — enhanced precision, understanding treatment effect heterogeneity, and imbalance adjustment — requires background information about experimental units. For example, one may group similar observations on the basis of such variables and then assign treatment within those blocks. Use of covariates after data have been collected raises extra concerns and requires special justification. For example standard regression tables only approximate the statistical inference that experimentalists desire. The standard linear model may also mislead via extrapolation. After providing some general background about how covariates may, in principle, enable pursuit of precision and statistical adjustment, this paper presents two alternative approaches to covariance adjustment: one using modern matching techniques and another using the linear model — both use randomization as the basis for statistical inference. 1 What is a manifest effect? A manifest effect is one we can distinguish from zero. Of course, we cannot talk formally about the effects of an experimental treatment as manifest without referring to probability: a scientist asks, “Could this result have occurred merely through chance?” or “If the true effect were zero, what is the chance that we’d observe an effect as large as this?” More formally, for a frequentist, saying a treatment effect is manifest is saying that the statistic we observe casts a great deal of doubt on a hypothesis of no effects. We are most likely to say that some observed effect casts doubt on the null hypothesis of no effect when we have a large sample and/or when noise in the outcome that might otherwise drown out the signal in our study has been well controlled. Fisher reminds us that while randomization alone is sufficient for a valid test of the null hypothesis of no effect, specific features of a given design allow equally valid tests to differ in their ability to make a treatment effect * Assistant Professor, Dept of Political Science, University of Illinois @ Urbana-Champaign Corresponding Author Contact Information: 231 Computing Applications Building, 605 E Spring- field Ave Champaign IL 61820 —217.333.3881 — jwbowers@illinois.edu. Acknowledgements: Many thanks to Jamie Druckman, Don Green, Ben Hansen, Jim Kuklinski, Thomas Leeper, Costas Panagopoulos, and Cara Wong. Parts of this work were funded by NSF Grants SES-0753168 and SES-0753164. 1