Subgroup Analysis in Social Experiments: Measuring Program Impacts Based on Post-Treatment Choice LAURA R. PECK ABSTRACT A fundamental question within the field of program evaluation is “Do social programs work?” Although experiments allow us to answer this question with certainty, they have some limitations. Experiments generate mean program impacts and even mean impacts by subgroup, but they often leave unexplored the impacts on subgroups determined by treatment use. This work proposes a methodology for analyzing the impacts of social programs on previously unexamined subgroups. Rather than using a single trait to define subgroups—which is currently the dominant method of subgroup analysis—the proposed approach estimates the impact of programs on subgroups identified by a post-treatment choice while still maintaining the integrity of the experimental research design. Analysis of data from the experimental evaluation of New York State’s Child Assistance Program (CAP) provides an application of the proposed technique. INTRODUCTION One of the fundamental questions within the field of program evaluation is “Do social programs work?” To answer this question, many social scientists generally prefer experiments when possible, where individuals are randomly assigned to treatment and control groups so that the only difference between the two groups (other than random sampling error) is that treatment group members are offered a program and control group members are not. The outcomes of the control group members provide a counterfactual, that is, what would have happened in the absence of the treatment. This ideal counterfactual allows evaluators to net out the effects of historical and maturation trends, of selection bias, of regression artifacts and of any other plausible rival explanations of observed changes in outcomes. The difference between treatment group outcomes and control group outcomes is the impact of the program. As a Laura Peck • School of Public Affairs, Arizona State University, P.O. Box 870603, Tempe, AZ 85287-0603, USA; Tel: (1) 480-727-7081; Fax: (1) 480-965-9248; E-mail: Laura.Peck@asu.edu. American Journal of Evaluation, Vol. 24, No. 2, 2003, pp. 157–187. All rights of reproduction in any form reserved. ISSN: 1098-2140 © 2003 by American Evaluation Association. Published by Elsevier Inc. All rights reserved. 157