Are cost-inclusive evaluations worth the effort? Patricia M. Herman *, Deirdre J. Avery, Crystal S. Schemp, Michele E. Walsh University of Arizona, United States If the goal of evaluation is to provide pertinent information on social programs to program managers, stakeholders, and policy makers, its sole focus cannot be program outcomes or effective- ness. Relative costs are as important as relative effectiveness when choosing between program alternatives or among a set of programs competing for scarce funds. Although a number of authors have been calling for an increase in the consideration of costs in evaluation, the number of effectiveness or outcomes studies still far outweighs the number of cost-inclusive (e.g., cost-effectiveness or cost-benefit) evalua- tions (Drummond, O’Brien, Stoddart, & Torrance, 1997; Fals- Stewart, Yates, & Klostermann, 2005; Levin & McEwan, 2001; Rossi, Freeman, & Lipsey, 1999; Yates & Taub, 2003). If this deficiency is due to concern on the part of evaluators as to whether the methods involved are within their capabilities, or even more importantly, worth the time and effort required to apply them, this paper hopes to allay those fears. This paper presents the process taken and results generated by three experienced evaluators during their first attempt to perform cost-inclusive evaluation. Although we will briefly explain each technique used and the results of the analyses, our goal is neither to explain them in detail, nor to focus on the cost-effectiveness results. Instead, we will focus on the evaluators’ experiences in terms of the time taken to perform the analyses, any unexpected difficulties experienced, and the ‘‘results’’ in terms of new insights into the programs studied and a clearer vision for future evaluation needs. The paper is organized as follows. The first section presents a brief overview of cost-inclusive evaluation. Subsequent sections present the three analyses performed by the evaluators. Each evaluator performed a different type of cost-inclusive evaluation: ‘‘standard’’ cost-effectiveness analysis, threshold or break-even analysis, and a simulation model. Although all three examples come from the field of tobacco control, the general techniques used and insights gained should be applicable to evaluation in any field. The paper ends with a summary of the challenges faced and benefits gained by these evaluators through performing these cost- inclusive evaluations and lessons learned. 1. Including costs in evaluation: the basics Evaluators often feel inadequately trained to undertake cost- inclusive evaluations. Adding to evaluators’ lack of confidence regarding these evaluations is the inconsistency in the terms used across texts and across industries, areas and topics. The inclusion of both costs and effects in an intervention’s evaluation has been called economic evaluation, efficiency assessment, cost-related analysis, cost-inclusive evaluation, cost-effectiveness analysis, and cost-benefit analysis. All these terms are correct after a fashion; however, the last two also have more specific definitions. In this text we will use the term cost-inclusive evaluation to indicate any evaluation that includes both a program’s effects or outcomes and its cost. Evaluation and Program Planning 32 (2009) 55–61 ARTICLE INFO Article history: Received 21 December 2007 Received in revised form 11 July 2008 Accepted 22 August 2008 Keywords: Cost-effectiveness Cost-benefit Efficiency Break-even Simulation model Tobacco control ABSTRACT Relative costs are as important as relative effectiveness when choosing between program alternatives or among a set of programs competing for scarce funds. Nevertheless, the number of cost-inclusive evaluations remains comparatively small. This article presents the results of three first-time cost-inclusive evaluations each performed by an experienced evaluator. Each evaluator performed a different type of cost-inclusive evaluation using different tobacco control programs as examples: ‘‘standard’’ cost-effectiveness analysis, threshold or break-even analysis, and a simulation model. Results are presented in terms of the challenges faced, and informational and insight benefits gained, as well as in terms of program cost-effectiveness. All three evaluators agreed that the benefits from performing cost-inclusive evaluations are well worth time and effort involved. They also found that this type of evaluation provides abundant information that can be used to improve program effectiveness and cost-effectiveness. ß 2008 Elsevier Ltd. All rights reserved. * Corresponding author at: Evaluation, Research, and Development Unit, Department of Psychology, University of Arizona, P.O. Box 210462, Tucson, AZ 85621-0462, United States. E-mail address: pherman@email.arizona.edu (P.M. Herman). Contents lists available at ScienceDirect Evaluation and Program Planning journal homepage: www.elsevier.com/locate/evalprogplan 0149-7189/$ – see front matter ß 2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.evalprogplan.2008.08.008