Prehospital and Disaster Medicine http://pdm.medicine.wisc.edu Vol. 22, No. 5 SPECIAL REPORT 1. Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland USA 2. Harvard Business School, Allston, Massachusetts USA 3. Dartmouth Medical School, Hanover, New Hampshire USA Correspondence: Paul Bolton, MBBS, MPH Center for Refugee and Disaster Studies Johns Hopkins Bloomberg School of Public Health 615 N. Wolfe St., Room E8646 Baltimore, Maryland 21205 USA E-mail pbolton@jhsph.edu The case studies described here were funded directly by the implementing partners: FilmAid International (Kenya), World Vision and War Child Holland (Northern Uganda). Keywords: disaster; evaluation; humanitarian;impact; method; objective; program; response Abbreviations: None. Web publication: 11 October 2007 Expanding the Scope of Humanitarian Program Evaluation Paul Bolton, MBBS, MPH; 1 Judith Bass, PhD; 1 Laura Murray, PhD; 1 Katharine Lee, MA, MPH; 2 William Weiss, DrPH, MA; 1 Sharon M. McDonnell, MD, MPH 3 Introduction Impact evaluation is a core element of humanitarian assistance programs since it provides the basis for improvements in future programs. 1–5 A positive development during the last decade has been the movement to introduce standards and measures of health status into program impact evaluations. 6,7 Examples of standards that are measured include: (1) liters of water per person per day; (2) square meters of shelter space per person; (3) measles vaccination coverage; and (4) kilocalories per person per day. Common health status mea- sures that now are included in the evaluation of humanitarian assistance pro- grams include: (1) number of deaths per day; (2) percent of children <5 years of age with acute malnutrition; and (3) incidence rates of measles and acute watery diarrhea. 8,9 Recently, functional status has been included as a measure of health status in the evaluations of psychosocial programs. 10 A further pos- itive development is the effort to standardize the measurement(s) of health status in emergencies. 9 All of these measures are an improvement over the past practice of evaluation that was based on program outputs (such as the number of tons of food shipped, the number of shelter tarpaulins distributed, or the number of latrines constructed) because program outputs alone do not demonstrate the actual impact of the program on the lives of those being served. While humanitarian programs have clearly made progress in assessing impact, such assessments typically are limited to assessing a small number of pre-defined objectives. This limited focus has several benefits including pro- grams and associated information systems that are easier to implement, stan- dardize, control quality, and are less resource-intensive. However, humanitarian assistance programs can have important, unexpected impacts beyond these pre-determined objectives. 1,11–14 An example of an unexpected impact emerged during a qualitative study by two of the authors (PB and WW), in Angola during the late 1990s. Interviewers were told by Angolan villagers how the glint of new metal roofs in the sun, visible from great distances, had Abstract The effectiveness of humanitarian programs normally is evaluated according to a limited number of pre-defined objectives. These objectives typically rep- resent only selected positive expected impacts of program interventions and as such, are inadequate benchmarks for understanding the overall effective- ness of aid. This is because programs also have unexpected impacts (both pos- itive and negative) as well as expected negative impacts and expected positive impacts beyond the program objectives. The authors contend that these other categories of program impacts also should be assessed, and suggest a method- ology for doing so that draws on input from the perspectives of beneficiaries. This paper includes examples of the use of this methodology in the field. Finally, the authors suggest future directions for improving this type of expanded assessment and advocate for its widespread use, both within and without the field of disaster response. Bolton P, Bass J, Murray L, Lee K, Weiss W, McDonnell SM: Expanding the scope of humanitarian program evaluation. Prehospital Disast Med 2007;22(5):390–395.