THREE ACHILLES HEELS OF PROGRAM THEORY EVALUATION Presenters: Patricia Rogers (Royal Melbourne Institute of Technology) Bron McDonald (Department of Natural Resources and Environment, Victoria) Discussant: Nick Tilley, (Nottingham Trent University and UK Home Office) OVER VIEW OF SYMPOSIUM Program theory evaluation is an attractive technique which encourages participation by staff, can be used for incremental building of organisational capacity to evaluate, and has spin-off benefits in program planning and management. But, after more than a decade of using it, there remain challenges in the theory and practice of program theory evaluation. This presentation focuses on 3 of these challenges: • the continuing difficulty of attributing the observed outcomes to the program; • the tendency to focus on intended outcomes to the exclusion of unintended outcomes; and • practical issues in identifying and understanding how different groups of service users experience the program, and follow different paths to reach the intended outcomes, or to reach other outcomes FORMAT Three panellists will combine their complementary experience to examine these issues. The symposium is organised in the form of a dialogue between theory and practice, focusing on each issue in turn. Patricia Rogers is a Director of the Program for Public Sector Evaluation at the Royal Melbourne Institute of Technology. She has a special interest in program theory evaluation, having used it, and encouraged others to use it, since 1987. She will address each issue “in theory”, through presenting a summary of each issue, the challenges posed, and different solutions which have been posed. She will also begin with an introduction to program theory evaluation to ensure the symposium is accessible and useful even for participants who are unfamiliar with the approach. Bron McDonald is the leader of the Evaluation Pilot Program in the Department of Natural Resources and Environment (Victorian State Government) which works to develop methods for evaluating projects and programs within the Primary Industries Division of the department. She will address each issue “in practice” through presenting examples of actual evaluation work that has addressed these challenges. Nick Tilley is Professor of Sociology at Nottingham Trent University, currently seconded to the Home Office, and co-author of Realistic Evaluation – a book which argued that program evaluation should be built on program models which identified the contexts within which desirable processes operated. He will be a discussant for each issue.