A Framework for Health IT Evaluation Jim Warren, Malcolm Pollock, Karen Day, and Yulong Gu National Institute for Health Innovation, University of Auckland PO Box 92019, Auckland, New Zealand jim@cs.auckland.ac.nz Sue White Whitehouse Associates, Health Information Management Consultancy PO Box 12323, Thorndon, Wellington, New Zealand sue.white@whitehouse.net.nz Abstract This paper provides background and overview of a Health IT Evaluation Framework that has been developed to support the National Health IT Plan and New Zealand health innovation generally. The framework recommends a pragmatic approach that includes use of both quantitative data (particularly data based on the transactional logs of operational IT systems), and qualitative data systematically gathered through stakeholder interviews. An Action Research orientation is recommended where the evaluators actively seek to understand barriers and find pointers to potential solutions. The investigation protocol is recommended to be iterative and flexible, and to involve dissemination of intermediate findings for feedback and broad dissemination of final results. Moreover, evaluation should be integrated with implementation, rather than a standalone post implementation activity. No single type of measurement should dominate the evaluation, which should employ a measurement framework including work and communication patterns, organisational culture, safety, effectiveness, system integrity and usability, as well as vendor factors, project management, participant experience and governance. 1. Introduction The National Health IT Plan proposes an improved and rationalised health IT infrastructure for New Zealand that will ultimately support a transformed and more sustainable healthcare system. A range of innovative health IT projects is required to realise the plan, and these projects require evaluation. Moreover, it is vitally important that we learn, at a national level, from the experience acquired in each project, both in terms of its successes as well as opportunities for improvement. For this reason the National Institute for Health Innovation (NIHI) under commission of the National Health IT Board (NHITB) has developed this Framework for Health IT Evaluation. The framework provides guidelines intended to promote consistency and quality in the process of evaluation, in its reporting and in the broad dissemination of the findings. It should also promote, to some degree, the efficiency of evaluation since the effort that has gone into the framework can be leveraged by each evaluation project team. Herein we present elements of the framework’s rationale and recommendation. The framework has been tested in the context of NIHI’s evaluation of electronic referral (eReferral) projects. The eReferral reports [1-4] provide exemplars of application of the framework. We are currently applying the framework to evaluation of the National Shared Care Planning pilot for long-term condition management and have recommended its use for evaluation of the national Health Identity programme. 2. Action Research (AR) approach An AR approach to evaluation is recommended. With respect to AR, the minimal use in a project where evaluation and implementation overlap is to allow stakeholders, particularly end users of the software, to be aware of the evaluation results on an ongoing basis so that they: (a) are encouraged by the benefits observed so far and (b) explicitly react to the findings so far to provide their interpretation and feedback, as described in [5]. At the most aggressive level, one may view the entire implementation and concurrent evaluation as an undertaking of the stakeholders themselves with IT and evaluation staff purely as the facilitators of change. For instance, Participatory Action Research (PAR) methodology has