1 EVALUATING ‘WHAT WORKS’ IN NATIONAL QUALIFICATIONS FRAMEWORKS: EFFECTIVENESS, EFFICACY AND IMPACT Keith Morrison Abstract Whilst national qualifications frameworks (NQFs) have seen a meteoric rise, the evidence that they ‘work’ is limited, equivocal, inconclusive, contradictory and contingent. There is insufficient empirical evidence that they demonstrate fitness for purpose and achievement of purpose, and reasons for this are suggested. Problems in determining the links between cause and effect are introduced and suggestions are made for evaluating their achievement of purposes, how much impact NQFs have had (and on what and whom) and how effective and efficacious they are. The paper argues that a much stronger evidence base is required of their effectiveness, efficacy and impact, to identify those features which contribute to success (or otherwise), and to understand what is needed to ensure that they are fit for purpose. The paper suggests how this might be approached. Key words: National Qualifications Frameworks; what works; effectiveness; evaluation; evidence-based; impact; efficacy; causality; fitness for purpose; NQF. Introduction National qualifications frameworks (NQFs) have seen a meteoric rise, but the evidence that they ‘work’ in meeting their diverse purposes is limited, equivocal, inconclusive, contradictory and contingent. There is a significant lack of empirical evidence that they ‘work’, that they are effective, efficacious and demonstrate impact, fitness for purpose and achievement of purpose. This is partly because there are so many purposes that it is difficult to disaggregate and disentangle the relationship between cause and effect and to rule out whether an effect is the consequence of a particular cause or some other cause, as many NQFs are not developed in isolation but in tandem with, or in the context of, other policies and developments by governments. It is uncertain which aspects of the NQF contribute to its effectiveness, efficacy and impact, whether certain factors are more significant than others, and whether it is the interaction of factors. It is important to evaluate whether the benefit is worth the cost in fulfilling the purposes of NQFs. Currently the evidence base is too weak and the jury is still out on their value in terms of many of the purposes for which they have been introduced. Though NQFs have been around for a quarter of a century, a much stronger evidence base is required of their effectiveness, efficacy, achievement of purposes and impact, to identify those features which contribute to their success, and what is needed to ensure that they are fit for purpose. What evidence exists shows variability of practice, effectiveness, efficacy and impact.