1 Meta-analysis of AusAID surveys of current and former scholarship awardees Dita Nugroho and Petra Lietz Australian Council for Educational Research DRAFT ONLY - 28 November 2011 1. Background In January 2011, AusAID Scholarships Section commissioned a study to synthesise the existing data that have been collected on AusAID scholarships and subsequently develop a consistent methodology for collecting future data. The current body of information was generated by student surveys and tracer studies that have been overseen by both the Scholarships Section and by AusAID Country Posts. The study will consist of a number of components: a) a meta-analysis of the existing-body of AusAID survey to establish a ‘big-picture’ of the impact of scholarships and a baseline for further analysis; b) mapping of existing approaches to student/alumni surveys to establish how often and by who awardees are surveyed, and what they are asked; and c) the development of a common methodology to support a regular program of reviews to be rolled-out across the program. This report presents the findings from the first part of the review, a systematic analysis of existing AusAID scholarships survey data to create a coherent picture of the outcomes of scholarships and provide a baseline for future analysis and surveys. 2. Surveys included AusAID Scholarships Section oversees and therefore houses On-Award surveys of current students, totalling eight surveys dating back to 2001. Additionally, through contacting Country Programs, 17 survey-based Post- Award surveys of scholarships alumni were identified and included in this review. Once they were identified, the surveys were scored for quality based on the survey design and content (following Rao et.al. 2008) for the purpose of this review. They were assessed for their sampling design, sample size sufficiency, representativeness of the target population and questionnaire pre-testing. All eight On-Award surveys, which utilised fairly consistent methodology and had above acceptable response rates, were deemed of high quality. Of the 17 Post-Award surveys, five were deemed of high quality, six of medium quality and six were deemed of low quality. Although none of the surveys were excluded from analysis based on quality alone, the quality of studies will be used in a short analysis following each meta- analysis result to establish the extent to which studies of different qualities impacted upon the result. 3. Study limitations The data utilised by this study were derived from survey reports and not the original survey datasets themselves. Additionally, the extent of methodological information contained within the reports varied greatly, and was extremely limited in some. This restricted this review’s ability to take into account aspect of non-response bias, for example, and a systematic handling of missing data. Surveys included in this review were undertaken by a number of different consultants and contractors at different times, in different countries and using different instruments and methodologies. Therefore, the heterogeneity challenge in combining information from these surveys is great. Although the surveys often appear to have shared foci of interest, the differences in question wording and response formats are often so great that they are not comparable. Where it is a possibility, some responses where collapsed or recalibrated (following guidelines in Kiecolt & Nathan 1985). However, it is important to note that differences in question wording and response options may affect response behaviours (see Lietz 2010). Where this applies, it will be noted under each result of the selected variables in the following section.