Improving Personalized Feedback at the Workplace with a Learning Analytics enhanced E-portfolio M. van der Schaaf Utrecht University, The Netherlands 3508 TC Utrecht, The Netherlands +31 (0)30 2534944 m.f.vanderschaaf@uu.nl G. Clarebout Maastricht University, The Netherlands ABSTRACT During workplace based learning, e.g. clinical practice or during an internship, there is an urgent need for solutions to restore and to guarantee the quality of feedback for (becoming) professionals. In continuing education at the workplace the use of Electronic portfolios (EPs) is a crucial means for acquiring the requisite professional knowledge and skills. Although EPs provide a useful approach to view each trainee’s progress, often only limited use is made of the rich contextual learning assessment data to support responsive adaptation for more efficient and rewarding training and hence to provide personalized feedback. This contribution advocates that EPs enhanced with a Learning Analytics engine, may increase the quality and efficiency of workplace-based feedback and assessment. This contribution addresses this by outlining an approach that is applied in a European 7th framework project, called WATCHME (www.project-watchme.eu). The aim of the contribution is to provide insight in underlying rationales to improve workplace-based feedback and assessment and how this is applied in an EP environment that is enhanced with Learning Analytics. Keywords Learning analytics; workplace-based learning; competencies; electronic portfolios. 1. INTRODUCTION Feedback at the workplace is crucial for trainees to become professionals. Paralleling the movement towards alternative assessments of students (Boud, 1990; Birenbaum 1996), (becoming) professionals are increasingly assessed using competence-based assessment instruments, such as portfolios. A portfolio contains selected evidence of trainees’ learning processes, their performances and products in various contexts, accompanied by supervisors’ comments and reflections (Wolf & Dietz, 1998). Depending on its content and mode of presentation an electronic portfolio (E-portfolio) can do justice to the fact that professional practice is complex and context dependent. In this paper we use Entrustable Professional Activities (EPAs) to describe units of professional practice that underlie workplace-based feedback and assessment (Gilhooly, Schumacher, West & Jones, 2014; Jones, Rosenberg, Gilhooly, & Carraccio, 2011; Ten Cate, 2013). EPAs are tasks or responsibilities entrusted to be executed by an unsupervised learner once sufficient specific competence has been obtained. EPAs are independently executable within a time frame, observable and measurable in their process and outcome, and, therefore, suitable for entrustment decisions. This is a promising route that is now being explored and implemented in several countries across the globe (e.g. USA, Canada, Australia, Singapore, The Netherlands). So far the implementation of E-portfolios in workplace- based learning is often ineffective; its quality (in terms of validity and reliability) is generally low and moreover the impact on learning is limited (Van Schaik, Plan, & O’Sullivan, 2013). This seems especially the case when the E-portfolios are not tailored to show what really happened in the workplace (Van der Schaaf, Stokking, & Verloop, 2008). Part of this failure may be attributed to a wish to translate competencies, designed as rather theoretical descriptions of professionals, into items in a portfolio for assessment. Furthermore, potential data about trainees’ behaviour in the workplace are often underused, because the management of the data is too complex for the trainees and their supervisors. This paper addresses this by outlining an iterative development approach that is applied in a European 7th framework project, called WATCHME (www.project-watchme.eu). The project uses an E-portfolio system that is enhanced with a Learning Analytics (LA) engine to provide personalized (just-in-time) assessment and feedback. LA include the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs (Clow, 2013; Ferguson, 2012; Siemens & Long, 2011). The design approach for the LA engine that drives the E- portfolio is of a cyclical nature based on ongoing refinement and improvement of the engine during successive phases of collection, analysis and visualising information (Baker & Yacef, 2008; Elias, 2011). Though LA are driven by a computerised processing of large amounts of data, the analytical process is a ”single amalgam of human and machine processing which is instantiated through an interface that both drives and is driven by the whole system, human and machine” (Dron & Anderson, 2009, p. 369). Student Models will be used as a means of analysis, the results of which will lead to two types of feedback: Just-in Time feedback messages and visualization of both individual and aggregated data. In order to provide meaningful just-in-time information, the Student Model should represent the actual internal state of each trainee as well as their actual learning context. For this, it must be able to interpret the contents of the E-portfolio. The Student Model should also contain enough pedagogical knowledge in order to be able to