An examination of two mental workload measurement approaches to understanding multimedia learning Eric N. Wiebe a, * , Edward Roberts b , Tara S. Behrend c a Department of Math, Science and Technology Education, 510M Poe Hall, North Carolina State University, Raleigh, NC 27695, USA b Department of Technology, Kerr Scott Hall, Appalachian State University, Boone, NC 28608, USA c Department of Organizational Sciences & Communication, 600 21st Street, N.W., George Washington University, Washington, DC 20052, USA article info Article history: Available online 13 January 2010 Keywords: Mental workload Cognitive load theory Cognitive load measurement Subjective ratings Multimedia learning abstract This study reports on an examination of two measures of mental workload: the NASA-TLX and Paas’ Sub- jective Cognitive Load (SCL) measure. The goal was to assess the relative efficacy of the measures in the design and research of multimedia learning environments. Benchmarks based on the literature as to the goals for mental workload measurement in learning research are established. A multifaceted study was conducted which manipulated various aspects of mental workload in order to study the utility of these two measures in detecting changes in load and their relationship to learning outcomes. The results indi- cate that the weighted version of the NASA-TLX provided little additional value over the unweighted ver- sion of the measure. While both the NASA-TLX and SCL measures were sensitive to changes in both intrinsic and extraneous load, the study revealed differences in the measures based on levels of each of these load factors. The study also concludes that a better understanding of the third factor, germane load, will be needed to both expand the theoretical framework about mental workload in instructional settings and further understand the utility of these two measures. Ó 2009 Elsevier Ltd. All rights reserved. 1. Introduction There has been a long-standing interest in measuring mental workload as a means of better understanding man–machine inter- faces. While the context has historically been system control (e.g., aircraft, power plant control rooms, etc.), in the last 10 years, there has been increased interest in the effect of mental workload on learning in computer-based environments (cf., Ayres & van Gog, 2009; Mayer & Moreno, 1998; Sweller, Van Merrienboer, & Paas, 1998). As the workplace continues to move from physical work to knowledge-based activities using computing devices, interest in the optimization of mental workload through the appropriate design of content and interfaces has increased. Similarly, comput- ers have steadily increased their presence in instructional settings, resulting in a similar growth in the recognition of the need for understanding the dynamics of information flow and the manage- ment of mental workload (Borgman et al., 2008). 1.1. Measurement of mental workload The measurement of mental workload can be used as a tool for basic research, but also as either a formative or summative tool for guiding the design of control systems or learning environments. There is also the potential, though few examples exist, of using real-time feedback of mental workload for adaptive systems. When choosing an instrument, two key factors need to be considered (Eggemeier, Wilson, Kramer, & Damos, 1991). Sensitivity is the capability to detect differences in the levels of workload relevant to the task of interest. The instrument has to be able to measure meaningful differences in workload between differing designs of an interface. Diagnosticity is the capability of the instrument to dis- criminate between different types of mental workload. Depending on the theoretical framework being used and the context of the task, these dimensions of workload might be between modalities (e.g., visual versus auditory) or the relationship of the load to the central task – that is, is the load intrinsic to, or extraneous relative to the task. Both of these definitions of mental workload have been of interest to instructional scientists. In many cases, the modality of the information is seen as influencing the intrinsic and/or extra- neous load on the learner. Additional factors of interest when selecting an instrument in- clude both psychometric and logistical issues (Rubio, Diaz, Martin, & Puente, 2004). Selectivity/validity is the characteristic indicating that the instrument is not only sensitive to the elements of mental workload of interest, but is also insensitive to other variables (e.g., affective dimensions) that are not of interest. Another psychomet- ric characteristic of general interest is that the instrument has high 0747-5632/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.chb.2009.12.006 * Corresponding author. Tel.: +1 919 515 1753; fax: +1 919 515 6892. E-mail address: eric_wiebe@ncsu.edu (E.N. Wiebe). Computers in Human Behavior 26 (2010) 474–481 Contents lists available at ScienceDirect Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh