Uncovering the Problem-Solving Process: Cued Retrospective Reporting Versus Concurrent and Retrospective Reporting Tamara van Gog, Fred Paas, Jeroen J. G. van Merrie ¨nboer, and Puk Witte Open University of the Netherlands This study investigated the amounts of problem-solving process information (“action,” “why,” “how,” and “metacognitive”) elicited by means of concurrent, retrospective, and cued retrospective reporting. In a within-participants design, 26 participants completed electrical circuit troubleshooting tasks under different reporting conditions. The method of cued retrospective reporting used the original computer- based task and a superimposed record of the participant’s eye fixations and mouse– keyboard operations as a cue for retrospection. Cued retrospective reporting (with the exception of why information) and concurrent reporting (with the exception of metacognitive information) resulted in a higher number of codes on the different types of information than did retrospective reporting. Keywords: process-tracing techniques, verbal reports, eye tracking, problem solving, knowledge elici- tation Process-tracing methods, such as concurrent reporting (“think- ing aloud”), retrospective reporting, eye tracking, and decision analysis, can be used to elicit information that allows for making inferences about the cognitive processes underlying problem- solving performance (Cooke, 1994). Hence, these methods are widely applied, for example, in usability studies, to investigate how people interact with a system or device in order to improve it (e.g., van den Haak, De Jong, & Schellens, 2003); in the design of expert systems, to uncover expert cognitive processes in order to model a system (see Richman, Gobet, Staszewski, & Simon, 1996); and in educational research, either to uncover problem- solving processes as a goal in itself or to improve instruction (e.g., Renkl, 1997). Problem solving is defined as getting from an initial problem state to a desired goal state, without knowing exactly what actions are required to get there (Newell & Simon, 1972). In problem solving, different types of knowledge are applied. Domain knowl- edge (principles) is used to mentally represent the problem and to narrow down the problem space to those problem-solving opera- tors (i.e., solution steps or actions) that may be relevant for this kind of problem. It interacts with strategic knowledge (heuristics, systematic approaches to problem solving), which is used to select operators that are most likely to lead to the goal state. Metacog- nitive knowledge is used to monitor this process of selection and application of operators by keeping track of the progress toward the goal state. For the design of instruction that makes all the knowledge used in the problem-solving process explicit to learners (e.g., process- oriented worked examples; see van Gog, Paas, & van Merrie ¨nboer, 2004), a process-tracing technique is required that is able to uncover information about problem-solving actions taken (“ac- tion”), domain principles used (“why”), strategies used (“how”), and self-monitoring (“metacognitive”). However, the results ob- tained with the two most widely applied verbal methods, concur- rent reporting and retrospective reporting, suggest that neither of these methods is suitable for providing a comprehensive picture of the problem-solving process in terms of action, why, how, and metacognitive information. With the method of concurrent reporting (Ericsson & Simon, 1993; van Someren, Barnard, & Sandberg, 1994), participants are instructed to think aloud, that is, to verbalize all thoughts that come to mind while working on a task (i.e., online). In retrospective reporting (Ericsson & Simon, 1993), participants are instructed to report the thoughts they had while they were working on a task immediately after completing it (i.e., off-line). It should be noted that in order to allow for valid inferences about the cognitive processes underlying task performance, the wording of verbaliza- tion instructions and prompts is crucial. Only when the instructions and prompts are worded in such a way that the evoked responses do not interfere with the cognitive processes can concurrent and retrospective reporting result in verbal protocols that reflect the reported cognitive processes (Ericsson & Simon, 1993). However, there is an important distinction between the infor- mation contained in concurrent and retrospective protocols, related to their online and off-line generation, respectively. Concurrent protocols reflect the information available in short-term memory during the process, whereas retrospective protocols reflect the memory traces of the process that are retrieved from short-term memory (in tasks of very short duration) or long-term memory directly after it is finished (Camps, 2003; Ericsson & Simon, Tamara van Gog, Fred Paas, Jeroen J. G. van Merrie ¨nboer, and Puk Witte, Educational Technology Expertise Center, Open University of The Netherlands, Heerlen, the Netherlands. This research project is funded by the Netherlands Organization for Scientific Research (NWO, The Hague, Project No. 411-01-010). We gratefully acknowledge K. Lynn Taylor’s efforts at providing the effect sizes for the data reported in Taylor and Dionne (2000). Correspondence concerning this article should be addressed to Tamara van Gog, Educational Technology Expertise Center, Open University of the Netherlands, P.O. Box 2960, 6401 DL, Heerlen, the Netherlands. E-mail: tamara.vangog@ou.nl Journal of Experimental Psychology: Applied Copyright 2005 by the American Psychological Association 2005, Vol. 11, No. 4, 237–244 1076-898X/05/$12.00 DOI: 10.1037/1076-898X.11.4.237 237