Phishing IQ Tests Measure Fear, not Ability Vivek Anandpara Andrew Dingman Markus Jakobsson Debin Liu Heather Roinestad January 15, 2007 Abstract We argue that phishing IQ tests fail to measure susceptibility to phish- ing attacks. We conducted a study where 40 subjects were asked to answer a selection of questions from existing phishing IQ tests in which we var- ied the portion (from 25% to 100%) of the questions that corresponded to phishing emails. We did not find any correlation between the actual number of phishing emails and the number of emails that the subjects indicated were phishing. Therefore, the tests did not measure the ability of the subjects. To further confirm this, we exposed all the subjects to existing phishing education after they had taken the test, after which each subject was asked to take a second phishing test, with the same design as the first one, but with different questions. The number of stimuli that were indicated as being phishing in the second test was, again, indepen- dent of the actual number of phishing stimuli in the test. However, a substantially larger portion of stimuli was indicated as being phishing in the second test, suggesting that the only measurable effect of the phish- ing education (from the point of view of the phishing IQ test) was an increased concern—not an increased ability. Keywords: phishing, phishing education, phishing IQ test 1 Introduction Popular media routinely covers the mounting problem of phishing. Financial institutions frequently alert clients of the risks of identity theft, and many pro- vide detailed descriptions of common attacks and how to avoid falling victim to these. With this popular focus on the problem, we must ask ourselves why the recent trends show an increase in the number of people that fall victim to phishing. Furthermore, we must pose the question whether current educational efforts are meaningful and whether the ways in which vulnerabilities are assessed work. To be able to ask these questions, it is important first to understand why phishing works. This question has been asked by several researchers recently [5, 6, 7, 21], and a collection of insightful conclusions have been found. One reason that phishing works is that most people do not have a detailed understanding 1