An ACT-R Model of Credibility Judgment of Micro-blogging Web Pages Q. Vera Liao (liao28@Illinois.edu) Department of Computer Science, University of Illinois, 201 N. Goodwin Ave Urbana, IL 61801 USA Peter Pirolli (pirolli@parc.com) Palo Alto Research Center, 3333 Coyote Hill Rd. Palo Alto, CA 94304 Wai-Tat Fu (wfu@illinois.edu) Department of Computer Science, University of Illinois, 201 N. Goodwin Ave Urbana, IL 61801 USA Abstract In this paper, we propose an ACT-R cognitive model for making credibility judgments about the credibility of Twitter authors. We abstracted the cognitive processes involved in three levels: attending to information on Web page, comprehending information to identify credibility cues, and integrating credibility cues to make a judgment. We represent basic knowledge required for making credibility judgment using declarative memory in ACT-R which is seeded with experiences of Twitter messages that have been passed through a Latent Dirichlet Allocation topic modeling process. Comparisons of model credibility judgments to human credibility judgments from controlled experiments show weak to strong correlations that range from r = 0.31 to r = 0.83 depending on the specific task. Keywords: Web credibility judgment, ACT-R When people make credibility judgments about Web-based content and its sources, people must perceive, comprehend and deliberate on the merits and flaws of available cues to make the judgment. Complexity arises from the fact that the judgment is rarely based on a single cue, but requires the integration of multiple cues. These cues may interact with or contradict each other, and accumulate over the course of interaction with the Web content. We present a cognitive modeling approach to investigate multi-cue Web credibility judgment. Cognitive models have been applied to explain and predict human interaction with Web-based content, primarily focusing on relevance-based browsing or search. For example, MESA (Miller & Remington, 2004) and SNIF-ACT (Fu & Pirolli, 2007) are models that simulate how users navigate through websites to search for information relevant to a given task. Web credibility judgment is a complex high-level cognitive process that may be highly dependent on the goal of the user. Therefore, instead of building a universal model, our goal is to propose a framework that can be easily modified for different contexts, and demonstrate it with a specific task. In this study, we attempt to build an ACT-R model of credibility judgment when processing Twitter micro-blogging content. Website credibility models are often conceptualized along two dimensions. One dimension, represented by stage models (Wathen & Burkell, 2002), focuses on the iterative process of credibility evaluation, i.e., how the assessment takes place when users open a page, read the contents, and are further involved with the site. The other dimension, following a bottom-up approach, seeks to examine what elements on a Web page, and to what extent, impact users’ credibility judgments. Detailed cognitive models have the potential to model the iterative processes of stage models and the impact of specific Web cues in different task and content contexts. We chose to analyze a task with simplified Twitter page, which allows us to ignore the complex interactions between multiple types of information cues but focus on the iterative process of attending to, processing and evaluating information on a Web page. This study was also motivated by the potential value of building predictive models for evaluating information credibility of micro-blogging, and more broadly, user generated contents on Internet. In the following section, we will first introduce the modeling task and a preliminary study conducted with the task. Conclusions drawn from the preliminary study are incorporated into the ACT-R model. In the second part we will describe the ACT-R model. Lastly, we will present a model validated by human data from a second experiment with the same credibility judgment tasks. Modeling Task and Preliminary Study The modeling task was based on a Twitter study conducted by Canini et al.(2011). Twitter is the popular micro-blogging service that enables users to add text-based posts of up to 140 characters, known as "tweets", on their own page. The goal of the study was to explore what factors on a Twitter page may impact users’ credibility judgment about the Twitter author. Understanding this process is important because it may help improve the design of micro- blogger recommendation systems and user interfaces to help users to discover credible sources and content. In the Canini et al. (2011) experiment, participants were presented with a page generated to represent individual Twitter users. Each of these pages included a user name and icon, a set of social status statistics (number of following, followers and tweets), 40 latest tweets by the user, and a 103