Novel tools for assessing student discussions: Modeling threads and participant roles using speech act and course topic analysis Jihie Kim, Erin Shaw, Grace Chern and Roshan Herbert University of Southern California, Information Sciences Institute 4676 Admiralty Way, Marina del Rey, CA 90292 USA {jihie,shaw}@isi.edu, {gchern,rherbert}@usc.edu Abstract. This paper describes new techniques for assessing pedagogical discourse via threaded discussions that are based on an analysis of speech acts and course topics. The context is an undergraduate computer science course. We use speech act analysis to assess the effect of instructor participation on student participation and to create thread profiles that help identify threads characterized by agreement, debate, and unresolved issues, such as threads that may have unanswered questions. For the first time, we integrate an analysis of course topics to help identify discussions of particular interest or confusion. Introduction On-line collaborative discussions (OCDs) play an important role in learning. Making use of the resulting dialog to assess understanding is an open research problem. In previous work, we developed measures of discussion activities that relied on the quantity and quality of contributions and terms [3,5]. This paper extends our existing assessment measures [2a,2b,2c] to include an analysis of speech acts and course topics. 1. Classifying Student Contributions by Speech Acts Our dataset consists of one semester of discussions from an undergraduate Operating Systems course at the University of Southern California. For conversation analysis, we defined a set of speech acts (SAs) [4] that relate pairs of messages in the dataset. We use the six SA categories shown in Table 1, derived by merging the original twelve [2a], using a Kappa value [1] to measure agreement between annotators. Table 2 shows the distribution of SAs in the set and examples of the cue words used to identify them. Table 1: New speech act categories and descriptions QUES Question ELAB Elaboration ANNO Annotation CORR/OBJ Correction/Objection/Criticism ANS/SUG Answer/Suggestion ACK/SUP Support/Acknowledgement/Complement Table 2: Distribution of speech act and surface cue words in dataset Speech Act Freq. % Sample Surface Cue Words ACK/SUP 56 7.54 “good job” ”you got it” “agree” “good/nice/correct answer” ANS/SUG 321 43.2 “perhaps” “how about” “you might” “you probably” “maybe” CORR/OBJ 41 5.52 “doesn’t mean” “are you sure” “what/how about” “not work” ELAB 53 7.13 “…and” “also” “by the way” “same question” “so…” QUES 269 36.2 “how” “what” “can we” “are”/”is” “why” “was wondering” In Proceedings of AI in Education Conference (AIED 2007)