Alternative Research Methods (Frick, et al., AECT 2009) — 1 Alternative Research Methods: MAPSAT Your Data to Prevent Aggregation Aggravation Theodore Frick, Andrew Barrett, Jake Enfield, Craig Howard and Rodney Myers Department of Instructional Systems Technology, School of Education, Indiana University Bloomington Paper presented at the Annual Conference of the Association for Educational Communications and Technology Louisville, Kentucky October 28, 2009 Overview In traditional quantitative research methods that are based on algebraic linear models, we typically obtain separate measures of variables, and then we statistically analyze relations among measures (e.g., linear, curvilinear or logistic regression analysis). That is, we relate measures. This approach, which assumes linear and additive models, can result in aggregation aggravation—i.e., obfuscation of important relationships due to assumptions in the approach. In traditional measurement we aggregate units when we obtain a value for a variable. For example, we aggregate (count) the number of inches when we measure a person's height, or we count the number of years when we measure someone's age. We repeat this process of independent aggregations for more persons' heights and ages. Then we attempt to do a statistical analysis of these sets of independent measures, such as correlation or linear regression. This kind of thinking stems from algebra—e.g., y = Bx + C, where variable y is measured separately from variable x, and a functional relationship is assumed to exist between x and y, where B is the slope and C is a constant. Alternatively, we could measure relations directly. This is not a play on words, but a significant paradigm change in conceptualizing educational research problems and how we collect and analyze data: map relations instead of measuring variables, and then analyze relation maps instead of statistically associating variables. We call this alternative approach MAPSAT: Map & Analyze Patterns & Structures Across Time. MAPSAT is a logical analysis of relations, not a statistical analysis of separate measures. In MAPSAT, there are two approaches that can be taken. In the Analysis of Patterns in Time (APT) approach, we map temporal relations. In the Analysis of Patterns in Configuration (APC) approach, we construct a map of affect-relations in a system. MAPSAT is a form of network measurement and analysis. More specifically, Dynamic Bayesian Network Analysis (DBNA) and Social Network Analysis (SNA) are similar to MAPSAT in that they are types of network analysis and are grounded in mathematical digraph theory (Thompson, 2008; Jensen & Nielsen, 2007; Brandes & Erlebach, 2005). These three approaches to network analysis are more closely related, compared with extant methods of measurement and regression analysis described above. While MAPSAT APC methods and SNA do have common aims, the advantages of MAPSAT are its theory basis (ATIS: Thompson, 2006b; 2008) and ability to measure structural properties of hypergraphs of multiple sets of affect-relations. Moreover, MAPSAT APT methods differ from DBNA in that Bayes Theorem is not assumed nor used in computing conditional probabilities in APT; rather relative frequencies of temporal sequences determine APT conditional probabilities. Examples of APT Frick (1990) invented a procedure called Analysis of Patterns in Time (APT) in order to map temporal relations. Phenomena are observed and coded with categories in classifications. The resulting temporal maps are then queried for temporal sequences of events. For example, Frick (1990) created temporal maps of student engagement and interactive instruction and found that, when interactive instruction was occurring, the temporal likelihood of student engagement was very high (0.97). However, when non-interactive instruction was occurring, then the probability of student engagement was much less (0.57). Regression analysis of the same data (when engagement and interactive instruction were aggregated separately) was only able to predict 32 percent of the variance in student engagement—i.e. aggregation aggravation. Frick, Chadha, Watson and Zlatkovska (2008) used APT in a study of teaching and learning quality in postsecondary education. Based on student course evaluations (n = 464 in 12 different courses), they found that when students agreed that First Principles of Instruction occurred in their courses (Merrill, 2002) and they also agreed that Academic Learning Time occurred (ALT: Berliner, 1990; Rangel & Berliner, 2007), students were about 5 times more likely to be rated at a High Mastery Level by their course instructors than they were when they did not