Commentary: Analyzing Strategic Processing: Pros and Cons of Different Methods

The authors in this section reviewed variable-centered (e.g., regression), personcentered (e.g., cluster analysis), and qualitative approaches to analyzing a variety of types of data collected on learning strategies. The authors discussed data collected from questionnaires, think-aloud protocols, eye tracking, and computer log files. Research on strategy use has also considered student discourse in dyads (e.g., Miller, Cromley, & Newcombe, 2015) and coding of gestures as indicative of strategy use (e.g., Alibali & Goldin-Meadow, 1993). The authors clearly laid out the different types of research questions that different analyses are suited for and various assumptions about the data that are required for the analyses. For most analyses, when data meet the assumptions of the test (e.g., normal distribution) the parametric tests are used, and when data fail one or more assumptions, nonparametric tests are used. Although mathematical transformations or bootstrapping are other options, they are not commonly used in education research, so I do not discuss them here.

In this chapter, I lay out some different data analysis options for the researcher working with strategy data. I lay out the options, in each case taking into account a number of considerations: (1) Do the data meet the assumptions for a candidate data analytic approach (e.g., normality, linearity)? (2) Are sample sizes large enough for the candidate data analytic approach? (3) Is data reduction needed (e.g., combining fine-grained codes from think-alouds, creating factor scores)?, and (4) Do large sample sizes need to be planned for in advance (e.g., full structural equation modeling)? Taking these considerations into account allows the analyst to maximize statistical power, while preserving the validity (i.e., trustworthiness) of the results from each analysis.

Variable-centered analyses

Chi Squared Test

In addition to a long tradition of t test and ANOVA analyses to compare strategy treatment groups to control groups, some early work in strategy use during learning used chi squared tests to look for disproportionate use of certain strategies in think-aloud studies between novice/expert (e.g., Ennis & Safrit, 1991) or tutored/untutored participants (e.g., Azevedo & Cromley, 2004). One advantage of the chi squared test is that it takes account of different learners being more or less talkative—some people are simply more chatty than others (see Table 24.1). It is important to take account of these differences, because analyses of raw counts of many types of data will not show the differences that researchers are trying to test; they will simply show effects of talkativeness, and talkativeness is rarely related to learning. One disadvantage of the chi squared test is that, as a non-parametric test, it has low statistical power to detect an association, if an association between variables is indeed present. Another disadvantage is that while a chi squared test may seem to show associations between codes, it merely shows co-occurrence across an entire transcript (or log file, etc.).

Table 24.1 Summary of Uses for, Pros of, and Cons of the Statistics Reviewed in this Chapter