Contextual influences in verbal reporting and the contextual knowledge of the investigator in verbal protocol analysis
As is true in any disciplined investigation, the collection of verbal reports and the analysis of such data are carefully planned and closely related. It is therefore important to develop an understanding of the ways that analysis is intertwined with measurement, and the influence of the context of the study on the data being generated. To illustrate the contextual influence of the research task on the analysis of verbal report data, we present two of our own prior studies that share the same context, but differ because the research questions and subsequent data analyses were guided by different research questions, although both studies were intended to contribute to understanding of strategic processing during reading.
To begin with, the first study (Cho, Woodward, Li, & Barlow, 2017) was informed by literature on strategic processing of internet sources (e.g., Afflerbach & Cho, 2009) and explored how adolescents’ internet reading strategies contribute to the quality of their generated questions about a given topic. The research task involved participants thinking aloud during a one-hour information searching session to learn about mountaintop mining, a controversial issue, with a variety of types and sources of text available on the internet. An important element of the research design for this study (Cho et al., 2017) was recognizing the affordances and obstacles created by authentic and unconstrained textual environments. One example is participant-directed source selection, as compared to a more conventional situation of reading in a controlled environment where participants interact with predetermined texts, such as a replication of a search engine with selected texts available. A review of the methods used in previous studies of internet reading strategies indicated that there was potential in creating a task which utilized an unconstrained search environment with an open-ended task in order to best capture how students read in an internet setting (Cho, 2014; Coiro & Dobler, 2007; DeSchryver, 2015).
Therefore, our research design sought to increase ecological validity in the study of strategic processing during internet reading, which involved controlling particular elements of the task (e.g., predetermined task, prompt, and time) in order to collect data that would respond to our research questions while leaving others open (e.g., unconstrained textual space, navigation, choice of text) in order to replicate an authentic reading space. We used analytical rubrics to judge the qualities of individual students’ use of four major strategies (i.e., information location, meaning making, selfmonitoring, and source evaluation) that are prominent features in internet reading. This scoring procedure allowed us to gain “quality scores” to build a structural model that statistically accounts for the association of those strategies with the outcomes of internet reading such as knowledge gain and question generation on the topic they read about.
During our analysis of verbal protocols, however, it became apparent that there were additional influences, other than the cognitive strategies used by participants, on the information sources chosen by the participants and the related information they learned. We encountered many instances in which readers verbalized their stances, positions, or attitudes toward the task of internet reading, beyond what information they were attending to and how they were comprehending it. For example, while accessing a website run by a group of environmental scientists in the mining-affected Appalachian states, one high school reader reported:
I’m a science person, not political expert or any... I prefer to use the website of these scientists and the articles here, or whatever it has me to look at... something that is objective and looks scientific. I would not invest my time reading some of the previous articles I found, which were mostly authored by people from interest groups.
Options for coding the above verbal protocol data include self-monitoring, source evaluation, or a hybrid strategy that subsumes both. However, the complicated and idiosyncratic nature of strategy use can evoke further questions about the motivation of the reader pursuing a particular kind of source authority as informed by the reflection of how I come to know and what source to be chosen for my learning. Although such imposing of personal dispositions toward source authority was not the primary focus of the current study, we, as the investigators, understood that the unconstrained online task setting not only affords the reader freedom of information access but also encourages her to be held accountable in seeking more reliable sources. We cataloged numerous instances of such verbal reports that were deserving of further attention. Consequently a consideration of the task environment, in which adolescent readers were verbalizing their thoughts in an unconstrained setting for locating and processing varied internet sources on a controversial issue, led to our further investigation of the data from an alternative theoretical approach that we believe can assist in examining different aspects of strategic online reading.
Our contextual interpretation of particular verbal protocols extended the scope of the strategic processing we analyzed in online reading. Therefore, in order to better understand why participants engaged in the cognitive processing documented in our first study, we decided to conduct a further qualitative verbal protocol analysis for a subset of the original data. We focused on how verbal report data could demonstrate the enactment of readers’ orientations and attitudes toward knowledge (what counts as knowledge) and knowing (how one comes to know) in the vast information space on the internet (Greene, Muis, & Pieschl, 2010; Hofer, 2004). Accordingly, verbal protocol analysis was driven by a research question generated in part by the initial analysis of data which revealed readers’ epistemic beliefs: how do adolescent readers activate and engage epistemic beliefs when performing a critical online reading task? That is, we focused on what we refer to as epistemicprocessing in online reading (Cho, Woodward, & Li, 2018).
Because the verbal report data was generated in an authentic space (i.e., participants’ internet moves were not restricted, and they could engage in self-selected strategies in relation to their beliefs), our data analysis reflected the richness of the context itself. For example, Figure 23.2 draws on excerpts from two contrasting cases that imply how readers’ verbal reports could be interpreted in relation to the context in which they were given. In the first example, the student demonstrated an emotional connection to the topic throughout the session (e.g., “is upsetting”). As she encounters new information—the influence of mountaintop mining on streams—she recalls previously learned information about the mining practices, deforestation, and the greenhouse
Figure 23.2 Contextualized Coding and Analysis of Verbal Protocols
effect (e.g., “I’m thinking about the sites that I’ve read today”). She then connects the two ideas, making a knowledge claim that there is a connection between the greenhouse effect and the disappearance and polluting of streams. The ongoing dialogue that this participant created among multiple internet sources and herself contributed to understanding her epistemic engagement in interrelating and reconciling different perspectives. A similar contextual relevance is found when examining the second excerpt in Figure 23.2, but the comment therein is representative of the epistemically naive process that this participant engaged in throughout his session. Above all things, this reader gets lost in his text-selection (e.g., “I have no idea ... where I have to start”). While he does engage in several strategic approaches, the actions taken fail to demonstrate the depth of knowledge investigation required to learn from the selected texts. This pattern of reading is explained through the epistemic processing analysis, which identifies the confusion he expressed in identifying high-quality sources.
When conducting an analysis of verbal report data, there are a number of influences that the study design and methods employed have on both the data generated and the subsequent data analysis. As glimpsed from our work, however, with the nuanced knowledge of the investigators about the context of research task, verbal protocol analysis may evolve and create a new opportunity to examine different aspects of strategic processing. To highlight, the analyses of verbal reports using different lenses provided by affiliated paradigms can inform our understanding of the broad range of factors that operate with strategic processing at the intersection of cognitive and epistemic engagement.